Ehcache 3 size in memory - ehcache-3

In Ehcache 2, it was possible to get the size in memory using calculateInMemorySize() e.g.:
CacheManager manager = CacheManager.create();
Cache cache = new Cache(
new CacheConfiguration("test", 100)
.eternal(true)
.statistics(true));
manager.addCache(cache);
assertEquals(0, cache.calculateInMemorySize());
cache.put(new Element(0, "A"));
cache.put(new Element(1, "B"));
cache.put(new Element(2, "C"));
assertTrue(cache.calculateInMemorySize() > 0);
manager.shutdown();
What is the equivalent in Ehcache 3.6.1?
In the following, the getOccupiedByteSize() method returns -1:
StatisticsService statisticsService = new DefaultStatisticsService();
CacheManager manager = CacheManagerBuilder.newCacheManagerBuilder()
.using(statisticsService)
.build(true);
CacheConfiguration<Integer, String> config =
CacheConfigurationBuilder.newCacheConfigurationBuilder(
Integer.class, String.class, ResourcePoolsBuilder.heap(100))
.withExpiry(ExpiryPolicy.NO_EXPIRY)
.build();
Cache<Integer, String> cache = manager.createCache("test", config);
cache.put(0, "A");
cache.put(1, "B");
cache.put(2, "C");
CacheStatistics stats = statisticsService.getCacheStatistics("test");
TierStatistics heap = stats.getTierStatistics().get("OnHeap");
assertTrue(heap.getOccupiedByteSize() > 0);
manager.close();
It does work for a memory unit limited cache. e.g. when the cache configuration is:
CacheConfiguration<Integer, String> config =
CacheConfigurationBuilder.newCacheConfigurationBuilder(
Integer.class, String.class,
ResourcePoolsBuilder.newResourcePoolsBuilder().heap(100, MemoryUnit.MB))
.withExpiry(ExpiryPolicy.NO_EXPIRY).build();
This is not desirable as the cache must be configured with an element limit, not a memory limit.
As per Henri's answer below, it is possible to get the memory size using Sizeof.deepSizeOf(), however some knowledge of the cache implementation is required to exclude unwanted objects e.g.:
SizeOfFilter ehcache = new SizeOfFilter() {
#Override
public Collection<Field> filterFields(Class<?> klazz, Collection<Field> fields) {
if (EhcacheBase.class.isAssignableFrom(klazz)) {
for (Field field : fields) {
if ("store".equals(field.getName())) {
return Collections.singletonList(field);
}
}
return Collections.emptyList();
}
return fields;
}
#Override
public boolean filterClass(Class<?> klazz) {
return true;
}
};
SizeOf sizeOf = SizeOf.newInstance(ehcache);
long size = sizeOf.deepSizeOf(cache);
There is an enhancement request to simplify it here: https://github.com/ehcache/ehcache3/issues/2500

It's not possible. By design.
We do not calculate size when the limit is entries. For performance reason. Keeping track of the size is quite costly.
However, I think in your case, I would do a sizeof. If you don't need it that often, just do a sizeof of the cache and it will give you the current cache size nicely.
SizeOf sizeOf = SizeOf.newInstance();
long deepSize = sizeOf.deepSizeOf(cache); // in bytes

Related

Spark mapPartitionsToPai execution time

In the current project I am working, we are using spark as computation engine for one of workflows.
Workflow is as follows
We have product catalog being served from several pincodes. User logged in from any particular pin code should be able to see least available cost from all available serving pincodes.
Least cost is calculated as follows
product price+dist(pincode1,pincode2) -
pincode2 being user pincode and pincode1 being source pincode. Apply the above formula for all source pincodes and identify the least available one.
My Core spark logic looks like this
pincodes.javaRDD().cartesian(pincodePrices.javaRDD()).mapPartitionsToPair(new PairFlatMapFunction<Iterator<Tuple2<Row,Row>>, Row, Row>() {
#Override
public Iterator<Tuple2<Row, Row>> call(Iterator<Tuple2<Row, Row>> t)
throws Exception {
MongoClient mongoclient = MongoClients.create("mongodb://localhost");
MongoDatabase database = mongoclient.getDatabase("catalogue");
MongoCollection<Document>pincodeCollection = database.getCollection("pincodedistances");
List<Tuple2<Row,Row>> list =new LinkedList<>();
while (t.hasNext()) {
Tuple2<Row, Row>tuple2 = t.next();
Row pinRow = tuple2._1;
Integer srcPincode = pinRow.getAs("pincode");
Row pricesRow = tuple2._2;
Row pricesRow1 = (Row)pricesRow.getAs("leastPrice");
Integer buyingPrice = pricesRow1.getAs("buyingPrice");
Integer quantity = pricesRow1.getAs("quantity");
Integer destPincode = pricesRow1.getAs("pincodeNum");
if(buyingPrice!=null && quantity>0) {
BasicDBObject dbObject = new BasicDBObject();
dbObject.append("sourcePincode", srcPincode);
dbObject.append("destPincode", destPincode);
//System.out.println(srcPincode+","+destPincode);
Number distance;
if(srcPincode.intValue()==destPincode.intValue()) {
distance = 0;
}else {
Document document = pincodeCollection.find(dbObject).first();
distance = document.get("distance", Number.class);
}
double margin = 0.02;
Long finalPrice = Math.round(buyingPrice+(margin*buyingPrice)+distance.doubleValue());
//Row finalPriceRow = RowFactory.create(finalPrice,quantity);
StructType structType = new StructType();
structType = structType.add("finalPrice", DataTypes.LongType, false);
structType = structType.add("quantity", DataTypes.LongType, false);
Object values[] = {finalPrice,quantity};
Row finalPriceRow = new GenericRowWithSchema(values, structType);
list.add(new Tuple2<Row, Row>(pinRow, finalPriceRow));
}
}
mongoclient.close();
return list.iterator();
}
}).reduceByKey((priceRow1,priceRow2)->{
Long finalPrice1 = priceRow1.getAs("finalPrice");
Long finalPrice2 = priceRow2.getAs("finalPrice");
if(finalPrice1.longValue()<finalPrice2.longValue())return priceRow1;
return priceRow2;
}).collect().forEach(tuple2->{
// Business logic to push computed price to mongodb
}
I am able to get the answer correctly, however mapPartitionsToPair is taking a bit of time(~22 secs for just 12k records).
After browsing internet I found that mapPartitions performs better than mapPartitionsToPair, but I am not sure how to emit (key,value) from mapPartitions and then sort it.
Is there any alternative for above transformations or any better approach is highly appreciated.
Spark Cluster: Standalone(1 executor, 6 cores)

Concatenate multiple PDF/A with different conformance levels

Is it possible to concatenate a number of pdf/a (with possibly different conformance levels: some pdf/a-1b, some pdf/a-3b ecc) into a single pdfa ?
I was thinking that using the latest level (3-a or 3b) would be ok but I get errors when validating with VeraPDF:
Here is my code (where :
public static byte[] CreateConformantCopy(List<byte[]> sourcePdfs)
{
var version = PdfVersion.PDF_1_7;
var type = PdfAType.PDF_A_3B;
WriterProperties wp = new WriterProperties();
wp.UseSmartMode();
wp.SetPdfVersion(version.ToPdfVersion());
PdfOutputIntent oi = new PdfOutputIntent("Custom", "", "http://www.color.org", "sRGB IEC61966-2.1", Assembly.GetExecutingAssembly().GetManifestResourceStream("xxx.Resources.sRGB_CS_profile.icm"));
using (var mergedPdf = new MemoryStream())
{
var writer = new PdfWriter(mergedPdf, wp);
using (PdfADocument newDoc = new PdfADocument(writer, type.ToPdfAConformanceLevel(), oi, new DocumentProperties() { }))
{
Document document = new Document(newDoc, PageSize.A4.Rotate());
newDoc.SetTagged();
newDoc.GetCatalog().SetLang(new PdfString(Thread.CurrentThread.CurrentUICulture.Name));
newDoc.GetCatalog().SetViewerPreferences(
new PdfViewerPreferences()
.SetDisplayDocTitle(true)
.SetCenterWindow(true)
);
PdfMerger merger = new PdfMerger(newDoc);
for (int k = 0; k < sourcePdfs.Count; k++)
{
using (var inDoc = PdfHelper.GetDocument(sourcePdfs[k]))
{
var numberOfPages = inDoc.GetNumberOfPages();
merger.Merge(inDoc, 1, numberOfPages);
}
}
newDoc.Close();
}
return mergedPdf.ToArray();
}
}
PDF/A-1 and PDF/A-2 have several differences in the requirements. So, merging them together might not be possible. Looking on your validation errors, I think this is exactly the case. For example, the very first one is about XMP metadata. The PDF/A-2 is more strict here, and you get this error because your first file (which is probably a valid PDF/A-1) does not actually satisfy the PDF/A-2 rules.
What is possible however is to attach a PDF/A-1 document to PDF/A-2 one. This does not even require the use of PDF/A-3, which allows arbitrary attachments. The PDF/A-2 standard does allow attaching valid PDF/A-1 (as well as PDF/A-2 documents).

How to place the Same Digital signatures to Multiple places in PDF using itextsharp.net

I have implemented Digital Signature using iTextSharp Dll to sign PDF files with a single signature creating empty signature fields and update the signature field with signed hash working fine. Now, I want to place the same digital signature in every page of pdf. It's my client requirement.
I’m using the following code:
public class MyExternalSignatureContainer : IExternalSignatureContainer
{
private readonly byte[] signedBytes;
public MyExternalSignatureContainer(byte[] signedBytes)
{
this.signedBytes = signedBytes;
}
public byte[] Sign(Stream data)
{
return signedBytes;
}
public void ModifySigningDictionary(PdfDictionary signDic)
{
}
}
Below code used in program
PdfReader reader = new PdfReader(unsignedPdf);
FileStream os = File.OpenWrite(tempPdf);
PdfStamper stamper = PdfStamper.CreateSignature(reader, os, '\0');
PdfSignatureAppearance appearance = stamper.SignatureAppearance;
appearance.Reason = "Reason1";
appearance.Contact = "";
appearance.Location = "Location1";
appearance.Acro6Layers = false;
appearance.Image = null;
appearance.SignatureRenderingMode = PdfSignatureAppearance.RenderingMode.DESCRIPTION;
appearance.SetVisibleSignature(new iTextSharp.text.Rectangle(36, 748, 144, 780), 1, null);
for (int i = 1; i < 8; i++)
{
var signatureField = PdfFormField.CreateSignature(stamper.Writer);
var signatureRect = new Rectangle(200, 200, 100, 100);
signatureField.Put(PdfName.T, new PdfString("ClientSignature_"+i.ToString()));
PdfIndirectReference PRef = stamper.Writer.PdfIndirectReference;
signatureField.Put(PdfName.V, PRef);
signatureField.Put(PdfName.F, new PdfNumber("132"));
signatureField.SetWidget(signatureRect, null);
signatureField.Put(PdfName.SUBTYPE, PdfName.WIDGET);
PdfDictionary xobject1 = new PdfDictionary();
PdfDictionary xobject2 = new PdfDictionary();
xobject1.Put(PdfName.N, appearance.GetAppearance().IndirectReference);
xobject2.Put(PdfName.AP, xobject1);
signatureField.Put(PdfName.AP, xobject1);
signatureField.SetPage();
PdfDictionary xobject3 = new PdfDictionary();
PdfDictionary xobject4 = new PdfDictionary();
xobject4.Put(PdfName.FRM, appearance.GetAppearance().IndirectReference);
xobject3.Put(PdfName.XOBJECT, xobject4);
signatureField.Put(PdfName.DR, xobject3);
stamper.AddAnnotation(signatureField, i);
}
IExternalSignatureContainer external = new ExternalBlankSignatureContainer(PdfName.ADOBE_PPKMS, PdfName.ADBE_PKCS7_DETACHED);
MakeSignature.SignExternalContainer(appearance, external, 8192);
stamper.Close();
byte[] SignedHash = DoEsign(SHA256Managed.Create().ComputeHash(appearance.GetRangeStream());
os.close();
reader.close();
reader = new PdfReader(tempPdf))
os = File.OpenWrite(signedPdf)
IExternalSignatureContainer external1 = new MyExternalSignatureContainer(SignedHash);
MakeSignature.SignDeferred(reader, signatureFieldName, os, external1);
os.close();
reader.close();
Please suggest me to complete the task
To give all signature fields the same single value wrapping the newly created signature container, they must all reference the same indirect object as value. Unfortunately iText creates the indirect object for the signature value only after the application code had the chance to add its additional fields which in turn require a reference to that signature value object. Thus, the application code has to anticipate the object number that indirect object will have.
This anticipation or prediction of the object number is very delicate, it depends on the exact same use case and can also become incorrect as the result of minor changes in the iTextSharp library
To make this easier, the application code should add those signature fields with their signature value references as late as possible, so there are as few other new indirect objects created as possible until iText creates the value indirect object.
As it turns out, the ModifySigningDictionary method of an IExternalSignatureContainer is a good position for that.
As soon as one adds one's code there, another issue pops up: There is no means to set the anticipated object number in a PdfIndirectReference instance externally. One way to get around this is to mimic such a reference using a PdfLiteral. (Well, probably one could also use reflection for this.)
Furthermore it turns out that one best creates the appearance streams to use by all one's additional signature fields before building that PdfLiteral mimicking a PdfIndirectReference as this simplifies the calculation of the object number iText will use for the actual value object.
With this in mind, here a proof-of concept. This proof of concept makes use of an IExternalSignature instance for actually signing. This is not a necessary precondition, one can also use an IExternalSignatureContainer instead with only a few changes, even an ExternalBlankSignatureContainer as in the question to later finalize the signature using MakeSignature.SignDeferred.
So given cipher parameters cp (private key material, e.g. pk.Key for an Org.BouncyCastle.Pkcs.AsymmetricKeyEntry pk) and a certificate chain chain, one would use
PdfReader reader = new PdfReader(SRC);
FileStream os = new FileStream(DEST, FileMode.Create, FileAccess.Write);
PdfStamper stamper = PdfStamper.CreateSignature(reader, os, '\0');
PdfSignatureAppearance appearance = stamper.SignatureAppearance;
appearance.Reason = "Reason1";
appearance.Contact = "";
appearance.Location = "Location1";
appearance.Acro6Layers = false;
appearance.Image = null;
appearance.SignatureRenderingMode = PdfSignatureAppearance.RenderingMode.DESCRIPTION;
appearance.SetVisibleSignature(new iTextSharp.text.Rectangle(10, 10, 100, 100), reader.NumberOfPages, null);
IExternalSignature externalSignature = new PrivateKeySignature(cp, "SHA-256");
AllPagesSignatureContainer allPagesContainer = new AllPagesSignatureContainer(appearance, externalSignature, chain);
MakeSignature.SignExternalContainer(appearance, allPagesContainer, 8192);
with this external signature container class
public class AllPagesSignatureContainer : IExternalSignatureContainer
{
public AllPagesSignatureContainer(PdfSignatureAppearance appearance, IExternalSignature externalSignature, ICollection<X509Certificate> chain)
{
this.appearance = appearance;
this.chain = chain;
this.externalSignature = externalSignature;
}
public void ModifySigningDictionary(PdfDictionary signDic)
{
signDic.Put(PdfName.FILTER, PdfName.ADOBE_PPKMS);
signDic.Put(PdfName.SUBFILTER, PdfName.ADBE_PKCS7_DETACHED);
PdfStamper stamper = appearance.Stamper;
PdfReader reader = stamper.Reader;
PdfDictionary xobject1 = new PdfDictionary();
PdfDictionary xobject2 = new PdfDictionary();
xobject1.Put(PdfName.N, appearance.GetAppearance().IndirectReference);
xobject2.Put(PdfName.AP, xobject1);
PdfIndirectReference PRef = stamper.Writer.PdfIndirectReference;
PdfLiteral PRefLiteral = new PdfLiteral((PRef.Number + 1 + 2*(reader.NumberOfPages - 1)) + " 0 R");
for (int i = 1; i < reader.NumberOfPages; i++)
{
var signatureField = PdfFormField.CreateSignature(stamper.Writer);
signatureField.Put(PdfName.T, new PdfString("ClientSignature_" + i.ToString()));
signatureField.Put(PdfName.V, PRefLiteral);
signatureField.Put(PdfName.F, new PdfNumber("132"));
signatureField.SetWidget(appearance.Rect, null);
signatureField.Put(PdfName.SUBTYPE, PdfName.WIDGET);
signatureField.Put(PdfName.AP, xobject1);
signatureField.SetPage();
Console.WriteLine(signatureField);
stamper.AddAnnotation(signatureField, i);
}
}
public byte[] Sign(Stream data)
{
String hashAlgorithm = externalSignature.GetHashAlgorithm();
PdfPKCS7 sgn = new PdfPKCS7(null, chain, hashAlgorithm, false);
IDigest messageDigest = DigestUtilities.GetDigest(hashAlgorithm);
byte[] hash = DigestAlgorithms.Digest(data, hashAlgorithm);
byte[] sh = sgn.getAuthenticatedAttributeBytes(hash, null, null, CryptoStandard.CMS);
byte[] extSignature = externalSignature.Sign(sh);
sgn.SetExternalDigest(extSignature, null, externalSignature.GetEncryptionAlgorithm());
return sgn.GetEncodedPKCS7(hash, null, null, null, CryptoStandard.CMS);
}
PdfSignatureAppearance appearance;
ICollection<X509Certificate> chain;
IExternalSignature externalSignature;
}
The predicted indirect object number of the signature value in the line
PdfIndirectReference PRef = stamper.Writer.PdfIndirectReference;
PdfLiteral PRefLiteral = new PdfLiteral((PRef.Number + 1 + 2*(reader.NumberOfPages - 1)) + " 0 R");
strictly depends upon the use case being "exactly one signature field per page". For different use cases the estimate the prediction would differ.
I stress this here once again because e.g. the OP of this question did not take this into account when trying "to place multiple signatures on single page".
Another strict requirement for the object number prediction above is that the PdfStamper is created as above, i.e. not in append mode. If the signature is applied as an incremental update, i.e. in append mode, the lines above have to be replaced by
stamper.Writer.AddToBody(new PdfNull(), stamper.Writer.PdfIndirectReference, true);
PdfIndirectReference PRef = stamper.Writer.PdfIndirectReference;
PdfLiteral PRefLiteral = new PdfLiteral((PRef.Number + reader.NumberOfPages) + " 0 R");
This made a difference in the context of this question; the first line, adding an indirect null object to the PDF, is necessary to make sure that in case of PDFs with object streams the object stream object number has already been determined and does not slip between the next objects, resulting in an off-by-one error for our prediction.
Beware: While this procedure creates something which does not violate the letter of the PDF specifications (which only forbid the cases where the same field object is referenced from multiple pages, be it via the same or via distinct widgets), it clearly does violate its intent, its spirit. Thus, this procedure might also become forbidden as part of a Corrigenda document for the specification.

Rules are skipped in KnowledgeBase

we are using drools 5.5 final version.we have thousands of objects and two rules so we are getting objects in chunk(100 size) wise and creating knowledge base for every chunk and firing rules.since creation of Knowledge Base is expensive we are getting performance issue.So we are creating Knowledge Base once and using that knowledge base for every chunk in this case after 4 to 5 chunks got executed from 6th chunk on wards rules are not getting fired though match is there .please suggest what can be done.
sample code
public static KnowledgeBase getPackageKnowledgeBase(PackageDescr pkg){
KnowledgeBuilderConfiguration builderConf = KnowledgeBuilderFactory.newKnowledgeBuilderConfiguration();
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder(builderConf);
kbuilder.add(ResourceFactory.newDescrResource(pkg), ResourceType.DESCR);
Collection<KnowledgePackage> kpkgs = kbuilder.getKnowledgePackages();
if(kbuilder.hasErrors()){
LOGGER.error(kbuilder.getErrors());
}
KnowledgePackage knowledgePackage = kpkgs.iterator().next();
KnowledgeBase kbase= KnowledgeBaseFactory.newKnowledgeBase();
kbase.addKnowledgePackages(Collections.singletonList(knowledgePackage));
return kbase;
}
using method
chunkSize=100;
int start = 0;
Count = -1;
KnowledgeBase kbase=getPackageKnowledgeBase(pkgdscr)//pkgdscr contails all rules got from db
while(Count!=0 && Count <= chunkSize ){
LOGGER.debug("Deduction not getting "+mappedCustomerId);
Objects inputObjects = handler.getPaginatedInputObjects(start);
Count = inputObjects.size();
start=start+chunkSize;
StatefulKnowledgeSession ksession = kbase.newStatefulKnowledgeSession();
for(Object object:inputObjects){
ksession.insert(object);
}
ksession.fireAllRules();
ksession.dispose();
}
Below is the essential part of your loop. Looks to me that this loop terminates as soon as Count exceeds chunkSize (100). You sure this never happens?
while(Count!=0 && Count <= chunkSize ){
Objects inputObjects = ...;
Count = inputObjects.size();
...
StatefulKnowledgeSession ksession = ...;
for(Object object:inputObjects){
ksession.insert(object);
}
ksession.fireAllRules();
...
}

Load the graph from the titan db for a specified depth in a single or efficitent query

We are using titan db to store the graph infomation. we have cassandra + es as backend storage and index. We are trying to load the graph data to represent the graph in the webui.
This is the approach i am following.
public JSONObject getGraph(long vertexId, final int depth) throws Exception {
JSONObject json = new JSONObject();
JSONArray vertices = new JSONArray();
JSONArray edges = new JSONArray();
final int currentDepth = 0;
TitanGraph graph = GraphFactory.getInstance().getGraph();
TitanTransaction tx = graph.newTransaction();
try {
GraphTraversalSource g = tx.traversal();
Vertex parent = graphDao.getVertex(vertexId);
loadGraph(g, parent, currentDepth + 1, depth, vertices, edges);
json.put("vertices", vertices);
json.put("edges", edges);
return json;
} catch (Throwable e) {
log.error(e.getMessage(), e);
if (tx != null) {
tx.rollback();
}
throw new Exception(e.getMessage(), e);
} finally {
if (tx != null) {
tx.close();
}
}
}
private void loadGraph(final GraphTraversalSource g, final Vertex vertex, final int currentDepth,
final int maxDepth, final JSONArray vertices, final JSONArray edges) throws Exception {
vertices.add(toJSONvertex));
List<Edge> edgeList = g.V(vertex.id()).outE().toList();
if (edgeList == null || edgeList.size() <= 0) {
return;
}
for (Edge edge : edgeList) {
Vertex child = edge.inVertex();
edges.add(Schema.toJSON(vertex, edge, child));
if (currentDepth < maxDepth) {
loadGraph(g, child, currentDepth + 1, maxDepth, vertices, edges);
}
}
}
But this is taking a bit lot of time for the depth 3 when we have more number of nodes exist in the tree it is taking about 1 min to load the data.
Please help me are there any better mechanisms to load the graph efficiently?
You might see better performance performing your full query across a single traversal execution - e.g., g.V(vertex.id()).outV().outV().outE() for depth 3 - but any vertices with very high edge cardinality are going to make this query slow no matter how you execute it.
To add to the answer by #Benjamin performing one traversal as opposed to many little ones which are constantly expanding will indeed be faster. Titan uses lazy loading so you should take advantage of that.
The next thing I would recommend is to also multithread each of your traversals and writes. Titan actually supports simultaneous writes very nicely. You can achieve this using Transactions.