Is there a way to control either the number of items stored on a data page or its total size on the clipboard?
Hi,
we would like to use a parameterized node scope data page to act as a cache of frequently used static data read from the DB but also be in control of the number of items being put on the clipboard. Size of entries read from DB is about 2 - 6 MB each and their number may grow in the future. So in order to not have all memory used up by this data page we would like to at least restrict the number of items stored on it or ideally even check on the the actual size of the data page on the clipboard (given that this is not hitting performance too much).
Would it be possible to use "Post load processing" activity on the data page in order to have it check the current item count (or total clipboard size) on the data page after each load operation and if a given threshold is crossed, have, for example, the oldest item removed from it? Are there any concurrency considerations to be taken into account when trying to remove an entry this way? Has this been put in place already by someone in this way or another?
Kind regards,
Martin
***Updated by moderator: Lochan to add Categories***
Pegasystems
US
Keep the declare-on-change rule in mind for this sort of thing. It is capable of suspending operation of one flow and calling another flow if it detects some condition that may warrant investigation or approval. /Eric
Vodafone GmbH
DE
Hi Eric,
thanks a lot for your response. I am not sure if suspending a flow here would be the right approach. We are looking at a high throughput fully automated processing application that would make use of of this data page as a cache for large objects.
As I described in response to Phani's suggestion, I am able to identify the point in time at which a new entry is being loaded into the page by using the "Load Activity" field and I am also able to remove entries, but I still would need to be able to asses how many and which entries are present on the data page, i.e. get a listing of all available entries. I have tried to get these details using Java API (tools.findPage("D_MyDataPage") but this does not give me information about the contents of the page.
Any ideas?
Thanks a lot and kind regards,
Martin
Pegasystems Inc.
IN
Hi Martin, Good Morning!
>>> Is there a way to control either the number of items stored on a data page?
could you please refer DATA-ADMIN-SYSTEM-SETTINGS PEGA-RULES!DATAPAGES/MRUCAPACITY
the read only data pages that're accessed greater than 10 minutes ago will be automatically deleted, if the result set count reaches the number/capacity defined via this setting.
Thank you!
psahukaru
Vodafone GmbH
DE
Hi Phani,
thanks a lot for your response. This is indeed a useful setting which I was not aware of. However, what we are looking for is a more fine grained control over a single dedicated Data Page.
For one thing, we would need to have this removal strategy applied only to a single Data page and not all data pages. Also, we are looking at a high performance application in which it is very likely that all entries will be accessed within 10 minutes, so we would need to be able to set a retention period that is lower than 10 minutes for this particular data page only.
Meanwhile, I was checking on the "Post loading activity" approach: I have written and registered an activity which is run whenever a new entry is being loaded onto the data page and that will remove an entry from it via Page-Remove D_MyDataPage["MyEntryKey"]. This seems to be working fine, even for node scope data pages getting accessed by multiple threads in parallel.
The only problem now is: How can I determine which entries are present on the Data page in order to decide which one to remove? So far I have not been able to get a listing of all entries on the data page or a listing of all available keys. Without knowing, which keys are present, I cannot sensibly remove an entry.
Any ideas?
Thanks a lot and kind regards,
Martin
Pegasystems Inc.
IN
Hi Martion, Good Morning!
>>> "Post loading activity" approach
yes, the activity configured at 'post load activity' field runs on the data page result set as an aftermath operation.
>>> How can I determine which entries are present on the Data page in order to decide which one to remove? So far I have not been able to get a listing of all entries on the data page or a listing of all available keys. Without knowing, which keys are present, I cannot sensibly remove an entry.
- I haven't tried but could you please share your observations with a precondition ' @(Pega-RULES:Page).PageExistsWithClass(D_MyDataPage["MyEntryKey"], tools)' to remove the page.
Thank you!
psahukaru
P.S: Re: How can i flush all datapage instance without passing any parameter ? (a contrast discussion)
Vodafone GmbH
DE
Hi Phani,
thanks a lot for your input.
I have tried getting hold of an entry on the data page via @(Pega-RULES:Page).PageExistsWithClass(D_MyDataPage["MyEntryKey"]) but to no avail as I always get FALSE returned.
However, running it without parameter/key returns TRUE@(Pega-RULES:Page).PageExistsWithClass(D_MyDataPage).
I also tried @PageExists(D_MyDataPage["MyEntryKey"] ) but with the same effect.
I am afraid I will have to give up on this idea.
Kind regards,
Martin
Pegasystems Inc.
IN
>>> I have tried getting hold of an entry on the data page via @(Pega-RULES:Page).PageExistsWithClass(D_MyDataPage["MyEntryKey"]) but to no avail as I always get FALSE returned.
- does the page exist? sorry to ask
- is the syntax correct while removing the page? (D_MyDataPage[MyEntryKey:Param.MyEntryKey]) or D_MyDataPage[Param.MyEntryKey]
Can we try directly doing a page-remove without precondition?
Page-Remove: D_MyDataPage[MyEntryKey:Param.MyEntryKey]
Is D_MyDataPage is of strucutre Page or Page list?
Please share your thoughts/observations, Thank you!
psahukaru
Vodafone GmbH
DE
Hi Phani,
I ran the same tests with my Data Page and with page OperatorID. The latter is working fine, Data Page is not working, so syntax is correct. Alos, page is present as I can access it and print some values like pzInskeys of the loaded entries from it.
Directly removing a hard coded works as I have done via Page-Remove in the “Post load activity” but this is not the point: I would need to get some details on the DataPage: What’s the number of loaded keys or alternatively what is their combined size in order to determine if it is time to remove one or more entries. Which keys have been loaded in order to decide which one to remove.
Kind regards,
Martin
Pegasystems
US
>>> I have tried to get these details using Java API (tools.findPage("D_MyDataPage") but this does not give me information about the contents of the page.
That findPage will return you an object of type ClipboardPage. Start from your developer portal and look at "help - > APIs - > Engine". On left margin, click on PublicAPI. On the page that comes up, you can see the "findPage" you used. Then click on "ClipboardPage" next to the findPage writeup. This will bring you to the "ClipboardPage writeup which will tell you all the things you can call once you have your ClipboarPage object. See if any of those lead you to some of the info you are looking for. /Eric
Vodafone GmbH
DE
Hi Eric,
sorry for the late response but I was on a much needed holiday :-)
Looking at the Public API and methods of ClipboardPage it does not seem to give me access to what I need. I have been playing around with
tools.findPage("D_MyDataPage")
which is returning an object of type Clipboard page and also had the XML representation of this object written to the log, just to see, what kind of properties/embedded pages etc. are present on it.
Result is, that there does not seem to be any reference to the actual entries of this data page. It seems, as if DataPages LOOK like they contain a page group of embedded pages/entries inside of them but technically/under the hood they are a different kind of thing.
So I am afraid, there is no way to inspect DataPages using Engine API (at least not using the classes I am aware of).
Kind regards,
Martin