Field Level Auditing - Performance Issue
I have a requirement where I have implemented Field-Level auditing on 200+ properties. I have noticed significant performance impacts doing this (30-40% longer times with field level auditing turned on moving from screen to screen). Is there a different way to "batch" these requests OOTB of improve performance?
The client is on version 8.3. We are aware of the field-level auditing UI improvements in 8.4 and will be taking advantage of those next month. Are there also performance improvements for this in 8.4?
Any suggestions would be helpful.
Thanks
Pegasystems Inc.
AU
Auditing 200+ fields sounds like a response to "can we audit everything just in case?" ... the performance impact is the consequence of over-auditing. Challenge whether all 200+ of those fields really need to be audited.
The pre-8.4 pattern for Field Level Auditing is via a Declare Trigger that calls the TrackSecurityChanges activity. I'm looking at the 8.4 implementation of this activity, but the Java step here effectively loads the current (pre-change) instance to a separate clipboard page named "CopyOfInstance" and uses that as the basis for the comparison.
I've not seen it done, but in theory:
- Your Declare Trigger could run a new Activity that packages CopyOfInstance (pre-change), and the Primary page (post-change) into a Queue Processor job; then
- Your Queue Processor resurrects the Primary page and CopyOfInstance to the clipboard, and then calls TrackSecurityChanges. Perhaps it may give the same result?
You can't just make a Queue Processor around TrackSecurityChanges, as by the time the Queue Processor job runs the changed instance will have been saved. CopyOfInstance and the Primary page will therefore be the same and you won't see any audit output. The goal of the above is to stage a Queue Processor to work with the same data values asynchonously that the Declare Trigger would see synchronously.
Also note that by delegating auditing to a Queue Processor:
Auditing 200+ fields sounds like a response to "can we audit everything just in case?" ... the performance impact is the consequence of over-auditing. Challenge whether all 200+ of those fields really need to be audited.
The pre-8.4 pattern for Field Level Auditing is via a Declare Trigger that calls the TrackSecurityChanges activity. I'm looking at the 8.4 implementation of this activity, but the Java step here effectively loads the current (pre-change) instance to a separate clipboard page named "CopyOfInstance" and uses that as the basis for the comparison.
I've not seen it done, but in theory:
- Your Declare Trigger could run a new Activity that packages CopyOfInstance (pre-change), and the Primary page (post-change) into a Queue Processor job; then
- Your Queue Processor resurrects the Primary page and CopyOfInstance to the clipboard, and then calls TrackSecurityChanges. Perhaps it may give the same result?
You can't just make a Queue Processor around TrackSecurityChanges, as by the time the Queue Processor job runs the changed instance will have been saved. CopyOfInstance and the Primary page will therefore be the same and you won't see any audit output. The goal of the above is to stage a Queue Processor to work with the same data values asynchonously that the Declare Trigger would see synchronously.
Also note that by delegating auditing to a Queue Processor:
- The timestamp on the history entries would be up to 30 seconds after the actual save action. The value of auditing is diminished if you can't reliably correlate timestamps
- The user on the history entries will likely be the Queue Processor name rather than the human who made the changes. This also diminishes the value of your auditing.
Both may be actionable, but by the time you've engineered around these issues, you may have done a lot of customization even though you've wrapped the OTB TrackSecurityChanges.
So, again, challenge whether the scale of what is being audited is necessary. Perhaps also prove the concept of field level auditing 200+ fields on the newer versions of Pega using the new pattern, to observe any performance differences. If you need access to the latest version of Pega software to try it out, try the cloud-based Community Edition.
Soft Serve, Inc.
US
Thank you for the thorough response...
The use case for this is this...
They have a case that routes from a user to a manger, and they want that manager to be able to quickly see all fields that were changed/updated by the user. Field-level auditing was the easy way to implement this. So yes, they are monitoring everything because they need to quickly view all changes.
Any other suggestions to do this without custom Java code?
Pegasystems Inc.
AU
Hi @MattS629
Ah, so if its all changes then a challenge with Field Level Auditing is that as your data model evolves you need to remember to go and add the new elements to the Data Transform in Field Level Auditing.
I would recommend investigating the "Compare Pages" API (start with the Activity pxComparePages). This is the same capability that Pega uses for its "Rule compare" behavior of Activities and Data Transforms, and essentially creates a listing on the clipboard of the differences between two clipboard pages that are intended to be different snapshots of the same instance. Whilst rule compare compares two instances of a rule, it can be used on any type.
If you can get the pre-changed instances onto the clipboard, and 'compare' it with the post-changed instance you are about to save.
My understanding is that the new case type configured auditing in 8.4 (that you mentioned in your original post) now uses this API instead of the Trigger & Data Transform pattern. Beware of investing a bunch of engineering in 8.3 around this API when 8.4 out of the box may already do it for you. I've not used the 8.4 variant yet but this article on the latest field-level auditing capability in 8.5 suggests there is still an exercise in choosing the individual fields to audit. To "audit all" may still need your own solution around pxComparePages.