Skip to main content

Support Doc

 Preethi Kumarswamy (Preethi_K)
PEGA
Principal Knowledge Management Specialist
Pegasystems Inc.
IN
Preethi_K Member since 2018 34 posts
PEGA
Posted: Jul 5, 2024
Last activity: Sep 3, 2024
Posted: 5 Jul 2024 4:10 EDT
Last activity: 3 Sep 2024 4:26 EDT

Implementing high performance and more resilient case archival method – InMemory Pipeline archival [SDR-B32]

Understand the new InMemory Pipeline archival process and the enhancements it brings with it. Learn to implement the InMemory Pipeline mode of archival to eliminate archival performance issues. The scenarios describe the issues with case archival that users have reported.

InMemory Pipeline archival

Implementing InMemory Pipeline archival

Diagnosing and debugging Archival issues

Scenario 1: Crawler and Copier tasks run in asynchronous mode

Scenario 2: Out-of-the-box Archive job fails to archive cases

Scenario 3: Performance degradation of Indexer job while archiving

Scenario 4: Archival and Purging Indexer jobs run in loop 

Scenario 5: Purge and Archive jobs fail

Scenario 6: Archival job fails to pick cases

Scenario 7: Pega Archive fails 

 

 

InMemory Pipeline archival

This new archival process executes all stages of the traditional archival process within a single job, pyPegaArchiverUsingPipeline. This job triggers the pzPerformArchiveUsingPipeline activity. An important parameter in this activity is pipelineduration. This parameter allows users to specify the duration (in minutes) for the archival pipeline. 

Note: The Archival code is redesigned to minimize the updates to the pr_metadata table to enhance performance and resiliency and simplify the adoption of the archival process.

Enhancements

InMemory Pipeline archival process brings with it the following enhancements:

  • Simpler setup
  • Improved resiliency
  • Maximum performance

Simpler setup

  • User needs to schedule a single job, pyPegaArchiverUsingPipeline, to run regularly and specify the duration to run the process considering the low-load time-window.
  • A single run executes multiple cycles of 4 steps (Crawler, Copier, Indexer, and Purger) internally.
  • Setting Buffer time between jobs is not required since there is only one job.
  • Archival process ends before the expected End time that is specified by pipelineduration.

Improved resiliency

  • Defragmentation and DB connections of pr_metadata are reduced significantly as pipeline is InMemory.
  • The pr_metadata table is truncated at the end of each cycle.
  • If a failed archival process is reattempted a few times and if the reattempts do not help, job exits to avoid a runaway scenario and creates PDC alert for user to address the underlying issue. This is because most of the archival failures are caused by environmental issues and reattempts may not help in such scenarios.
  • Despite all the safety mechanism/measures, if there is a runaway scenario where archival keeps running due to some unknown issue, user can stop the archival process easily without restarting the Util node.

Maximum performance

  • Most of access to the pr_metadata table is replaced by InMemory Pipeline for higher performance.

Easier Adoption

  • User must adjust only maxCrawlerRequestors and maxCopierRequestors.
  • User is not expected to tune the following parameters:
    • limitPerPolicy
    • indexerQueryLimit
    • purgeQueryLimit
    • maxIndexerRequestors
    • maxPurgerRequestors

Implementing InMemory Pipeline archival

  1. Perform the following steps:
    1. Disable pyPegaArchiver, pyPegaIndexer, and pyPegaPurger jobs
    2. Set dataarchival/batchPipelineEnabled to True
    3. Enable pyPegaArchiverUsingPipeline
    4. Specify Start Time and pipeline duration in minutes. The Pipeline job archives as many cases as possible during the specified duration.
  2. For higher performance, adjust the following parameters:
    1. MaxCrawlerRequestors 
    2. MaxCopierRequestors
  3. If you have deployed synchronous Crawler+Copier hotfix (HFix-85147) , then perform the following steps:
    1. Work with the Pega Cloud support team to enable auto_vacuum for pr_metadata.
    2. Disable Vacuum freeze for pr_metadata.
Note: To prevent the PEGA0130 long running job alerts from occurring, adjust the Long running job scheduler threshold timing. The maximum limit for the value of the Long running job scheduler threshold timing is one hour. If the job run requires more than an hour, then safely ignore the alert.

Compatibility

InMemory Pipeline archival process is compatible with the following Pega Platform versions:

  • Pega 8.7.3
  • Pega 8.7.6
  • Pega 8.8.3
  • Pega Infinity ‘23
Note: If your database is not supporting the InMemory Pipeline archival performance, consult Pega for support by creating a Support Ticket. For information related to creating a Support Ticket, refer My Support portal FAQ

 

Diagnosing and debugging Archival issues

Scenario 1: Crawler and Copier tasks run in asynchronous mode

Crawler and Copier tasks in the pzPerformArchiver job scheduler run in asynchronous mode. Hence, there is a requirement to set some buffer time while defining job interval so that no overlap happens. This results in loss of time leading to less throughput. 

Errors

Not Applicable

Explanation

This issue was determined to be a product enhancement request.

Environment

Pega Platform™ 8.7.3 in Pega Cloud® services 2.24

Solution

Apply HFix-85147.

Implement the InMemory Pipeline archival method.

Scenario 2: Out-of-the-box Archive job fails to archive cases

Out-of-the-box Archive job fails to archive cases despite the cases are resolved. 100% success message for archival displays though the cases are not archived.

Errors

Not Applicable

Explanation

Archival is happening as expected. Errors occur due to unresolved child cases.

Environment

Pega Platform™ 8.7.2 in Pega Cloud® services 2.27.1

Solution

Resolve the child cases and set their status to Resolved.

Scenario 3: Performance degradation of Indexer job while archiving

Performance degradation of Indexer job while archiving and purging when value of maxIndexerRequestors is set to 1.

Errors

Not Applicable

Explanation

Forward chaining gets enabled during the Archival Indexing process and causes the delay.

Environment

Pega Platform™ 8.7.3 in Pega Cloud® services 2.24

Solution

Apply HFix-85094.

Scenario 4: Archival and Purging Indexer jobs run in loop

Archival and Purging Indexer jobs run in loop when indexing failed entries get added. This causes a pause in the Archival and Purge activities.

Errors

Not Applicable

Explanation

A defect in Pegasystems’ code or rules.

Environment

Pega Platform™ 8.7.3 in Pega Cloud® services 2.24

Solution

Apply HFix-85146.

Alternatively, perform the following local-change:

  1. Disable the job
  2. Manually execute the indexer

Scenario 5: Purge and Archive jobs fail

Purging and Archiving fail when user archives old/outdated data with the Pega recommended configuration.

Errors

Not Applicable

Explanation

This issue was determined to be a product enhancement request.

Environment

Pega Platform™ 8.3.6 in Pega Cloud® services 2.20.9

Solution

This issue is addressed in Pega 8.7.

Scenario 6: Archival job fails to pick cases

Archival job does not pick cases.

Errors

Not Applicable

Explanation

Data-Retention-Policy does not have the updated Archival settings.

Environment

Pega Platform™ 8.7.3 in Pega Cloud® services 2.24.4

Solution

Perform the following local-change:

  1. Delete the Data-Retention-Policy instance for a particular case for which Archival is enabled.
  2. Create a Data-Retention-Policy instance with the updated Archival settings.

Scenario 7: Pega Archive fails

Unable to archive data using Pega Archive.

Errors

Not Applicable

Explanation

This issue was determined to be a product enhancement request.

Environment

Pega Platform™ 8.7.1

Solution

An enhancement request, FDBK-110971, is created for consideration by Pega Product Management.

Remember: Case Archival is not supported for DB2 database. For more information, refer the Archiving and expunging case data article. Case archival is available for on-premises and Pega Cloud clients for the following databases:
  • PostgreSQL
  • Oracle
  • MS SQL
An enhancement request is created for support for IBM DB2 during case archival.
To see attachments, please log in.
Pega Platform 8.8.3
Pega Platform 8.7.6
Pega Platform 8.7.3
Case Management
How To
Troubleshooting

Did you find this content helpful?
Yes
No

100% found this content useful
  • Reply
  • Tyler Head Simran Singh

Related content:

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice