Pega has identified an issue with the czThumbnail property, as such we are proactively notifying clients since this issue could lead to the DB increasing in size unnecessarily.

Problem Summary

Pega Customer Service (CS) 8.7. and above, displays attachments in the live chat and stores the attachment stream in the czThumbnail property.

Clients configure the czThumbnail property to store "thumbnails" (reduced-size versions of pictures or videos) in the external S3 repository. However, the property is storing them in their primary database instead. This causes increased DB utilization for chat and email interaction attachments and performance issues.

This issue occurs with clients whose databases are at least 300 GB (or more) in size.
 

Problem Remediation

A utility (see attachments) is available for impacted clients, which cleans up the existing DB instances. The utility opens attachments, removes the czThumbnail property, if present, and saves the attachment.  For more information about the utility please refer to the Utility steps below.

The remediation for this issue is addressed in the product in Pega platform versions 24.1, 23.1.2, and 8.8.5.

In addition, we have created hotfixes for various platform versions that remove the code that stores czThumbnail in the database.

A restart will be needed for the hotfix changes to apply.

Remediation Matrix

As always, be sure you have appropriate backups in place before applying the hotfixes. 

Utility steps to clean up the Data-Work-Attach-File instances with czThumbnail

• Import the branch Thumbnailcleanup and product file ThumbnailCleanNRR.
• Add Thumbnailcleanup to your chat application. Make sure the application is available in system runtime context.
• Make sure CountOfUpdatedDWAF DSS value is 0. Once the utility is started this DSS holds count of records for which records are cleaned.

• Enable CLEANUPDWAF Log category to Info. Please update pyConfiguredResetTime property as required to make log category enable for more than 24hrs. The value is in hours. If the utility is stuck or any records are not cleaned up with error, we are logging the details. Also, we are logging the count of records that are cleaned for each Job scheduler run.
• Update the start date and end date in FetchAttachmentsWithTN RD. Start end is the date when czThumbnail is introduced which is on April 5th, 2022 or you can update the start date from April 5th to date when your system is above 8.7.1 version. End date is the date when the Hotfix/Local change to remove the logic to store czThumbnail in DB is introduced in production system.

• For now, in our system every Queue processor will run in 6 partitions. To increase the parallel processing please run pxAlterStreamPartitions activity with param.partitions = 20, param. Stream = CZCLEANUPDATAWORKATTACHFILE (In ALL CAPS)
• Before starting the utility, please run CleanedUpDWAF RD and check if the records with pyLabel is empty in the system. If the system has any records in the given date range. Please update RD as below and run the JS till the above records are cleaned. This has to be done in start or end of the utility.

• Enable czFetchAttachmentsJob scheduler. Currently it runs for every 30 mins which pushes 500 records on each run. Please adjust the time in your system as per the size and time it takes in respective system by observing few runs. Also for now it pushes 500 records you can reduce the number if the system is slow.

• If CleanedUpDWAF RD is taking lot of time. Please comment the steps which execute this RD and code to save it into DSS and update jump conditions in czFetchAttachmentsTo Clean activity. As DSS will not be updated please follow below steps to monitor the utility.

Please monitor the system for every 1-2 hrs.

• Run the CleanedUpDWAF RD for every 1-2 hrs and see if the count of cleaned up records is incrementing or not.
• If not it means the current batch of records are stuck and please follow above steps to update it to next batch. If you see the counter is not incrementing .Please follow below steps to update the batch.
  • Please run FetchattchmentsWithTN RD .
  • Get the date of the last record in the batch.
  • Please add this date as the start date in FetchattchmentsWithTN

Please monitor the logs for the logs that are printed in the utility.