Can Dataflow instances be deleted?
Can instances of DATA-DECISION-SERVICE-DATAFLOW be deleted (for e.g. Batch)?
We don't have any batch nodes configured and we don't use dataflows in our application. However although logging is set to error, the events of checking node availability still get printed in our JBOSS server logs, lines like :
2023-07-06 23:59:43,112 INFO [com.pega.dsm.dnode.impl.dataflow.service.leaderpulse.CheckNodeAvailabilityTask] (LeaderPulse-DataFlow-Pulse-Executor:17) Run waiting for resources [DF-9035] did not get resources assigned on service [Batch]
Our logs get so filled up with these line items, the size of server.log files sometimes go upto 800 mb.
I was therefore thinking if this could somehow be disabled, maybe by deleting the dataflow instance.
Pegasystems Inc.
GB
@SayoniM1 , you can manage data flow instances by using the Call instruction with several activities.
To delete a data flow run and associated statistics, use the Call Data-Decision-DDF-RunOptions.pxDeleteRunById method in an Activity rule.
This will help you manage unused data flow instances and prevent log files from filling up with unnecessary entries.
Also you can comment out the problematic logger to avoid seeing "com.pega.dsm.dnode.impl.dataflow.service.leaderpulse" INFO msgs in server.log.
Update the standalone-full.xml with below logger and restart the system.
<logger category="com.pega.dsm.dnode.impl.dataflow.service.leaderpulse"> <level name="ERROR" /> </logger>
This can be done if you are using a purely Case Management application and the Pega-DecisionEngine agent is running on all nodes, creating the DF cases which is not needed.
We recommended to disable the agent by opening each agent schedule and unchecking 'Enable this agent' button under Agent wide settings. This agent shouldn't be running in this case, irrespective of whether it is generating logs or not.
Scotiabank
CA
@MarijeSchillern Thanks a lot Marije! Could you please specify the name of the agent which needs to be disabled? I tried looking for the agent, but couldn't find out yet which agent is causing this.
Pegasystems Inc.
GB
@SayoniM1 for the client example I provided it was the ADMSnapshot agent specifically. However we asked them to disable the complete Agent Schedule as they weren’t using Adaptive models for their Case Management app.
Also the DF- cases generating the info logs can be deleted. These are instances of Pega-DM-DDF-Work class and can be deleted from DB backend.
Along with this, we can delete the DF- related instance from couple of other tables using the below format. You'd need to pass the exact DF- id:
Select * from pegadata.pc_work_dsm_batch where pyid like 'DF-%';
Select * from pr_data_decision_ddf_run_opts where pyworkobjectid like 'DF-%';
Select * from pr_data_decision_df_part where pxworkid like 'DF-%';
Select * from pr_data_decision_ddf_error where pxworkid like 'DF-%';
Select * from pr_data_decision_df_met where pxworkid like 'DF-%';
Select * from pr_log_dataflow_events where pyrunid like 'DF-%';
Select * from pr_data_decision_ddf_runtime where pyrunid like 'DF-%';
Can you let us know how you get on?
If I've answered your question could you mark 'Accept Solution' button against my previous reply?
Scotiabank
CA
@MarijeSchillern Thanks Marije, do give us a couple of days to observer how the fix works.