@FromRajesh below some suggestions .  I am not an SME in stream / kafka issues but this type of issue is likely because queries hitting the PR_DATA_QP_RUN_PARTITION table were caused by the increased processing from the pyFTSIncrementalIndexer.

When it comes to the performance of Queue processor( a background processing in Pega), there are two aspects:- 

1. Message processing time:-

The time to process a message depends on the amount of work done by the processing activity. It will be helpful to check the performance of the QP activity 

and identify any bottleneck under load testing.

Also please refer to the Best practices for writing activities for background jobs

2. Total message throughput :-

Scale out by increasing the number of processing nodes. If the CPU usage is well below maximum,  you do not need to scale nodes right away.

Check threads per node and  total number of threads   is matching with the Kafka partition count. 

Increasing the Kafka partition is not a very straight forward recommendation as it can have other implications. 

Increasing the partitions should also be accompanied by increasing the thread counts per background nodes. Higher number of threads can cause higher CPU usage and hence processing slowness.

Moreover, the DSS setting to increase partition affects any queue processor that you are going to create. Thus there can be impact on the Kafka side itself.

So we do not recommend increasing partition count increase.

Furthermore, increasing the partition count will not help in the situation were the activity itself faces bottleneck. Once the record is being from Kafka and pushed to processing to the activity 

Next Step :-  (****not for prod environments****)

If you have seen the processing time even goes up then it is the activity that took so long.

• You should see PEGA0117 alerts in PDC. 

Configure default partition count via Dynamic System Setting:

• Owning Ruleset = Pega-Engine
• Setting Purpose = prconfig/dsm/services/stream/pyTopicPartitionsCount/default
• Value = N [ N = No. of partitions]
• Write down the list of all queue processors
• Stop all queue processors
• Stop all pega nodes

Check ERROR logs  

Check DSS timeout for the cluster node connection management in master election and admin operation at elasticsearch end.

Ruleset : Pega-SearchEngine
Purpose : indexing/distributed/cluster_management_timeout_millis
 

Reset queue processor partitions by running the following statement

• delete from pr_data_decision_df_part where pxWorkID in ('<qp-name-1>', '<qp-name-2>, .., '<qp-name-134>'')

Where <qp-name-x> corresponds to the list of processors.

Truncate the Stream service tables in the Pega Platform

• truncate table pr_data_stream_nodes
• truncate table pr_data_stream_sessions
• truncate table pr_data_stream_node_updates

delete relevant records from pr_data_qp_run_partition.

• delete from pr_data_qp_run_partition where pxqueueprocessorname in (‘<QP1>’….’<QPn>’)

Delete kafka directories from all stream nodes

• Start stream nodes
• Start the rest of the pega nodes
• Restart queue processors if not started. Verify partition count on the stream landing page.
• Pick one queue processors and verify that it uses the configured number of partitions:
• Open associated data flow
• On the distribution tab check number of partitions. It should match the configured number.

If you need further investigation I suggest that you log a support INC incident on the MSP.  In the ticket please provide following details:

1. What is the system name that can be checked from PDC.
2. Share the name of the custom queue processor.
3. Share more metrics from the activity processing side, like if any step is taking longer, etc.
4. Let us know the timestamp of your next load test which we can verify from PDC