Best Practice to Move from BIX Batch Extracts to Kafka-Based Streaming in Pega
We currently have BIX-based data extraction implemented in our Pega application and are exploring the possibility of introducing Kafka for near real-time data streaming. I would like guidance on the recommended approach and best practices.
Current Architecture
-
We have 52 BIX extract rules, each created for a different class
-
Data extraction runs daily using a Job Scheduler
-
The Job Scheduler invokes an OOTB BIX extraction activity
-
Extracted data is written as files
-
Files are then moved to an external FTP location (Sterling) using:
-
File Listener
-
FTP Server configuration
-
-
This is currently a batch-oriented process
New Requirement
We are planning to introduce Kafka to support data streaming / near real-time integration instead of (or along with) the existing batch file-based approach.
Virtusa-Polaris
IN
Additional questions:
-
Is Kafka-based streaming helpful for my use case?
-
What is the recommended Pega approach to publish data to Kafka?
-
Using Kafka Connect / Pega Stream Service
-
Using Queue Processors
-
Using Data Flows
-
Using Real-time event publishing (case processing hooks)
-
-
Should we:
-
Replace the existing BIX extracts completely?
-
Or keep BIX for batch use cases and introduce Kafka separately for streaming?
-
-
How can we efficiently handle multiple classes (52 classes) when publishing data to Kafka?
-
Are there any OOTB integrations or best practices for Kafka in recent Pega versions?
-
What are the performance and scalability considerations when moving from file-based BIX extracts to Kafka?