Question
Highmark Health
Last activity: 1 May 2025 9:31 EDT
How to parameterize the kafka topic name in Data set?
How can I parameterize the Kafka topic name in dataset?
We have a situation where our Kafka cluster is shared across QA and UAT environments. To prevent QA data from landing in UAT (and vice versa), planning to append the environment name as a suffix to each Kafka topic name. Using the same topic name across environments would result in data corruption.
For example, our Kafka topic names follow this pattern:
Insurance-Health-Medical-QA1Insurance-Health-Medical-QA2Insurance-Health-Medical-UAT1Insurance-Health-Medical-UAT2
Thanks,
Sudheer.
Accepted Solution
Highmark Health
@Sudheer Nimmagadda yes, we can use 'application settings' to parameterize it.
HCA Healthcare
US
@Sudheer Nimmagadda To parameterize the Kafka topic name in your dataset and avoid hardcoding environment-specific values, you can use environment variables or configuration parameters within your data pipeline or ingestion framework (e.g., Spark, NiFi, or a custom app). Define a parameter like env.name=QA1 or env.name=UAT2 in your configuration file or environment setup. Then, construct the topic name dynamically by using string interpolation, such as Insurance-Health-Medical-${env.name}. This way, the dataset will always send or receive messages to the correct environment-specific Kafka topic. This approach prevents cross-environment data mixing and makes deployments cleaner across QA and UAT environments.
Please let me know if you need any more info. Thanks
Accepted Solution
Highmark Health
@Sudheer Nimmagadda yes, we can use 'application settings' to parameterize it.