Question
Infosys
AU
Last activity: 12 Jan 2023 16:49 EST
History table extract
We are planning to extract History table records to external system for data analytics and all the required columns are exposed in the table.
1. Solution for consideration is BIX - However, volumes are quite high - 47 million for last 3 years. we might have to run extract rule in smaller chunks to accommodate the memory/performance. Since, all columns are exposed, considering another solution which is a direct export from DB itself.
Which one could be a better solution and any other suggestions?
2. We have another problem with description of history messages which get stored in "pyMessageKey" column and it has field value stored in DB rather than the actual description. If we follow(from point 1) either BIX or direct extract from DB solution, how to transform these fieldvalues into actual description. Don't think BIX can do this transformation since it is not complete ETL tool. Any thoughts?