Question
CBA (Commonwealth Bank of Australia)
IN
Last activity: 6 Aug 2024 8:41 EDT
File Ingestion for multiple files from a manifest file
Team,
Greetings!
Is there a way of ingesting multiple files in Pega for customer data ingestion process.
1 manifest file with more than 1 data files present in it, without any performance issues.
Any suggestions would be useful!
Thanks!
Pegasystems Inc.
IN
Hi @Priyesh, please take a look at the Pega docs https://docs.pega.com/bundle/customer-decision-hub/page/customer-decision-hub/implement/batch-import-data-jobs.html Additionally, you can refer the Pega academy as well.
CBA (Commonwealth Bank of Australia)
IN
We already have an Ingestion Case Type in place.
Where we are using "use a file path to export/import data" in the source file dataset to read 1 file at a time.
Now, we wanted to use "use a manifest file to import data"
Is it good to use "use a manifest file to import data"? keeping performance in mind. will it club all the data from say 2 files and send to the table?
Also, Batch Data jobs look's similar to our manual case type. In articles related to Batch Data jobs, I couldn't find any module course or challenge to use the manifest file for multiple files.
how should we proceed to achieve the requirement?
Thanks! Priyesh A
CBA (Commonwealth Bank of Australia)
IN
@achas1 tried with "use a manifest file to import data" in file data set, but it's not able to parse the file's.
Getting error:
com.pega.platform.integrationengine.repository.RepositoryException: There was an issue creating stream for the file.
CBA (Commonwealth Bank of Australia)
IN
After looking into the error, I think the data file has to be present in the same folder.
Do we have any other approach to accept multiple files?
Thanks! Priyesh A
Pegasystems Inc.
GB
Pega supports ingesting multiple files for the customer data ingestion process. However, each data file should have its own manifest file. The manifest file contains metadata about the transferred data files and is used for record and file validations. If you're getting an error while creating a stream for the file, it might be due to an issue with the file path or the file itself. Please ensure that the file path in the manifest file is relative to the root path defined in the repository. Also, check the file for any issues that might prevent it from being parsed correctly.
⚠ This is a GenAI-powered tool. All generated answers require validation against the provided references.
Setting up data ingestion > 3 Create the AccountFile data set
Batch import data jobs > Creating a new batch import data job
Use Case - Setting up data ingestion > 1 Other details
If you need further help please log a support INC via the MSP and provide the INC id here so that we can help track it.
CBA (Commonwealth Bank of Australia)
IN
The above approaches mentioned seems to use Batch Data Import Job, where as we have our own ingestion process defined.
Created Data Jobs will be a huge task from scratch and all the existing rules will have to be withdrawn.
Please suggest any other approach which we can incorporate in our exiting logic.
Thanks!
Priyesh A