Question
Pegasystems Inc.
AU
Last activity: 9 Feb 2018 16:52 EST
When does a BIX extract sequence number reset?
We are doing our first tests of BIX 7.3.1 on Pega Cloud. At the moment we are running a CSV extract from Designer Studio by running the pxExtractDataWithArgs activity. The CSV files are packaged in a ZIP file and available via SFTP. The ZIP files are named according to the convention:
BIX_[AppliesToClassOfExtractRule]_[ExtractRuleName]_[SequenceNumber].zip
... where [SequenceNumber] appears to be intended to represent the n'th time that Extract rule has been run, so as to assure uniqueness of the output.
We've run a few sets of extracts on each of three different days, and have noticed that the sequence number resets back to 1 on each day. My guess at the moment is:
* Sequence numbers may be reset each day; AND/OR
* Sequence numbers may be reset when the environment is restarted
... either of which would be an explanation.
The data in the CSV files is not impacted as the pxExtractDateTime field is sufficiently unique. Resetting sequence numbers however is a bad outcome as the filename of the ZIP (and the files it contains) may be duplicated with one produced before the 'reset' event but not yet consumed by the downstream ETL. This would result in data loss as the downstream ETL has no visibility of the file set that was overwritten.
I have workarounds in mind, but would like confirmation of the expected behaviour of the sequence numbering.
***Edited by Moderator Marissa to add SR Details***
Pegasystems Inc.
US
It looks like the sequence number is set by the number of extractions that have run on the current date that the extraction is running on. So if you want to be sure of unique filenames when running the same extraction multiple times a day you should use a combination of either the %d or %t wildcard with the %s. If you just use %s then you can have file1, file2, etc. today and then these same filenames tomorrow.
Pegasystems Inc.
AU
Thanks for your suggestion, Celeste.
The user guide suggests %d and %t are filename options on the File Specification tab for XML output. We are using CSV output on Pega Cloud, which has the following constraints (according to the User Guide):
1. File Specification tab for CSV extracts only prompts for Output Directory (with no wildcarding options).
2. File Specification tab is hidden when running on Pega Cloud, so there is seemingly no configuration control over the filename format.
Looks like I will be heading down the path of having an Activity wrap around pxExtractDataWithArgs where I can derive a genuinely-unique value to provide with the -k argument, and using this Activity for our Agent. I'm assuming for now the -k value will filter through to CSV/ZIP filenames.