Ingesting files in utf-16 format
we have successfully used a service package, service file, file listener, file dataSet, Activities ... to ingest data files in the pega cloud, and also ingested files on a timed basis, directly using an activity and file dataSets.
BUT ... we are now being sent a file in UTF-16 format and need to convert it at ingestion time to UTF-8 for processing, or be able to cleanly read UTF-16 to a file DataSet ...
I have spent some time using the Embed-Repository-File DataPages, and looked at the underlying Amazon S3 Activities that are used, and am pretty sure using java steps that we could just use byte arrays in java and do the conversion by creating a new file in the Repository. but that just seems ugly.
I can plainly see on a Service file that you can specify UTF-16 in the Data Description.
so ... the question is Does Pega have a functional model to handle the ingestion of data in UTF-16 format?
NOTE: the supplier of the files in UTF-16 to the Pega Cloud SFTP location will only deliver the Data Files, no manifest, no token control file. so the example process for File ingestion does not work in this case.
any suggestions please.