File Listener throwing messages while reading file in Azure
I am trying to access a file placed in the Azure Blob Storage by configuring a File Listener in Pega.
However when the file is placed in the Azure Blob Storage, the file listener is able to monitor the file location but does not start the processing.
Upon investigating the logs, I was able to find these messages in the logs especially
Skipping [null], it is a directory.
Can someone please help to identify this issue.
Thanks,
Rajesh
***Edited by Moderator Marije to add Product version***
***Edited by Moderator Marije to add Support Ticket INC-271536 created 4 May Closed 9 June Added BUG ids******
Accepted Solution
Updated: 24 Aug 2023 7:57 EDT
Pegasystems Inc.
GB
@RajeshS56 I notice that you logged a support ticket 4th May ('File Listener not able to process a large file in Azure Repo'). It has been closed with the following details:
Problem description: We are using a File Repository and a File Listener to process the file placed in the Azure Blob Storage. The File Listener and Service activity are working fine. However when a large file is placed in the Azure Blob, file listener is not able to process it and says no files to process though the file is present in the Azure Blob.
This issue is not happening when we place a smaller file in the Azure to test.
Root cause: Product bug
Outcome: File Listener not able to process a large file in Azure Repo (BUG-805588,BUG-805589)
Notes: Issue : File directory watcher unable to identify files with empty Content-MD5 Property.
Workaround description:
@RajeshS56 I notice that you logged a support ticket 4th May ('File Listener not able to process a large file in Azure Repo'). It has been closed with the following details:
Problem description: We are using a File Repository and a File Listener to process the file placed in the Azure Blob Storage. The File Listener and Service activity are working fine. However when a large file is placed in the Azure Blob, file listener is not able to process it and says no files to process though the file is present in the Azure Blob.
This issue is not happening when we place a smaller file in the Azure to test.
Root cause: Product bug
Outcome: File Listener not able to process a large file in Azure Repo (BUG-805588,BUG-805589)
Notes: Issue : File directory watcher unable to identify files with empty Content-MD5 Property.
Workaround description:
- For manually uploaded files, manually populate the content-MD5 property and try to reprocess.
- For files uploaded by service API, Use updated SDK to upload file, and make sure content-MD5 is set. See ref:https://learn.microsoft.com/en-us/answers/questions/282572/md5-hash-calculation-for-large-files
BUG-805589 and BUG-805588 are due to be fixed in Pega 8.7.6 and 8.8.4 respectively.
@Deepthi the above may help you.
Updated: 6 Apr 2023 6:34 EDT
Pegasystems Inc.
GB
@RajeshS56 please provide details of your Pega version.
Are you following steps from Configuring a Microsoft Azure repository?
Pega platform does not support Azure Data Lake Storage type although some use it for repository operations.
Note that ADLS/File listener support was added in later Pega releases.
Azure Data Lake Storage type support has been added for file listener usecases.
Please consult the relevant Resolved Issues documentation for the following versions:
8.6.4: ISSUE-711001
8.7: ISSUE-688454,
8.7.2: ISSUE-711002
8.8: ISSUE-711003
BUPA Australia Pty Ltd
AU
Hi @MarijeSchillern ,
Thanks for the response.
1. We are using Pega 8.7.2 version and using a Repository rule in Pega to connect to the Azure Blob storage.
2. Configured the File Listener to monitor the container location in the Azure Blob storage. The test connectivity is working fine but not sure why the file is not being picked by the File listener.
3. All other configurations are good. For now I have configured the Listener to run on one node.
Would you please take a look at the document attached.
HCL
NL
@RajeshS56 Hi Rajesh,
Could you please share the details if the issue is resolved? We do face same issue in 8.7.5 and would like to know how it is resolved.
Appreciate your help!
Cheers,
Deepthi
Accepted Solution
Updated: 24 Aug 2023 7:57 EDT
Pegasystems Inc.
GB
@RajeshS56 I notice that you logged a support ticket 4th May ('File Listener not able to process a large file in Azure Repo'). It has been closed with the following details:
Problem description: We are using a File Repository and a File Listener to process the file placed in the Azure Blob Storage. The File Listener and Service activity are working fine. However when a large file is placed in the Azure Blob, file listener is not able to process it and says no files to process though the file is present in the Azure Blob.
This issue is not happening when we place a smaller file in the Azure to test.
Root cause: Product bug
Outcome: File Listener not able to process a large file in Azure Repo (BUG-805588,BUG-805589)
Notes: Issue : File directory watcher unable to identify files with empty Content-MD5 Property.
Workaround description:
@RajeshS56 I notice that you logged a support ticket 4th May ('File Listener not able to process a large file in Azure Repo'). It has been closed with the following details:
Problem description: We are using a File Repository and a File Listener to process the file placed in the Azure Blob Storage. The File Listener and Service activity are working fine. However when a large file is placed in the Azure Blob, file listener is not able to process it and says no files to process though the file is present in the Azure Blob.
This issue is not happening when we place a smaller file in the Azure to test.
Root cause: Product bug
Outcome: File Listener not able to process a large file in Azure Repo (BUG-805588,BUG-805589)
Notes: Issue : File directory watcher unable to identify files with empty Content-MD5 Property.
Workaround description:
- For manually uploaded files, manually populate the content-MD5 property and try to reprocess.
- For files uploaded by service API, Use updated SDK to upload file, and make sure content-MD5 is set. See ref:https://learn.microsoft.com/en-us/answers/questions/282572/md5-hash-calculation-for-large-files
BUG-805589 and BUG-805588 are due to be fixed in Pega 8.7.6 and 8.8.4 respectively.
@Deepthi the above may help you.
HCL
NL
@MarijeSchillern Yes,It helped Indeed.
I did notice MD5 is missing and was uploading the files manually as a workaround.
I will follow up with the support team for the H-Fix.
Thanks again for the explanation and confirmation.
Cheers,
Deepthi
Updated: 29 Aug 2023 4:38 EDT
Pegasystems Inc.
GB
@Deepthi I'm glad it helped.
Please note, however, that I did not mention any hotfix therefore there is no need to contact our GCS department.
You will be able to download 8.7.6 Patch yourself once it becomes available. You can access that using My Software as described in The New Software Download Experience in My Pega is Live!
For details on the update process see the below:
Pega Software maintenance program.
Please keep an eye on the pega-infinity-patch-calendar page.
You can also carry out a PSC search using the tags 'Patch Release'