S3 Connection appears to disconnect.
Hi,
We see our S3 connection disconnecting intermittently. I suspect some sort of timeout. See message below. I'm not sure if there is some kind of stale connection or if there is something we can do on the application side. We can go into the application and resume the flow and it will work. I'm not clear as to what is causing the disconnect. Is this the app or an AWS type of issue.
Let me know if there are any ideas.
Thanks
2023-12-11 06:00:47,513 [ BlobWriter-0] [ ] [ ] [ ] ( aws.s3.AWSBlobWriter) ERROR - Exception occurred while uploading file (), in bucket (prod-pega-files) java.io.IOException: Pipe not connected at java.io.PipedInputStream.read(PipedInputStream.java:305) ~[?:?] at java.io.PipedInputStream.read(PipedInputStream.java:377) ~[?:?] at com.pega.platform.physicaldatalayer.internal.cloud.aws.s3.AWSBlobWriter.readAndUploadParts(AWSBlobWriter.java:172) ~[prprivcommon.jar:?]
***Edited by Moderator Marissa to add Support Case Details***
***Edited by Moderator Marije to add INC where URL was pasted***
Accepted Solution
Updated: 13 Feb 2024 9:16 EST
Pegasystems Inc.
GB
@EdH56292 The engineer helping you in the support ticket believes that from looking at the logs logs it looks there was some network issue which is intermittent and which is causing the disconnect .
In a similar case, the issue was resolved by using an HTTPS proxy instead of an HTTP proxy as a JVM argument. However, it's also important to ensure that the S3 bucket and object paths are correctly configured in your repository rule. If the issue persists, it might be beneficial to check with your AWS administrator to rule out any AWS-specific issues.
Can you please involve your network team and confirm and continue to work with our GCS team on this.
Pegasystems Inc.
GB
@EdH56292 I believe it would be best if you log a support incident for this via the MSP to investigate this in more depth. Please provide the INC id here so that we can help track it.
@szadp do you agree?
BlueCross BlueShield of MN
US
@EdH56292 I also have an incident INC-A28368.
This problem is still occurring. It is very random.
And we also see another type of message that is similar.
2023-12-23 22:45:56,737 [ PegaRULES-Batch-1] [ STANDARD] [ ] [ EIHMI:01.01.02] (narchitect_pxrepositorymanager) ERROR - There was an issue creating stream for the file.: Cannot find S3 object at Container/Bucket=prod-pega-files; prefix/root=file://connawss3:/;
Accepted Solution
Updated: 13 Feb 2024 9:16 EST
Pegasystems Inc.
GB
@EdH56292 The engineer helping you in the support ticket believes that from looking at the logs logs it looks there was some network issue which is intermittent and which is causing the disconnect .
In a similar case, the issue was resolved by using an HTTPS proxy instead of an HTTP proxy as a JVM argument. However, it's also important to ensure that the S3 bucket and object paths are correctly configured in your repository rule. If the issue persists, it might be beneficial to check with your AWS administrator to rule out any AWS-specific issues.
Can you please involve your network team and confirm and continue to work with our GCS team on this.
Pegasystems Inc.
GB
@EdH56292 I can see that the support ticket was closed as you have not had a re-occurrence of the issue and were not able to provide the required network trace to our support team.
As confirmed by GCS there is no script which would automatically test the S3 connection on restart
Please continue working with GCS if you need further help and mark Accept Solution on any reply here which helps you resolve your issue.