Discussion
Pegasystems Inc.
JP
Last activity: 1 May 2023 12:29 EDT
Maximum file size that can be sent and received via Pega REST Service / Connector
Hi,
Some customers may want to send and receive a huge file by String (Base64 encoding) thru Pega REST Service / Connector. In this post, I am sharing the maximum file size in this regard.
- Test scenario
| No | Scenario |
|---|---|
| 1 | Client sends a request by Connect REST. Once server receives it, server grabs a huge file from repository and sends it back to client after Base64 encoding by Service REST. |
| 2 | Client sends a huge file as Base64 encoded String to server by Connect REST. |
- Test environment
- Test result
(1) Using Oracle JDK 8
| Test scenario | 500MB (before Base64 encoding) | 550MB (before Base64 encoding) |
|---|---|---|
| 1 | Success |
Failure "java.lang.OutOfMemoryError: Requested array size exceeds VM limit" error occurs at Service REST step |
| 2 | Success |
Failure "java.lang.OutOfMemoryError: Requested array size exceeds VM limit" error occurs at Connect REST step |
(2) Using Oracle JDK 14
| Test scenario | 500MB (before Base64 encoding) | 550MB (before Base64 encoding) | 745MB (before Base64 encoding) | 775MB (before Base64 encoding) |
|---|---|---|---|---|
| 1 | Success | Success | Success |
Failure "Connect Rest Side: Content is too large(1073741820) bytes, to be stored in memory. Aborting" error occurs at Connect REST step |
| 2 | Success | Success | Success |
Failure "java.lang.OutOfMemoryError: Requested array size exceeds VM limit" error occurs at Service REST step |
*Notes
As of today, the current Pega Platform 8.4 only supports JDK 8. JDK 14 is not supported yet. However, we have conducted this experiment to see if JDK version makes any difference.
- Test evaluation
In Pega 8.4, you should use JDK 8. As long as you use JDK 8, the maximum file size that you can send and receive is around 500MB. 500MB is before Base64 encoding. Base64 encoding adds approximately 33%, so actual size will be 500*4/3=666.66...MB. 550MB, actual size 550*4/3=733.33...MB, threw an OOM error "java.lang.OutOfMemoryError: Requested array size exceeds VM limit". This error is happening at sending side, and it couldn't even reach out to receiving side. Also, this is a JDK-level error and there is no failure at Pega side.
For an experimental purpose, we've tried JDK 14 and we confirmed it behaves differently. 500MB, 550MB, 745MB were successful, and 775MB failed. 745MB's actual size is 993.33...MB and 775MB's actual size is 1033.33...MB. This error, "Connect Rest Side: Content is too large(1073741820) bytes, to be stored in memory. Aborting" is happening at receiving side. Note sending was at least successful. Also, this is a Pega-level error, not JDK-level error. In Pega's code (HttpClientImpl.java), the limitation is defined as "(Integer.MAX_VALUE-8)/2", and that means 1024MB fixed regardless of JDK version (8, 14, etc).
To conclude, 1GB restriction is hardcoded in Pega engine level, but it doesn't mean you can fully utilize it. Depending on JDK version, you may hit prior restriction in JDK level.
Thanks,