The Log Streaming to AWS S3 feature in Pega Cloud 3 will receive several improvements and changes in an upcoming Pega Cloud release. This will provide clients with more flexibility and better performance – while simplifying our ability to introduce future enhancements.
These changes are only applicable to clients using Pega Cloud 3
What is Changing?
Improved stability and performance
The S3 log streaming service has been optimized to improve log streaming performance by over 20%. Log events will arrive in the destination S3 bucket more quickly after the upgrade. These enhancements will also result in a more stable, reliable log streaming solution.
Organize files in S3 by Log Type
Clients can optionally select to organize log events in their S3 bucket by Log Type. Available Log Types are listed in Understanding and obtaining Pega log files.
Legacy Structure: /<s3-bucket-name>/logs/<year>/<month>/<day>/<log-file-name>
New Structure: /<s3-bucket-name>/logs/application/<cluster-id>/<pega-tier-id>/<pega-pod-id>/<date>/<log-type>/<log-file-name>
Solution will default to the legacy configuration where all Log Types are combined in a single log file. Clients must request the new folder structure by raising a cloud change (CC) request support ticket
Add a custom prefix to the S3 folder structure
Clients can optionally request a custom prefix name for their S3 folder structure. The prefix defaults to "logs" if none is selected.
Example: /<s3-bucket-name>/<custom-prefix-name>/....
Default: /<s3-bucket-name>/logs/....
If you choose to customize this field, Pega typically recommends that the prefix correlate to your Pega Environment Name (dt1, stg1, prod...)
Normalized & improved log event & message formats
All log events are normalized to a JSON-based event structure. Most escape characters “\” have been removed to improve readability. Log events now contain more metadata to better describe each event and help with analysis.
Please review the “Log Event Format Changes” at the bottom of this document to see examples, compare old versus new event format.
Removing multiple log events for PegaRULES stack traces
We’ve resolved the scenario where very large stack traces in PegaRULES log files generate separate log events for each line of the stack trace. This will greatly simplify readability of large stack traces and eliminate the need to ‘piece the stack trace back together’ on the client side.
Removing support for log compression formats beside GZIP
To improve reliability and log stream delivery performance – we are removing support for the following file compression options:
- HADOOP_SNAPPY
- Snappy
- ZIP
- Uncompressed (previous default)
All logs will be compressed via GZIP by default.
One S3 Log Streaming instance per Pega Infinity environment
The updated log streaming service will support streaming logs from one Pega Infinity environment. You must request multiple S3 log streaming instances if you’d like to stream logs from multiple Infinity environments.
For example, if you have a dt1, stg, and a prod environment for your application – you will need to request a log streaming instance for each of those environments (3 total). Each instance can stream to the same S3 bucket or different S3 buckets depending on your preferences.
FAQ’s
| Question | Response |
|---|---|
| Will the log streaming upgrade cause downtime? | No. The upgrade will happen seamlessly, and logs will continue to stream throughout the update process. |
| Does anything change related to S3 bucket permissions/authorization? |
No, clients do not need to change any permission configurations. The update is seamless and re-uses existing IAM and KMS policies. |
|
What changes do I need to review and consider? |
|
|
How many S3 Log Streaming instances should I have configured? |
One S3 log streaming instance can be configured per Pega Infinity environment. A single log streaming instance can stream logs to one client S3 bucket. |
Log Event Format Changes
Samples of log event format changes for each supported Log Type.
Notable attribute naming changes are as follows:
|
Attribute |
Old Attribute Ref |
New Attribute Ref |
|---|---|---|
|
Log Stream Name |
.logStream |
.stream |
|
Log Message |
.message |
.log.message |
|
Event Timestamp |
.timestamp |
.time |
PegaRULES - New Format (Sample w/ exception)
{
"environmentType": "Production",
"pod_name": "pega-dataflow-abcdefg-mqzdp",
"host": "ip-111-22-33-44.us-west-2.compute.internal",
"clusterName": "cluster-abcdefghijklmnop",
"container_name": "pega-web-tomcat",
"region": "us-west-2",
"time": "2023-01-11T18:50:50.377960107Z",
"namespace_name": "acme-appname-dt1",
"log": {
"source_host": "pega-dataflow-abcdefg-mqzdp",
"tenantid": "",
"pegathread": "STANDARD",
"nodeType": "Batch,RealTime",
"logger_name": "com.pega.dsm.dnode.impl.dataflow.listener.RetryLoggerListener",
"message": "[DF-Test-9002] Preparing retry 90. Next attempt in 2m on data flow [CaptureFeedback] on stage [Capture Response] with affected partitions: [1]",
"level": "ERROR",
"CorrelationId": "DF-Test-9002",
"userid": "",
"appender_ref": "PEGA",
"thread_name": "DataFlow-Service-PickingupRun-DF-Test-9002_Input:67, Access group: [ACME:CDHAdmins], Partitions=[1]",
"exception": {
"stacktrace": "org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [private.acme.captureresponse.event]\n",
"exception_class": "org.apache.kafka.common.errors.TopicAuthorizationException",
"exception_message": "Not authorized to access topics: [private.acme.captureresponse.event]"
},
"app": "",
"stack": ""
},
"stream": "PegaRULES"
}
PegaCLUSTER - New Format (Sample)
{
"namespace_name": "acme-appname-dt1",
"host": "ip-111-22-33-444.us-west-2.compute.internal",
"clusterName": "cluster-abcdefhijklmnop",
"region": "us-west-2",
"time": "2023-01-13T16:01:41.428484791Z",
"log": {
"message": "hz.client_1 [prpchz] [5.1.5] HazelcastClient 5.1.5 (20221114 - 8b36195, c52d6ff) is SHUTTING_DOWN",
"level": "INFO",
"app": "",
"source_host": "pega-background-1234abcd-hjdp7",
"userid": "",
"pegathread": "STANDARD",
"logger_name": "com.pega.hazelcast.v5.core.LifecycleService",
"stack": "",
"tenantid": "",
"appender_ref": "CLUSTER",
"nodeType": "BackgroundProcessing,Search,BIX,PegaMKTSMS,PegaMKTEmail,Custom1,Custom2,Custom3,Custom4,Custom5",
"thread_name": "Thread-57"
},
"environmentType": "Production",
"stream": "PegaCLUSTER",
"pod_name": "pega-background-1234abcd-hjdp7",
"container_name": "pega-web-tomcat"
}
PegaRULES-ALERT - New Format (Sample)
{
"clusterName": "cluster-abcdefghijklmnop",
"region": "us-west-2",
"host": "ip-111-22-333-444.us-west-2.compute.internal",
"environmentType": "Production",
"time": "2023-01-11T18:50:59.061944247Z",
"log": {
"nodeType": "BackgroundProcessing,Search,BIX,PegaMKTSMS,PegaMKTEmail,Custom1,Custom2,Custom3,Custom4,Custom5",
"appender_ref": "PEGA",
"level": "ALERT",
"app": "",
"CorrelationId": "222D95BC4C2C8AFD06A111186BFA923F",
"thread_name": "JOBSCHEDULER_THREAD_6",
"RequestorId": "BA24BVS4T3U131BBBV1T7A60H2EN0WFYPA",
"tenantid": "",
"message": "2023-01-11 18:50:59,061 GMT*8*PEGA0035*10001*10000*pega-background-abcdefg-fp2df*NA*NA*BA24BVS4T3U131BBBV1T7A60H2EN0WFYPA.....",
"pegathread": "STANDARD",
"source_host": "pega-background-abcdefg-fp2df",
"alertType": "performance",
"logger_name": "com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl",
"userid": "",
"stack": ""
},
"namespace_name": "acme-appname-dt1",
"pod_name": "pega-background-abcdefg-fp2df",
"container_name": "pega-web-tomcat",
"stream": "PegaRULES-ALERT"
}
PegaRULES-ALERTSECURITY - New Format (Sample)
{
"pod_name": "pega-web-abcdefg-aaaa",
"namespace_name": "acme-appname-dt1",
"host": "ip-111-22-33-444.us-west-2.compute.internal",
"time": "2023-01-11T19:20:35.730020166Z",
"log": {
"app": "ACME:01.01.01",
"stack": "acme-appname-dt1.pega.net|47.204.224.226",
"source_host": "pega-web-abcd123-aaaa",
"x-forwarded-client-cert": "By=spiffe://cluster.local/ns/acme-appname-dt1/sa/pega-infinity-sa;",
"InteractionNumber": "231",
"x-forwarded-port": "443",
"CorrelationId": "DDAABBDDCCCEEDDEA",
"nodeId": "pega-web-abcd123-aaaa",
"logger_name": "com.pega.pegarules.session.internal.mgmt.util.URLAccessContext",
"x-requested-with": "XMLHttpRequest",
"RequestorId": "DDAABBDDCCCEEDDEA",
"level": "ALERT",
"x-b3-traceid": "3b9de55a7e27f96673ebac8aec774ddd",
"tenantid": "",
"x-request-id": "783cf997-ad9f-4950-ab04-f71a9c25d877",
"x-b3-sampled": "0",
"message": "2023-07-13 19:20:35,729 GMT*8*SECU0008*0*0*pega-web-abcd123-aaaa...",
"thread_name": "http-nio2-8080-exec-3",
"userid": "",
"alertType": "security",
"pegathread": "TABTHREAD3",
"x-amzn-trace-id": "Root=1-64b04e83-4859a0ba5b20a10427abac7b",
"x-b3-spanid": "73ebac8aec774fed",
"x-forwarded-proto": "https",
"appender_ref": "PEGA",
"nodeType": "WebUser"
},
"internal_tag": "internal_security",
"container_name": "pega-web-tomcat",
"environmentType": "Production",
"stream": "PegaRULES-ALERTSECURITY",
"region": "us-west-2",
"clusterName": "cluster-abcd1234567abcd"
}
PegaRULES-SECURITYEVENT - New Format (Sample)
{
"stream": "PegaRULES-SECURITYEVENT",
"clusterName": "cluster-abcdefghijklmnop",
"region": "eu-west-3",
"pod_name": "pega-web-abcd123-gxdqq",
"namespace_name": "grpss-prc25-prod",
"environmentType": "Production",
"log": {
"tenantid": "",
"x-forwarded-proto": "https",
"InteractionNumber": "1",
"x-b3-parentspanid": "11223344556677",
"x-forwarded-client-cert": "By=test://cluster.local/ns/acme-appname-prod/sa/pega-infinity-sa;...",
"logger_name": "com.pega.pegarules.session.internal.mgmt.SecurityEventLogger",
"x-source": "intra",
"stack": "pega-web-abcd123-gxdqq|11.222.333.44|Rest|ddd|v1...",
"x-b3-sampled": "0",
"@timestamp": "2023-01-11T16:12:05.474Z",
"userid": "",
"nodeType": "WebUser",
"source_host": "pega-web-abcd123-gxdqq",
"x-b3-traceid": "1122334455",
"x-request-id": "ca3d4182-c6ff-4822-9ae3-7607199fb37c",
"x-envoy-internal": "true",
"level": "INFO",
"nodeId": "pega-web-abcd123-gxdqq",
"appender_ref": "SECURITYEVENT",
"x-b3-spanid": "ac002bbc65808b35",
"message": "{\"appName\":\"acmeappname\",\"eventCategory\":\"Authentication event\",\"eventType\":\"Login\",\"id\":\"8baa24bf-9a52-cccc-ba3d-a63b60e5d8ce\",\"ipAddress\":\"11.222.333.44\",\"nodeID\":\"pega-web-abcd123-gxdqq\",\"operatorID\":\"TTCDDADADSFEA\",\"outcome\":\"Success\",\"tenantID\":\"shared\",\"timeStamp\":\"Mon 2023 Jan 01, 16:12:05:474\"}",
"x-amzn-trace-id": "Root=1-64b56855-4c19251830e222004362d2f9",
"pegathread": "STANDARD",
"x-correlationid": "Id-5568b56452343f17da0c1398 1",
"@version": 1,
"x-forwarded-port": "443",
"RequestorId": "A655803F26A77CFF9A81C53077C666D83A",
"x-envoy-expected-rq-timeout-ms": "600000",
"thread_name": "http-nio2-8080-exec-91",
"x-dynatrace": "FW4;1649345699;2;61198609;",
"app": "acmeappname:23.06"
},
"host": "ip-11-222-333-444.eu-west-3.compute.internal",
"internal_tag": "internal_security",
"time": "2023-01-11T16:12:05.474565709Z",
"container_name": "pega-web-tomcat"
}
PegaDATAFLOW - New Format (Sample)
{
"clusterName": "cluster-abcdefghijklmnop",
"region": "us-west-2",
"host": "ip-111-22-33-444.us-west-2.compute.internal",
"environmentType": "Production",
"time": "2023-01-11T19:00:53.795100452Z",
"log": {
"nodeType": "BackgroundProcessing,Search,BIX,PegaMKTSMS,PegaMKTEmail,Custom1,Custom2,Custom3,Custom4,Custom5",
"source_host": "pega-background-abcd1234-fp2df",
"thread_name": "DataFlow-Event-Executor:23",
"app": "",
"tenantid": "",
"CorrelationId": "BJCTGMCCZB155IC2SGNHHN8JFEDD1CGHVA",
"pegathread": "STANDARD",
"message": "{\"type\":\".RunStatusTransitionMessage\",\"senderNodeId\":\"pega-background-5ff8c6b87c-fp2df\",\"timestamp\":1689274853794,\"messageId\":\"fc5ef353-7302-4fc1-8b21-6addda6c5025\",\"runId\":\"ST-Import-58030\",\"previousStatus\":\"PENDING_NODES\",\"newStatus\":\"PENDING_INITIALIZATION\",\"originator\":\"CheckNodeAvailabilityTask\",\"reason\":\"Resources are available to start run. Active runs will be [2] out of maximum [10]\"}",
"appender_ref": "DATAFLOW",
"stack": "",
"logger_name": "com.pega.dsm.dnode.impl.dataflow.service.DataFlowDiagnosticsFileLogger",
"userid": "",
"level": "INFO"
},
"namespace_name": "acme-appname-dt1",
"pod_name": "pega-background-abcd1234-fp2df",
"container_name": "pega-web-tomcat",
"stream": "PegaDATAFLOW"
}
Localhost_Access_Log - New Format (Sample)
{
"date":"2024-01-01T00:01:26.453192Z",
"log":{
"log":"- 127.0.0.6 - - [01/Jan/2024:00:01:25 +0000] \"GET /prweb/PRRestService/monitor/pingService/ping HTTP/1.1\" 200 426 1 http-nio2-8080-exec-748",
"appender_ref":"localhost_access_log",
"date":"2024-07-08T00:01:25.657329Z"
},
"pod_name":"pega-web-12345-abcde",
"container_name":"fluent-bit-helper",
"namespace_name":"test-abc-prod1",
"host":"ip-10-0-0-110.us-west-2.compute.internal",
"region":"us-west-2",
"environmentType":"Production",
"time":"2024-01-01T00:01:26.447355051Z",
"internal_tag":"internal_security",
"stream":"localhost_access_log",
"clusterName":"cluster-abc123"
}