Question
CBA
AU
Last activity: 31 Jan 2023 1:10 EST
Warning messages in pega log while Posting messages to external Kafka topic
Hi All,
We are getting lot of warning messages in pega logs when we write messages to external Kafka topic(avro message) . we analysed and found its due to field's definition in avro schema as it has data type as "null and string"(sample given below). but Kafka team clarified, its the standard to define non-mandatory fields. but in pega we are getting warning messages for those non mandatory fields and its flooding pega log with warnings.
to avoid writing these warning messages from pega logs we have created the log category with "Error" for below class com.pega.dsm.kafka.api.serde.avro.AvroSerdeException to write error and above on the pega logs for this class but it didnt help. we are still seeing the messages in pega logs.
Has anyone faced this issue. please help
Pega Version : 8.7.0
sample definition for non mandatory field
{ "name": "Version", "type": [ "null", { "type": "string", "avro.java.string": "String" }
sample definition for mandatory field
Hi All,
We are getting lot of warning messages in pega logs when we write messages to external Kafka topic(avro message) . we analysed and found its due to field's definition in avro schema as it has data type as "null and string"(sample given below). but Kafka team clarified, its the standard to define non-mandatory fields. but in pega we are getting warning messages for those non mandatory fields and its flooding pega log with warnings.
to avoid writing these warning messages from pega logs we have created the log category with "Error" for below class com.pega.dsm.kafka.api.serde.avro.AvroSerdeException to write error and above on the pega logs for this class but it didnt help. we are still seeing the messages in pega logs.
Has anyone faced this issue. please help
Pega Version : 8.7.0
sample definition for non mandatory field
{ "name": "Version", "type": [ "null", { "type": "string", "avro.java.string": "String" }
sample definition for mandatory field
"name": "toDate", "type": { "type": "string", "avro.java.string": "String" },
Warning message in Pega logs
Property couldn't be matched with the potential schema. com.pega.dsm.kafka.api.serde.avro.AvroSerdeException: Unsupported data type! NULL at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.resolveObjectByFieldType(ClipboardPageToGenericRecordConverter.java:97) ~[d-node.jar:?] at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.handleUnionType(ClipboardPageToGenericRecordConverter.java:108) ~[d-node.jar:?] at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.resolveObjectByFieldType(ClipboardPageToGenericRecordConverter.java:70) ~[d-node.jar:?] at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.convertClipboardPageToGenericRecord(ClipboardPageToGenericRecordConverter.java:51) ~[d-node.jar:?] at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.resolveObjectByFieldType(ClipboardPageToGenericRecordConverter.java:67) ~[d-node.jar:?] at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.convertClipboardPageToGenericRecord(ClipboardPageToGenericRecordConverter.java:51) ~[d-node.jar:?] at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.resolveObjectByFieldType(ClipboardPageToGenericRecordConverter.java:67) ~[d-node.jar:?] at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.convertClipboardPageToGenericRecord(ClipboardPageToGenericRecordConverter.java:51) ~[d-node.jar:?] at com.pega.dsm.kafka.api.serde.avro.AvroValuesSchemaRegistrySerde.serialize(AvroValuesSchemaRegistrySerde.java:99) ~[d-node.jar:?] at com.pega.dsm.kafka.api.serde.LazyExceptionHandlingSerDe.serialize(LazyExceptionHandlingSerDe.java:66) ~[d-node.jar:?]
***Edited by Moderator Marije to add Support Case Details***
***Edited by Moderator Marissa to add Product Version***
Accepted Solution
Updated: 31 Jan 2023 1:10 EST
Pegasystems Inc.
GB
@Prabhu Subramani I can see that a support incident was logged for this and our support team are looking into this. So far the analysis done is that the supported schema types are “RECORD, UNION, ARRAY, MAP , LONG, ENUM , STRING , INT , FLOAT , DOUBLE, BOOLEAN”, for anything else the unsupported warning along with AvroSerdeException is logged. Our support team will need to discuss with the Product team regarding the reason for not supporting NULL type.
@AshrafK4 I suggest you log an incident ticket via the MSP in order for your issue to be investigated seperately.
When you log the support ticket:
1. Can you try setting the log level to Error for the below class and check if the warnings are suppressed?
com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter
2. In your support ticket you'll need to share the below information,
- AVRO schema document
- Apart from the log files getting filled up with messages, do you have any other business impact ?
Tata Consultancy Services Ltd. Saudi Arabia
SA
@Prabhu Subramani any update on this issue. Since i am also facing the same.
Accepted Solution
Updated: 31 Jan 2023 1:10 EST
Pegasystems Inc.
GB
@Prabhu Subramani I can see that a support incident was logged for this and our support team are looking into this. So far the analysis done is that the supported schema types are “RECORD, UNION, ARRAY, MAP , LONG, ENUM , STRING , INT , FLOAT , DOUBLE, BOOLEAN”, for anything else the unsupported warning along with AvroSerdeException is logged. Our support team will need to discuss with the Product team regarding the reason for not supporting NULL type.
@AshrafK4 I suggest you log an incident ticket via the MSP in order for your issue to be investigated seperately.
When you log the support ticket:
1. Can you try setting the log level to Error for the below class and check if the warnings are suppressed?
com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter
2. In your support ticket you'll need to share the below information,
- AVRO schema document
- Apart from the log files getting filled up with messages, do you have any other business impact ?
Tata Consultancy Services Ltd. Saudi Arabia
SA
since logs are getting full we are seeing no space left on device. due to which kafka instance which is running on stream node is down and as well as web instance is not working properly. In all our system is going down frequently. i have to clear FS and restart trice a day, which is effecting out BAT.
Pegasystems Inc.
GB
@Shams Khan please log a support ticket, as already suggested.