Exception while consuming data from Collector #82
Description
Setup:
Running traceshop locally with HT java agent and Hypertrace
I see following Deserialization exceptions occur intermittently in Span-normalizer service while consuming from jaeger-span kafka topic
I am not sure if it happens with existing agents as well. If it is a known issue please let me know.
hypertrace-ingester | 2020-12-04 07:52:44.839 [hypertrace-ingester-job-3badbaeb-656c-40c6-affc-58da86a0133d-StreamThread-1] WARN o.a.k.s.e.LogAndContinueExceptionHandler - Exception caught during Deserialization, taskId: 0_0, topic: jaeger-spans, partition: 0, offset: 141413 hypertrace-ingester | java.lang.RuntimeException: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8. hypertrace-ingester | at org.hypertrace.core.spannormalizer.jaeger.JaegerSpanSerde$De.deserialize(JaegerSpanSerde.java:45) ~[span-normalizer-0.3.4.jar:?] hypertrace-ingester | at org.hypertrace.core.spannormalizer.jaeger.JaegerSpanSerde$De.deserialize(JaegerSpanSerde.java:38) ~[span-normalizer-0.3.4.jar:?] hypertrace-ingester | at org.apache.kafka.common.serialization.Deserializer.deserialize(Deserializer.java:60) ~[kafka-clients-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.SourceNode.deserializeValue(SourceNode.java:63) ~[kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:66) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.RecordQueue.updateHead(RecordQueue.java:175) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.RecordQueue.poll(RecordQueue.java:126) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.PartitionGroup.nextRecord(PartitionGroup.java:122) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:368) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:475) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:550) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8. hypertrace-ingester | at com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:141) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8$DecodeUtil.handleTwoBytes(Utf8.java:1909) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8$DecodeUtil.access$700(Utf8.java:1883) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8(Utf8.java:1411) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8.decodeUtf8(Utf8.java:340) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.CodedInputStream$ArrayDecoder.readStringRequireUtf8(CodedInputStream.java:809) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$KeyValue.<init>(JaegerSpanInternalModel.java:352) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$KeyValue$1.parsePartialFrom(JaegerSpanInternalModel.java:1261) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$KeyValue$1.parsePartialFrom(JaegerSpanInternalModel.java:1255) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:888) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span.<init>(JaegerSpanInternalModel.java:4100) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span$1.parsePartialFrom(JaegerSpanInternalModel.java:6560) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span$1.parsePartialFrom(JaegerSpanInternalModel.java:6554) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:158) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span.parseFrom(JaegerSpanInternalModel.java:4713) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at org.hypertrace.core.spannormalizer.jaeger.JaegerSpanSerde$De.deserialize(JaegerSpanSerde.java:43) ~[span-normalizer-0.3.4.jar:?] hypertrace-ingester | ... 13 more hypertrace-ingester | 2020-12-04 07:52:44.841 [hypertrace-ingester-job-3badbaeb-656c-40c6-affc-58da86a0133d-StreamThread-1] WARN o.a.k.s.p.i.RecordDeserializer - stream-thread [hypertrace-ingester-job-3badbaeb-656c-40c6-affc-58da86a0133d-StreamThread-1] task [0_0] Skipping record due to deserialization error. topic=[jaeger-spans] partition=[0] offset=[141413]