0

We are using Google cloud dataflow pipeline which runs in streaming mode means it runs 24/7 on a daily basis. Suddenly it got stuck and throws error as follow:

java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
        org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:435)
        org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:167)
        java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2338)
        java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2351)
        java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
        java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
        java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
        org.apache.beam.sdk.util.SerializableUtils$ContextualObjectInputStream.<init>(SerializableUtils.java:164)
        org.apache.beam.sdk.util.SerializableUtils$ContextualObjectInputStream.<init>(SerializableUtils.java:162)
        org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:68)
        org.apache.beam.runners.dataflow.util.CloudObjectTranslators$9.fromCloudObject(CloudObjectTranslators.java:359)
        org.apache.beam.runners.dataflow.util.CloudObjectTranslators$9.fromCloudObject(CloudObjectTranslators.java:341)
        org.apache.beam.runners.dataflow.util.CloudObjects.coderFromCloudObject(CloudObjects.java:122)
        org.apache.beam.runners.dataflow.util.CloudObjectTranslators.getComponents(CloudObjectTranslators.java:72)
        org.apache.beam.runners.dataflow.util.CloudObjectTranslators.access$100(CloudObjectTranslators.java:51)
        org.apache.beam.runners.dataflow.util.CloudObjectTranslators$6.fromCloudObject(CloudObjectTranslators.java:258)
        org.apache.beam.runners.dataflow.util.CloudObjectTranslators$6.fromCloudObject(CloudObjectTranslators.java:245)
        org.apache.beam.runners.dataflow.util.CloudObjects.coderFromCloudObject(CloudObjects.java:122)
        org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$2.typedApply(IntrinsicMapTaskExecutorFactory.java:345)
        org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$2.typedApply(IntrinsicMapTaskExecutorFactory.java:339)
        org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
        org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
        org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
        org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:121)
        org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1199)
        org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:147)
        org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1020)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        java.lang.Thread.run(Thread.java:745)

We have raised it to cloud Support they have asked us to upgrade pipeline to version 2.13 (Apache Dataflow version) but it does not solved our problem.

How can I fix it?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.