Unable to use google-cloud-pubsub with google dataflow

[seehans]

I am trying to initialize Pulisher using following snippet in dataflow code:

Publisher
  .defaultBuilder(topicName)
  .setChannelProvider(TopicAdminSettings
    .defaultChannelProviderBuilder()
    .setCredentialsProvider(FixedCredentialsProvider.create(credentials))
    .build())
  .build();

I am getting exception: "java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.pubsub.v1.TopicAdminSettings

Stacktrace:
exception: "java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.pubsub.v1.TopicAdminSettings at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:162) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:288) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:450) at com.FileNameParser$BucketFileNameParser.processElement(FileNameParser.java:93) Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.pubsub.v1.TopicAdminSettings at com.google.cloud.pubsub.v1.Publisher$Builder.<init>(Publisher.java:579) at com.google.cloud.pubsub.v1.Publisher$Builder.<init>(Publisher.java:528) at com.google.cloud.pubsub.v1.Publisher.defaultBuilder(Publisher.java:524) at com.ReadFile$ReadBucketFile.createPublisherWithCustomCredentials(ReadFile.java:176) at com.ReadFile$ReadBucketFile.publishMessage(ReadFile.java:144) at com.ReadFile$ReadBucketFile.processElement(ReadFile.java:118) at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:139) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190) at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47) at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:55) at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:160) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:288) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:450) at com.FileNameParser$BucketFileNameParser.processElement(FileNameParser.java:93) at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:139) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190) at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47) at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:55) at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52) at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:224) at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.start(ReadOperation.java:185) at com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:72) at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:710) at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker.access$500(StreamingDataflowWorker.java:99) at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker$8.run(StreamingDataflowWorker.java:799) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

[garrettjonesgoogle]

This is an unfortunate situation and won't get fixed until Dataflow upgrades their pubsub dependency to 0.20.0-beta or later. The versions before 0.20.0-beta of pubsub are alpha, and combining different versions of alpha pubsub is not supported.

[seehans]

@garrettjonesgoogle thank you for your reply. You said "combining different versions of alpha pubsub is not supported" does that mean, even if I use alpha version of pubsub with dataflow, it will not work?

[garrettjonesgoogle]

If you use the same version that dataflow depends on, it will work.

[seehans]

Thank you. I could not find a DataflowJavaSDK using Google Cloud Java Client for Pub/Sub for gRPC.

I appreciate your reply.