subscription_admin_client hanging on GCP when attempting to create subscriptions.

[iamsamskup]

Hi all,

I've run into an issue where subscription_admin_client.CreateSubscription() hangs when I'm trying to create subscription Ids on GCP. I'm getting no authentication errors and the code isn't throwing any exceptions. It just freezes at that line of code for the first iteration of the for-loop below until the kubernetes pod restarts it. The topics I'm attempting to subscribe to exist and I've been assured that all the permissions are set correctly. Furthermore I was able to create subscriptions, subscribe to them, and receive messages with no issues when using the pubsub emulator locally. Currently using 2.4.x, not sure which exact tag. Any insights on where to start looking next? TIA.

bool PubSubManager::createGCPSubscriptions(std::vector<std::pair<std::string, std::string>> topSubList) {

  bool createdAllTopics{true};

  try {
    google::cloud::pubsub::SubscriptionAdminClient subscription_admin_client(
        google::cloud::pubsub::MakeSubscriptionAdminConnection());
    WIRELOG_DEBUG("Subscription admin client created.")

    for (auto topicSubPair : topSubList) {
      try {
        WIRELOG_DEBUG("Creating subscription with projectId: " << projectID << " topic: " << topicSubPair.first << " subscription id : " << topicSubPair.second)
        auto sub = subscription_admin_client.CreateSubscription(
            google::cloud::pubsub::Topic(projectID, std::move(topicSubPair.first)), // topic Id
            google::cloud::pubsub::Subscription(projectID, std::move(topicSubPair.second))); //subscription Id
        WIRELOG_DEBUG("Subscription created")
        if(sub.status().code() == google::cloud::StatusCode::kAlreadyExists
            || sub.status().code() == google::cloud::StatusCode::kOk) {
          WIRELOG_DEBUG(topicSubPair.second << " created")
        }
        if (sub.status().code() == google::cloud::StatusCode::kAlreadyExists) {
          WIRELOG_DEBUG(topicSubPair.second << " already exists")
          continue;
        }
        if (!sub) {
          WIRELOG_DEBUG("Error create GCP subscription to topic: " << topicSubPair.second << " and error: " << sub.status().message())
          createdAllTopics =  false;
        }
        WIRELOG_DEBUG("The subscription was successfully created: " << sub->DebugString())
      } catch (std::exception &e) {
        WIRELOG_DEBUG("Caught exception while trying to create GCP subscriptions. Exception was: " << e.what())
        createdAllTopics =  false;
      }
    }
  } catch (std::exception &e) {
    WIRELOG_DEBUG("Caught exception creating subscription admin client. Error: " << e.what())
  }
  return createdAllTopics;
}

[coryan]

Hmm... Can you share how many subscriptions you are trying to create? There are relatively low quota limits for these things:

https://cloud.google.com/pubsub/quotas

You may also want to turn on logging in the library (TL;DR; set GOOGLE_CLOUD_CPP_ENABLE_CLOG=yes and GOOGLE_CLOUD_CPP_ENABLE_TRACING=rpc):

https://googleapis.dev/cpp/google-cloud-pubsub/latest/index.html#autotoc_md5

That may show you if the requests are failing with some kind of retryable error and what error that is. Finally, it is possible we need to set some kind of timeout on the RPCs. There are no public APIs to do this, but there are some internal-only APIs we could use to get you unblocked (pun intended).

[iamsamskup]

Attempting to create 9 subscriptions on 9 topics (one subscription per topic). I'll double check on the quotas and recompile with those flags and then get back to you. Thank you for the help!

[iamsamskup]

So we recompiled with those flags using the 2.5.0 tag and got no additional logs. Checked the quota limits and we're at 10k per topic so we're well below that threshold. I don't think I've mentioned this before but another team here is using the pubsub python library and are able to create subscriptions with no issues. Not sure if that info helps at all.

[coryan]

So we recompiled with those flags

These are environment variables, you should not need to recompile.

I don't think I've mentioned this before but another team here is using the pubsub python library and are able to create subscriptions with no issues.

Ack.

Not sure if that info helps at all.

That makes me believe it is the C++ code that needs fixing, but without more detailed logs would be hard to figure out what exactly.

[iamsamskup]

So I just realized I was still passing in the pubsub emulator port as an environment variable... Everything is working as expected now, my apologies for the false alarm. Thank you again for all the help!

[coryan]

No problem! I would rather folks asked more questions rather than getting stuck with a defect in our code or documentation.