Testing PubSub Locally with Python + Docker

Sometimes you just want to quickly prototype a streaming system that uses pubsub. You don’t need the full power of the production version in Google Cloud. You don’t want to go through all the long prerequisite setup steps as with the official pubsub emulator guide. You just want to summon pubsub with a single command. And you don’t want to write verbose Java code.

What do you do? Well, an underrated (or perhaps just not really well-documented) alternative is to just spin up a docker container:

docker run -it --rm -p 8085:8085 gcr.io/google.com/cloudsdktool/cloud-sdk \
    gcloud beta emulators pubsub start --host-port=0.0.0.0:8085

As simple as that – you have a fake pubsub emulator running in your machine at port 8085 (the default port). Now we want to use python to start sending and receiving messages. We can use google-cloud-pubsub (API reference is also in the link) by installing it as such:

pip install google-cloud-pubsub

By default, the google-cloud-pubsub plugs itself to Google Cloud’s endpoint. We can instead instruct google-cloud-pubsub to connect to our emulator with the PUBSUB_EMULATOR_HOST environment variable:

export PUBSUB_EMULATOR_HOST=localhost:8085

Now let’s have a publisher up and running. With the emulator environment var set we launch an interactive python console and punch in the following:

from google.cloud import pubsub

publisher = pubsub.PublisherClient()
pubsub_topic = 'projects/dummy-project/topics/test-topic'

publisher.create_topic(pubsub_topic)

# Use publish to start shipping messages...
publisher.publish(pubsub_topic, b'Hello?')
publisher.publish(pubsub_topic, b'Is anyone there?')

I just sent over some byte string payload, so let’s print it in a client. Spin up another interactive python console for the client:

from google.cloud import pubsub

subscriber = pubsub.SubscriberClient()
pubsub_topic = 'projects/dummy-project/topics/test-topic'
sub_name = 'projects/dummy-project/subscriptions/subslot123'

subscriber.create_subscription(name=sub_name, topic=pubsub_topic)

# Blocking call to start listening. Passes all messages to print method
subscriber.subscribe(sub_name, print).result()

Now, if you forget to set the environment variable (as you normally would), don’t worry – Google will not fail to remind you to use their production-ready version:

Traceback (most recent call last):
  File "", line 1, in 
  File "/home/bruce/.local/lib/python3.8/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 114, in __init__
    channel = grpc_helpers.create_channel(
  File "/home/bruce/.local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 195, in create_channel
    credentials, _ = google.auth.default(scopes=scopes)
  File "/home/bruce/.local/lib/python3.8/site-packages/google/auth/_default.py", line 321, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

And that’s it. Happy experimenting!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.