Sometimes you just want to quickly prototype a streaming system that uses pubsub. You don’t need the full power of the production version in Google Cloud. You don’t want to go through all the long prerequisite setup steps as with the official pubsub emulator guide. You just want to summon pubsub with a single command. And you don’t want to write verbose Java code.
What do you do? Well, an underrated (or perhaps just not really well-documented) alternative is to just spin up a docker container:
docker run -it --rm -p 8085:8085 gcr.io/google.com/cloudsdktool/cloud-sdk \ gcloud beta emulators pubsub start --host-port=0.0.0.0:8085
As simple as that – you have a fake pubsub emulator running in your machine at port 8085 (the default port). Now we want to use python to start sending and receiving messages. We can use google-cloud-pubsub (API reference is also in the link) by installing it as such:
pip install google-cloud-pubsub
By default, the google-cloud-pubsub plugs itself to Google Cloud’s endpoint. We can instead instruct google-cloud-pubsub to connect to our emulator with the PUBSUB_EMULATOR_HOST environment variable:
Now let’s have a publisher up and running. With the emulator environment var set we launch an interactive python console and punch in the following:
from google.cloud import pubsub publisher = pubsub.PublisherClient() pubsub_topic = 'projects/dummy-project/topics/test-topic' publisher.create_topic(pubsub_topic) # Use publish to start shipping messages... publisher.publish(pubsub_topic, b'Hello?') publisher.publish(pubsub_topic, b'Is anyone there?')
I just sent over some byte string payload, so let’s print it in a client. Spin up another interactive python console for the client:
from google.cloud import pubsub subscriber = pubsub.SubscriberClient() pubsub_topic = 'projects/dummy-project/topics/test-topic' sub_name = 'projects/dummy-project/subscriptions/subslot123' subscriber.create_subscription(name=sub_name, topic=pubsub_topic) # Blocking call to start listening. Passes all messages to print method subscriber.subscribe(sub_name, print).result()
Now, if you forget to set the environment variable (as you normally would), don’t worry – Google will not fail to remind you to use their production-ready version:
Traceback (most recent call last): File "", line 1, in File "/home/bruce/.local/lib/python3.8/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 114, in __init__ channel = grpc_helpers.create_channel( File "/home/bruce/.local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 195, in create_channel credentials, _ = google.auth.default(scopes=scopes) File "/home/bruce/.local/lib/python3.8/site-packages/google/auth/_default.py", line 321, in default raise exceptions.DefaultCredentialsError(_HELP_MESSAGE) google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
And that’s it. Happy experimenting!