In the previous article, Create a High-Availability Kubernetes Cluster on AWS with Kops, we have seen how to create a Kubernetes cluster and how to deploy the Phoenix Chat app. The single node configuration worked well, but when we tried to scale out we saw that the messages were not sent to all the browsers.
In the image below, we see a configuration where the two chat servers are isolated. When the load-balancer routes the two WebSocket connections into two different servers, there is no way for the two browsers to send messages one another.
This problem is often solved using an external component, like Redis, which helps to broadcast the messages to all the nodes in the cluster.
With Elixir is also possible to get rid of this component, purely relying on the communication between Elixir nodes (we will see this in the next article).
In this article we’ll see how we can scale the Phoenix Chat horizontally, running multiple servers and leveraging on Redis PubSub.
Redis PubSub
If you want to test this out on your local machine, you can find this version of the app on the poeticoding/phoenix_chat_example GitHub repo, under the pubsub_redis branch.
We are now going to use the Redis PubSub messaging implementation, so we can broadcast the messages to all the Chat nodes. But .. What is PubSub messaging? I think the AWS definition is simple and explanatory.
Publish/subscribe messaging, or pub/sub messaging, is a form of asynchronous service-to-service communication used in serverless and microservices architectures. In a pub/sub model, any message published to a topic is immediately received by all of the subscribers to the topic.
AWS – Pub/Sub Messaging
To understand better how PubSub works, let’s see it in action with Redis. The easiest way to run a Redis server on our local machine is using Docker.
$ docker run --name redis -d redis -p 6379:6379
In this way we run Redis in background, starting a container we call redis. In the Docker container we’ve just started, we now find the redis-cli which we can use to do some experiments subscribing to a channel and publishing messages on it.
To execute the redis-cli process in the redis container, we use the docker exec command, passing the -it option to enable the interactive mode. The cli will connect to the local redis server by default.
$ docker exec -it redis redis-cli
In the image above, we run 4 different clients. Three of them subscribe to my_channel and one publishes a message to the channel. We see how Redis sends this message to all the clients subscribed to the channel.
Phoenix PubSub Redis adapter
To be able to leverage on this Redis PubSub functionality in our app, we are going to use the phoenix_pubsub_redis adapter. This adapter is really easy to integrate: we just need to add it in the dependencies and configure it in our phoenix app.
#mix.exs
defp deps do
[
...
{:phoenix_pubsub_redis, "~> 2.1"}
]
end#config/config.exs
config :chat, Chat.Endpoint,
url: [host: "localhost"],
root: Path.expand("..", __DIR__),
secret_key_base: ".......",
debug_errors: false,
pubsub: [
name: Chat.PubSub,
adapter: Phoenix.PubSub.Redis,
host: "localhost", port: 6379,
node_name: System.get_env("NODE")
]In this case we just manually set the host and the port to localhost and 6379, but if you are running a different Redis setup you maybe need to change this setting. We also need to set a node_name for each chat server we run, so we differentiate the Elixir nodes inside the Redis PubSub channel. We’ll pass the node_name using the NODE environment variable.
That’s it, ready to roll! We start two chat servers, one on 4000 called n1 and the other one on 4001 called n2.
# Terminal 1
$ NODE=n1 PORT=4000 mix phx.server
# Terminal 2
$ NODE=n2 PORT=4001 mix phx.server
The messages are now correctly sent to all the browsers connected to the channel, regardless of whether they are connected to different chat servers or not.
Inspecting the Phoenix PubSub messages in Redis
To understand better what’s going on, we can inspect the messages sent to Redis by the nodes. All the chat nodes publish messages on the phx:Elixir.Chat.PubSub Redis channel, but these messages are binary encoded using the :erlang.term_to_binary function, so we can’t simply use the redis-cli to properly see them.
In the repo, always under the pubsub_redis branch and in the redis_print directory, I’ve put a super-small Elixir app we can use to subscribe and decode the binary messages.
defmodule RedisPrint do
def subscribe(host,port,channel) do
{:ok, pubsub} = Redix.PubSub.start_link(host: host, port: port)
{:ok, ref} = Redix.PubSub.subscribe(pubsub, channel, self())
receive_messages(pubsub,ref)
end
def receive_messages(pubsub,ref) do
receive do
{:redix_pubsub, ^pubsub, ^ref, :message, %{channel: _, payload: payload}} ->
:erlang.binary_to_term(payload) |> IO.inspect()
end
receive_messages(pubsub,ref)
end
end
The RedisPrint.subscribe/3function starts a PubSub process which connects to Redis and subscribes to a specific channel. It then start receiving messages, recursively calling receive_messages/2 and decoding the payload with :erlang.binary_to_term
Let’s test again the chat with two browsers and two servers, this time inspecting the messages in a separate iex session.
# Terminal 1
$ NODE=n1 PORT=4000 mix phx.server
# Terminal 2
$ NODE=n2 PORT=4001 mix phx.server
# redis_print
$ iex -S mix
...
%{
__struct__: Phoenix.Socket.Broadcast,
event: "new:msg",
payload: %{body: "hello from u1", user: "u1"},
topic: "rooms:lobby"
}
%{
__struct__: Phoenix.Socket.Broadcast,
event: "new:msg",
payload: %{body: "hello from u2", user: "u2"},
topic: "rooms:lobby"
}
When the user u1, connected to n1 on port 4000 , sends the message “hello from u1”, this message is sent through the WebSocket connection and once n1receives it, it’s then published to the phx:Elixir.Chat.PubSub Redis channel.
Wrap Up
As I said at the beginning, this is just one way to make our Phoenix Chat app distributed. We’ll see in further articles how, thanks to how Elixir nodes can communicate, we connect and broadcast the messages using another Phoenix PubSub adapter.
