Akka Clustering with Kubernetes

klusterd is an example Akka Cluster project that is packaged using Docker and deployed to Kubernetes.

Taking klusterd for a test drive

First, clone the project from GitHub.

git clone https://github.com/vyshane/klusterd.git

We’ll use sbt to package the application as a Docker image.

$ cd klusterd
$ sbt "docker:publishLocal"

This creates a local Docker image for us:

$ docker images | grep klusterd
vyshane/klusterd                      1.0                  43ee995e8adb        1 minute ago      690.7 MB

To deploy our klusterd image to our Kubernetes cluster:

$ cd ../deployment
$ ./up.sh

We can see that a klusterd pod comes up:

$ kubectl get pods
NAME                   READY     STATUS    RESTARTS   AGE
klusterd-6srxl         1/1       Running   0          1m

Let’s tail its log:

$ kubectl logs -f klusterd-6srxl

We can see that klusterd is running at the IP address 172.17.0.3 and listening on port 2551.

INFO  14:58:06.022UTC akka.remote.Remoting - Starting remoting
INFO  14:58:06.235UTC akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://klusterd@172.17.0.3:2551]

Since we have only launched one klusterd node, it is its own cluster seed node.

INFO  14:58:06.446UTC akka.actor.ActorSystemImpl(klusterd) - Configured seed nodes: akka.tcp://klusterd@172.17.0.3:2551

We have a cluster of one.

INFO  14:58:06.454UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Node [akka.tcp://klusterd@172.17.0.3:2551] is JOINING, roles []
INFO  14:58:06.482UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Leader is moving node [akka.tcp://klusterd@172.17.0.3:2551] to [Up]
INFO  14:58:06.505UTC akka.tcp://klusterd@172.17.0.3:2551/user/cluster-monitor - Cluster member up: akka.tcp://klusterd@172.17.0.3:2551

Now, let’s scale this cluster up. Let’s ask Kubernetes for 2 more nodes:

$ kubectl scale --replicas=3 rc klusterd

We can see from klusterd-6srxl’s logs that two more nodes join our cluster shortly after.

INFO  15:11:46.525UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Node [akka.tcp://klusterd@172.17.0.4:2551] is JOINING, roles []
INFO  15:11:46.533UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Node [akka.tcp://klusterd@172.17.0.5:2551] is JOINING, roles []
INFO  15:11:47.484UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Leader is moving node [akka.tcp://klusterd@172.17.0.4:2551] to [Up]
INFO  15:11:47.505UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Leader is moving node [akka.tcp://klusterd@172.17.0.5:2551] to [Up]
INFO  15:11:47.507UTC akka.tcp://klusterd@172.17.0.3:2551/user/cluster-monitor - Cluster member up: akka.tcp://klusterd@172.17.0.4:2551
INFO  15:11:47.507UTC akka.tcp://klusterd@172.17.0.3:2551/user/cluster-monitor - Cluster member up: akka.tcp://klusterd@172.17.0.5:2551

We can scale the cluster down:

$ kubectl scale --replicas=2 rc klusterd

And a node leaves the cluster. In our case, the original pod klusterd-6srxl was killed. Here are the logs from another node showing what happened:

WARN  15:19:31.855UTC akka.tcp://klusterd@172.17.0.4:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fklusterd%40172.17.0.3%3A2551-0 - Association with remote system [akka.tcp://klusterd@172.17.0.3:2551] has failed, address is now gated for [5000] ms. Reason: [Disassociated] 
WARN  15:19:35.376UTC akka.tcp://klusterd@172.17.0.4:2551/user/cluster-monitor - Cluster member unreachable: akka.tcp://klusterd@172.17.0.3:2551
WARN  15:19:35.405UTC akka.tcp://klusterd@172.17.0.4:2551/system/cluster/core/daemon - Cluster Node [akka.tcp://klusterd@172.17.0.4:2551] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://klusterd@172.17.0.3:2551, status = Up)]
INFO  15:19:45.384UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.4:2551] - Leader is auto-downing unreachable node [akka.tcp://klusterd@172.17.0.3:2551]
INFO  15:19:45.387UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.4:2551] - Marking unreachable node [akka.tcp://klusterd@172.17.0.3:2551] as [Down]
INFO  15:19:46.412UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.4:2551] - Leader is removing unreachable node [akka.tcp://klusterd@172.17.0.3:2551]
INFO  15:19:46.414UTC akka.tcp://klusterd@172.17.0.4:2551/user/cluster-monitor - Cluster member removed: akka.tcp://klusterd@172.17.0.3:2551
WARN  15:19:52.756UTC akka.tcp://klusterd@172.17.0.4:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fklusterd%40172.17.0.3%3A2551-3 - Association with remote system [akka.tcp://klusterd@172.17.0.3:2551] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://klusterd@172.17.0.3:2551]] Caused by: [No response from remote for outbound association. Associate timed out after [15000 ms].]
INFO  15:19:52.758UTC akka.tcp://klusterd@172.17.0.4:2551/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fklusterd%40172.17.0.3%3A2551-4 - No response from remote for outbound association. Associate timed out after [15000 ms].

To turn off everything:

./down.sh

Full Post + Comments

Multi-node Cassandra Cluster Made Easy with Kubernetes

Cassandra cluster in terminal window

I’ve just shared an easy way to launch a Cassandra cluster on Kubernetes.

The Kubernetes project has a Cassandra example that uses a custom seed provider for seed discovery. The example makes use of a Cassandra Docker image from gcr.io/google_containers.

However, I wanted a solution based on the official Cassandra Docker image. This is what I came up with.

First, I created a headless Kubernetes service that provides the IP addresses of Cassandra peers via DNS A records. The peer service definition looks like this:

<br />apiVersion: v1
kind: Service
metadata:
  labels:
    name: cassandra-peers
  name: cassandra-peers
spec:
  clusterIP: None
  ports:
    - port: 7000
      name: intra-node-communication
    - port: 7001
      name: tls-intra-node-communication
  selector:
    name: cassandra

Then I extended the official Cassandra image with the addition of dnsutils (for the dig command) and a custom entrypoint that configures seed nodes for the container. The new entrypoint script is pretty straight forward:

my_ip=$(hostname --ip-address)

CASSANDRA_SEEDS=$(dig $PEER_DISCOVERY_DOMAIN +short | 
    grep -v $my_ip | 
    sort | 
    head -2 | xargs | 
    sed -e 's/ /,/g')

export CASSANDRA_SEEDS

/docker-entrypoint.sh "$@"

Whenever a new Cassandra pod is created, it automatically discovers seed nodes through DNS.

Full Post + Comments

Kubernetes via Docker Compose

Kubernetes Logo

So I’ve decided to revisit my Kubernetes development cluster setup. The Mark 2 setup uses Docker Compose to launch the Kubernetes cluster in Docker.

To try it out, clone the GitHub repository:

git clone https://github.com/vyshane/docker-compose-kubernetes.git

The project also comes with:

Starting Kubernetes on Linux

On Linux we’ll run Kubernetes using a local Docker Engine. You will also need Docker Compose as well as the kubectl tool. To launch the cluster:

./kube-up.sh

Starting Kubernetes on OS X

On OS X we’ll launch Kubernetes inside a boot2docker VM via Docker Machine. You will need to have Docker Machine (v0.5.0 or newer), Docker Compose, and the kubectl tool installed locally. First start your boot2docker VM:

docker-machine start <name>
eval "$(docker-machine env $(docker-machine active))"

Then, launch the Kubernetes cluster in boot2docker via Docker Machine:

./kube-up.sh

The script will set up port forwarding so that you can use kubectl locally without having to ssh into boot2docker.

Checking if Kubernetes Is Running

kubectl cluster-info
Kubernetes master is running at http://localhost:8080
KubeUI is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-ui

Accessing Kube UI

You can access Kube UI at http://localhost:8080/ui.

To destroy the cluster

./kube-down.sh

This will also remove any services, replication controllers and pods that are running in the cluster.

Full Post + Comments

Setting up a Kubernetes Cluster using CoreOS on VMWare Fusion

Kubernetes Logo

I recently wanted to set up a local Kubernetes cluster for development on my Mac. I tried many howtos in various state of staleness and brokenness. The official Vagrant setup didn’t even work for me on OS X until yesterday.

While I was happy that the Vagrant setup was working again, my preferred development environment was on CoreOS via VMWare Fusion. The following is what worked for me. I’ve put together information from the Kubernetes “CoreOS Multinode Cluster” guide and the CoreOS “Running CoreOS on VMware” documentation.

Getting the latest CoreOS VMWare image

First, get the latest VMWare image for the CoreOS channel that you want. E.g. for the beta channel:

curl -LO http://beta.release.core-os.net/amd64-usr/current/coreos_production_vmware_ova.ova

Creating a virtual machine for the Kubernetes master

Open the downloaded file using VMWare Fusion. VMWare will create a .vmwarevm from the .ova. Give your virtual machine a name, e.g. “Kubernetes Master”.

We’ll use Cloud-Config to customise this CoreOS instance upon boot, turning it into our Kubernetes cluster master. Download the master.yaml file from the Kubernetes CoreOS getting started guide

Then create an .iso that VMWare can use as a config drive for the virtual machine.

mkdir -p /tmp/new-drive/openstack/latest/
cp master.yaml /tmp/new-drive/openstack/latest/user_data
hdiutil makehybrid -iso -joliet -joliet-volume-name "config-2" \ 
  -joliet -o master.iso /tmp/new-drive

You can get the virtual machine to access the .iso by going to the VM’s Settings, then CD/DVD (IDE).

Creating a Kubernetes minion virtual machine

Open the downloaded .ova file again using VMWare Fusion. Give your virtual machine a name, e.g. “Kubernetes Node 1″.

Download the node.yaml file from the Kubernetes CoreOS getting started guide. Edit the file and replace all instances of with the IP address of the master node. Then create a node config-drive.

mkdir -p /tmp/new-drive/openstack/latest/
cp node.yaml /tmp/new-drive/openstack/latest/user_data
hdiutil makehybrid -iso -joliet -joliet-volume-name "config-2" \ 
  -joliet -o node.iso /tmp/new-drive

Boot your Kubernetes node image using node.iso as a config drive. You can create multiple nodes and they will connect to the master.

Testing it out

Once you have both the master and minion up, you can use kubectl to check on the cluster.

kubectl -s <master-ip-address>:8080 get nodes

Full Post + Comments

Android bitmap image resource sizes for the different screen pixel densities

Here’s a handy table that you can refer to when creating image bitmap resources for your Android app.

Qualifier Approximate dpi Scaling ratio
ldpi 120 Baseline
mdpi 160 43 * ldpi
hdpi 240 64 * mdpi
xhdpi 320 86 * hdpi
xxhdpi 480 128 * xhdpi
xxxhdpi 640 1612 * xxhdpi
nodpi N/A Not scaled
tvdpi 213 1.33 * mdpi

xxxhdpi is only used for launch icons.

Source: Android Developer Guide: Providing Alternative Resources

Full Post + Comments

Where the magic happens

Home office

The man cave.

Full Post + Comments

Alexis, born 01/11/2014

Alexis

Full Post + Comments

The best read-it-later implementation that will never be

Google already has the tech to build the best read-it-later service bar none. It understands web content like no other and can even crawl JavaScript applications. Google can unshackle content that I actually want to read from all the surrounding noise.

Google can, but unfortunately it won’t. Doing so would anger content owners, the people that Google relies on to display its ads. And that’s a pity.

Full Post + Comments

Elixir Notes

Elixir books

I’ve been learning myself some Elixir lately. Unfortunately, I’ve only been able to sip while I really want to be gulping the stuff down. There aren’t enough hours in a day. Sometimes a week passes between study sessions and it’s hard for me to find my bearings when I come back to the language after a break. So I’ve started writing some study notes. I find that it helps me internalise the syntax and makes what I’m learning more sticky. The result is more than just a cheat sheet, but remains easy to glance through.

It’s a work in progress, but might be useful to somebody: Elixir notes.

Elixir is a functional, meta-programming aware language built on top of the Erlang VM. It is a dynamic language that focuses on tooling to leverage Erlang’s abilities to build concurrent, distributed and fault-tolerant applications with hot code upgrades.

Full Post + Comments

Using Retrofit and RxJava to interact with web services on Android

Retrofit from Square, and RxJava from Netflix, are a great combo for interacting with web services on Android.

Retrofit

Retrofit is a REST client for Android and Java. It allows you to turn a REST API into a Java interface by using annotations to describe the HTTP requests. It can then generate an implementation of the interface for you. This means that you can go from:

GET /users/{userId}/posts

to:

webService.fetchUserPosts(userId)

in a few lines of code. Retrofit is very easy to use and it comes with RxJava integration.

RxJava

RxJava is a Java implementation of Rx, the Reactive Extensions library from the .NET world. It allows you to compose asynchronous and event-based programs in a declarative manner. Let’s say that you want to implement something like this:

  • Start observing our current location. When a location change happens, in parallel
    • Send a web service request A
    • Send a web service request B
      • Using the result from B, send a web service request C
  • When the results for both A and C are back, update the UI

RxJava allows you to write out your program pretty much as above. It abstracts out concerns about things like threading, synchronization, thread-safety, concurrent data structures, and non-blocking I/O. You can tell RxJava to observe the location changes and perform the web service requests in a background thread, and pass you the final results in the UI thread. If any exceptions are thrown at any point in the chain of events, you get told about it in one convenient place. You’re not drowning in a spaghetti of onSuccess and onError callbacks. You’re building pipelines.

I’m sold. Show me the code!

Sounds good? I’ve got some code to get you started. RexWeather is a simple app that demonstrates the use of Retrofit and RxJava on Android. The app queries the current location, then fetches the current weather and seven day forecast from OpenWeatherMap.

Screenshot of RexWeather

You can grab the Android Studio project from GitHub. The code is BSD-licensed, so feel free to use and share.

Full Post + Comments