Scaling Out ZooKeeper

Posted on In Views: Word count in article: 1.3k Reading time ≈ 5 mins.
A hands-on guide to scaling ZooKeeper across data centers in the same city: expanding a single-DC three-node cluster into a 2-2-1 layout, adding nodes step by step, retiring the old ones, and controlling the risk of leader changes.

Background

Because Alibaba regularly runs network-isolation drills on its data centers, a ZooKeeper cluster deployed in a single data center cannot serve other data centers when that data center is cut off from the network. We therefore need to upgrade a single-DC ZooKeeper deployment to a multi-DC one. However, ZooKeeper uses strong synchronization—every request is synchronized internally—so if the latency between machines is high, ZooKeeper runs into all sorts of problems. The prerequisite for this solution is therefore that the multiple data centers are in the same city, with low latency between them.

This solution also works for ZooKeeper upgrades, scale-out, and machine replacement.

In a multi-DC setup, you typically have three data centers. In that case, a 2-2-1 distribution is recommended, deploying one extra ZooKeeper node in the data center that has more clients.

Problem to Solve

The current cluster has 3 ZooKeeper machines. We now need to add 3 more machines and retire one of the original 3.

Steps

  • Scale out with new ZooKeeper nodes
  • Retire the unneeded ZooKeeper node
  • Update all ZooKeeper nodes

Changing a ZooKeeper cluster actually requires great care: a mistake can take the entire cluster out of service and cause widespread program failures.
So the idea is to add machines gradually and minimize leader changes as much as possible.

Scaling Out with New ZooKeeper Nodes

When scaling out, the configuration of a newly added machine differs from the running ZooKeeper configuration by just one extra machine, which guarantees that the cluster leader does not change at all.

Old ZooKeeper configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
maxClientCnxns=300
dataDir=/dev/shm/zk/data
dataLogDir=/dev/shm/zk/logs

# the port at which the clients will connect
clientPort=2181
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=5
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

#minSessionTimeout=10000
minSessionTimeout=100
maxSessionTimeout=100000

server.1=zkA.jstorm.alibaba.com:2888:3888
server.2=zkB.jstorm.alibaba.com:2888:3888
server.3=zkC.jstorm.alibaba.com:2888:3888

Adding D

Configuration for the new node D.

The new configuration after scaling out is:

1
2
3
4
5
## Other settings are the same as the old configuration ##
server.1=zkA.jstorm.alibaba.com:2888:3888
server.2=zkB.jstorm.alibaba.com:2888:3888
server.3=zkC.jstorm.alibaba.com:2888:3888
server.4=zkD.jstorm.alibaba.com:2888:3888

A few things to note:

  • Remember to create the id file under the /dev/shm/zk/data directory.
  • In this example, the ZooKeeper directory is placed in shared memory, so you need a cron job that runs every minute to sync the new incremental data files to the local disk and delete the stale files on the local disk.
  • The id of a retired machine is retained and not overwritten, to avoid data corruption.
  • After adding D, make sure A/B/C/D ZooKeeper are all serving and the cluster has exactly one leader. If not, you need to redo the step. There are many ways to check; here is one:
1
2
3
4
5
6
7
8
9
10
 ~  echo srvr | nc zkD.jstorm.alibaba.com 2181
Zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT
Latency min/avg/max: 0/0/13432
Received: ***
Sent: ***
Connections: ***
Outstanding: 0
Zxid: 0x***
Mode: follower
Node count: ***

Adding E

Configuration for the new node E.

1
2
3
4
5
6
## Other settings are the same as the old configuration ##
server.1=zkA.jstorm.alibaba.com:2888:3888
server.2=zkB.jstorm.alibaba.com:2888:3888
server.3=zkC.jstorm.alibaba.com:2888:3888
server.4=zkD.jstorm.alibaba.com:2888:3888
server.5=zkE.jstorm.alibaba.com:2888:3888

A few things to note:

  • Remember to create the id file under the /dev/shm/zk/data directory.
  • In this example, the ZooKeeper directory is placed in shared memory, so you need a cron job that runs every minute to sync the new incremental data files to the local disk and delete the stale files on the local disk.
  • The id of a retired machine is retained and not overwritten, to avoid data corruption.
  • After adding E, make sure A/B/C/D/E ZooKeeper are all serving and the cluster has exactly one leader.

Adding F

Configuration for the new node F.

1
2
3
4
5
6
7
## Other settings are the same as the old configuration ##
server.1=zkA.jstorm.alibaba.com:2888:3888
server.2=zkB.jstorm.alibaba.com:2888:3888
server.3=zkC.jstorm.alibaba.com:2888:3888
server.4=zkD.jstorm.alibaba.com:2888:3888
server.5=zkE.jstorm.alibaba.com:2888:3888
server.6=zkF.jstorm.alibaba.com:2888:3888

A few things to note:

  • Remember to create the id file under the /dev/shm/zk/data directory.
  • In this example, the ZooKeeper directory is placed in shared memory, so you need a cron job that runs every minute to sync the new incremental data files to the local disk and delete the stale files on the local disk.
  • The id of a retired machine is retained and not overwritten, to avoid data corruption.
  • After adding F, make sure every ZooKeeper node is serving.

Updating D’s Configuration

After updating D’s configuration, restart D’s ZooKeeper and check that all ZooKeeper nodes are healthy.

1
2
3
4
5
6
7
## Other settings are the same as the old configuration ##
server.1=zkA.jstorm.alibaba.com:2888:3888
server.2=zkB.jstorm.alibaba.com:2888:3888
server.3=zkC.jstorm.alibaba.com:2888:3888
server.4=zkD.jstorm.alibaba.com:2888:3888
server.5=zkE.jstorm.alibaba.com:2888:3888
server.6=zkF.jstorm.alibaba.com:2888:3888

Updating E’s Configuration

After updating E’s configuration, restart E’s ZooKeeper and check that all ZooKeeper nodes are healthy.

1
2
3
4
5
6
7
## Other settings are the same as the old configuration ##
server.1=zkA.jstorm.alibaba.com:2888:3888
server.2=zkB.jstorm.alibaba.com:2888:3888
server.3=zkC.jstorm.alibaba.com:2888:3888
server.4=zkD.jstorm.alibaba.com:2888:3888
server.5=zkE.jstorm.alibaba.com:2888:3888
server.6=zkF.jstorm.alibaba.com:2888:3888

Updating the Old ZooKeeper Configuration

Steps:

  • Check the old cluster to determine which node is the leader and which are followers.
  • Update the ZooKeeper configuration.
  • Restart ZooKeeper.
  • Verify that the restarted ZooKeeper can serve.

A few things to note:

  • You must still operate one machine at a time—only move on to the next after the current one is done.
  • The configuration file now lists 6 machines, not 5.
  • Throughout the change, none of the ZooKeeper nodes’ roles should change at all.

In this example, assuming C is the leader, we first change A, and after that succeeds, we change B.

The configuration file for this step is:

1
2
3
4
5
6
7
## Other settings are the same as the old configuration ##
server.1=zkA.jstorm.alibaba.com:2888:3888
server.2=zkB.jstorm.alibaba.com:2888:3888
server.3=zkC.jstorm.alibaba.com:2888:3888
server.4=zkD.jstorm.alibaba.com:2888:3888
server.5=zkE.jstorm.alibaba.com:2888:3888
server.6=zkF.jstorm.alibaba.com:2888:3888

Taking Down the Old Leader

  • Kill the old ZooKeeper leader.
  • Check all ZooKeeper nodes to make sure they can all serve.

Updating the Configuration on All Machines

The configuration file for this step is:

1
2
3
4
5
6
## Other settings are the same as the old configuration ##
server.1=zkA.jstorm.alibaba.com:2888:3888
server.2=zkB.jstorm.alibaba.com:2888:3888
server.4=zkD.jstorm.alibaba.com:2888:3888
server.5=zkE.jstorm.alibaba.com:2888:3888
server.6=zkF.jstorm.alibaba.com:2888:3888

The new configuration file removes one ZooKeeper machine—the old leader.

A few things to note:

  • You must still operate one machine at a time—only move on to the next after the current one is done.
  • The configuration file lists 5 machines. The new configuration file must also be synced to the retired machine C, to prevent C from being mistakenly started later.
  • During the change, the machine hosting the leader must be changed last, which reduces the number of leader elections by one.