Nick Ginther's Blog

Upgrading Zookeeper from 3.4.1 to 3.8.0 in place

Recently, I had to upgrade the local stack deploy of a Zookeeper 3.4.1 cluster to 3.8.0. The requirements were: upgrade in place, with no data loss, and with minimal notice from any users. This was not just as simple as replacing the binary because of changes between Zookeeper 3.4.1 and 3.5 require a special workaround.

There is a known issue with upgrading Zookeeper from 3.4 to 3.5+, where ZK 3.5 expects a snapshot file to exist in dataDir (the dir where ZK data is stored, containing a version-2 directory). ZK 3.4 doesn't necessarily always create snapshots if the ZK transaction volume is low, as there is a threshold value called snapCount that after which, a snapshot will be created. By default, this value is 10,000, which is a problem for local development, since most developers weren't hitting 10,000 transactions. Therefore, a snapshot was never created, and upgrading to ZK 3.8 was failing with the following error:

java.io.IOException: No snapshot found, but there are log entries. Something is broken!

This error is a valid exception, meant to be a safeguard from data corruption. However, during the upgrade from 3.4 to 3.5+, we need a way to skip this validation, since we know our data is valid.

To do this we need two things: a snapshot.0 file, and the zookeeper.snapshot.trust.empty Java property. In the dataDir directory, in the subdirectory with the transaction .log files (in my case, named version-2), I placed an empty file named snapshot.0. Next, when running Zookeeper 3.8, I added -Dzookeeper.snapshot.trust.empty to the start command, so the command ended up looking something like: 

java -Dzookeeper.snapshot.trust.empty=true -cp "...java jar path..." org.apache.zookeeper.server.quorum.QuorumPeerMain zoo.cfg

Once ZK 3.8 starts, it will automatically create a snapshot in dataDir, so in the service startup wrapper, I detected if there were snapshots present other than snapshot.0. If there were, I deleted snapshot.0 and removed -Dzookeeper.snapshot.trust.empty=true from the startup command.

This was a relatively simple fix, but it was interesting to automatically instrument a workaround and detect when it wasn't needed anymore. I learned about the workaround from the Apache Zookeeper Jira ticket for this issue: ZOOKEEPER-3056. Not much else to say here but keep your Zookeepers updated and your Solrs happy!