How to read or dump the Zookeeper Transaction Log and Snapshots

How to read or dump the Zookeeper Transaction Log and Snapshots

Goal:

How to read or dump the Zookeeper Transaction Log and Snapshots.

Env:

Zookeeper 3.4.5 on MapR 5.1

Solution:

When troubleshooting, we may want to dig into the Zookeeper Transaction Log and Snapshots which are in binary.
If we want to get most readable information out of them, here are the commands:

1. Zookeeper Transaction Log

1	java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.LogFormatter log.100000001

The location of the Zookeeper transaction log by default on mapr env is:
/opt/mapr/zkdata/version-2
The output will print how many transactions are recorded in this Zookeeper Transaction Log in the end, for example:

1	EOF reached after 1404 txns.

To double confirm that, you can also "grep session" from the output, and count it by yourself:

1 2	#  java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.LogFormatter log.100000001  |grep session|wc -l 1404

Or if you want to know which Hadoop Ecosystem has the most calls to Zookeeper, one way is to check the "SetData" calls. For example:

1 2 3 4 5 6 7 8 9 10 11

#  java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.LogFormatter log.100000001|grep session|grep setData|awk '{print $12}'|awk -F/ '{print $3}'|sort|uniq -c|sort -n      15 historyserver      19 spark-historyserver      24 resourcemanager      26 controlnodes      26 webserver      39 hoststats      39 kvstore      56 cldb      57 nodemanager      84 nfs

 2. Snapshots

1	java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.SnapshotFormatter snapshot.0

The output will print the znodes list at that time, you can just "grep '/'" to get them, for example:

1 2 3 4 5 6 7 8

# java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.SnapshotFormatter snapshot.10000057c|grep '/' / /historyserver_locks /nodes /nodes/v5.poc.com /nodes/v5.poc.com/services ...