Zookeeper Zab一致性协议全流程详细解读

站长

2023年09月16日 14:30 · 阅读数 28

Zab协议的全称是 Zookeeper Atomic Broadcast （Zookeeper原子广播）

Zookeeper的Zab协议代码十分复杂，涉及多个实例和多线程的调用,通过Aop日志分析才弄明白。

ZAB协议流程

请求流程

Zookeeper Zab一致性协议全流程详细解读

核心步骤：

步骤一：将请求转发给leader，如果客户端所连的服务端是follwer节点

步骤二： leader发送Proposal，给所有follwer节点

步骤三：处理leader的Proposal，将提交放到pendingTxns，并且发送Ack请求

步骤四: 判断是否成功，发送commit请求,收到大于一半节点的ack则可以提交

步骤五：所有节点处理提交请求，修改数据库

具体处理逻辑在下面节点分析有更清楚的描述

查看节点状态

powershell命令

echo -----node1------;  echo stat | nc 127.0.0.1 2181 ;  echo -------node2-----; echo stat | nc 127.0.0.1 12181 ;  echo -------node3--------; echo stat | nc 127.0.0.1 22181 ;

状态结果

-----node1------
Zookeeper version: 3.9.1-SNAPSHOT-73ee39c034ea6eba2d46b0308c0fed68293e30c1-dirty, built on 2023-09-05 11:42 UTC
Clients:
 /127.0.0.1:12594[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: follower
Node count: 5

-------node2-----
Zookeeper version: 3.9.1-SNAPSHOT-73ee39c034ea6eba2d46b0308c0fed68293e30c1-dirty, built on 2023-09-05 11:42 UTC
Clients:
 /127.0.0.1:12596[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 014:
Connections: 1
Outstanding: 0
Zxid: 0x200000000
Mode: follower
Node count: 5

-------node3--------
Zookeeper version: 3.9.1-SNAPSHOT-73ee39c034ea6eba2d46b0308c0fed68293e30c1-dirty, built on 2023-09-05 11:42 UTC
Clients:
 /127.0.0.1:12598[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x200000000
Mode: leader
Node count: 5
Proposal sizes last/min/max: -1/-1/-1

分析

客户端连的是node1节点
node3是leader节点

执行分析

客户端执行,通过日志分析服务端的执行逻辑

create  /data11  1

日志通过aspectJ打印

public aspect ZookeeperAspect
{
    pointcut myPointCut(): execution ( public  * * (org.apache.zookeeper.server.Request));
    before() : myPointCut()
    {
        System.out.println("时间："+ java.time.LocalDateTime.now()+" 进入 " + thisJoinPoint  );
       if(thisJoinPoint.getArgs()!=null&&thisJoinPoint.getArgs().length>0){
          System.out.print(" 参数  ");
       for(Object  obj: thisJoinPoint.getArgs())
       {
          System.out.print(obj);
       }
        System.out.println(" 参数结尾  ");
        }
    }
}

node1

日志

时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.ZooKeeperServer.submitRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.ZooKeeperServer.enqueueRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.RequestThrottler.submitRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.ZooKeeperServer.submitRequestNow(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.quorum.FollowerRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(Request org.apache.zookeeper.server.quorum.QuorumZooKeeperServer.checkUpgradeSession(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(boolean org.apache.zookeeper.server.ZooKeeperServer.authWriteRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  14:45:23.960 [FollowerRequestProcessor:1] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11

时间：2023-09-05T19:45:23.973 进入 execution(void org.apache.zookeeper.server.SyncRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.ZKDatabase.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.SendAckRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.commit(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:12181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间：2023-09-05T19:45:23.976 进入 execution(DataTree.ProcessTxnResult org.apache.zookeeper.server.ZooKeeperServer.processTxn(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.common.PathTrie - data11
14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.DataTree - Digests are matching for Zxid: 200000002, Digest in log and actual tree: 4248210059
时间：2023-09-05T19:45:23.977 进入 execution(void org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾

分析

步骤一：

将请求转发给leader

通过ZooKeeperServer.processPacket 获取客户端的请求然后
- 调用followerRequestProcessor.processRequest
FollowerRequestProcessor处理,将请求提交给leader
- zks.getFollower().request(request); 请求提交给leader
然后使用CommitProcessor进行处理

步骤三：

处理leader的Proposal

QuorumPeer线程运行执行follower.followLeader();
通过follower.processPacket读取到leader的提案
- Leader.PROPOSAL类型
followerZooKeeperServer.logRequest添加到pendingTxns
SyncRequestProcessor 将已提交日志同步到磁盘
SendAckRequestProcessor 提交到磁盘后发送Ack日志

步骤五：

处理提交请求

与Proposal类似，使用 follower.processPacket处理
- Leader.COMMIT类型
- 处理pendingTxns的请求
commitProcessor
finalRequestProcessor
- 执行ZKDatabase.addCommittedProposal

node2

日志

时间：2023-09-05T19:45:23.973 进入 execution(void org.apache.zookeeper.server.SyncRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.ZKDatabase.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.SendAckRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.commit(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:12181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间：2023-09-05T19:45:23.976 进入 execution(DataTree.ProcessTxnResult org.apache.zookeeper.server.ZooKeeperServer.processTxn(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.common.PathTrie - data11
14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.DataTree - Digests are matching for Zxid: 200000002, Digest in log and actual tree: 4248210059
时间：2023-09-05T19:45:23.977 进入 execution(void org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾

分析

步骤三：

处理leader的Proposal

QuorumPeer线程运行执行follower.followLeader();
通过follower.processPacket读取到leader的提案
- Leader.PROPOSAL类型
followerZooKeeperServer.logRequest添加到pendingTxns
SyncRequestProcessor 将已提交日志同步到磁盘
SendAckRequestProcessor 提交到磁盘后发送Ack日志

步骤五：

处理提交请求

与Proposal类似，使用 follower.processPacket处理
- Leader.COMMIT类型
- 处理pendingTxns的请求
commitProcessor
finalRequestProcessor
- 执行ZKDatabase.addCommittedProposal

node3

日志

时间：2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.quorum.Leader.submitLearnerRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.quorum.LeaderZooKeeperServer.submitLearnerRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.PrepRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  14:45:23.963 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.SessionTrackerImpl - Checking session 0x100064ba8b80000
14:45:23.963 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.PrepRequestProcessor - Processing ACL: 31,s{'world,'anyone}
14:45:23.965 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.ZooKeeperServer - Permission requested: 4 
14:45:23.965 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.ZooKeeperServer - ACLs for node: [31,s{'world,'anyone}
14:45:23.965 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.ZooKeeperServer - Client credentials: ['ip,'127.0.0.1
14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.PrepRequestProcessor - Digest got from data tree is: 1371985504
14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.PrepRequestProcessor - Digest got from outstandingChanges is: 1442819401
时间：2023-09-05T19:45:23.972 进入 execution(void org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.972 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间：2023-09-05T19:45:23.972 进入 execution(Leader.Proposal org.apache.zookeeper.server.quorum.Leader.propose(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.quorum.Leader - Proposing:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间：2023-09-05T19:45:23.972 进入 execution(void org.apache.zookeeper.server.SyncRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.972 进入 execution(boolean org.apache.zookeeper.server.ZKDatabase.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.972 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.972 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.974 进入 execution(void org.apache.zookeeper.server.quorum.AckRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.commit(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.976 [LearnerHandler-/127.0.0.1:12551] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.Leader.ToBeAppliedRequestProcessor.processRequest(Request))
 参数  14:45:23.976 [LearnerHandler-/127.0.0.1:12565] DEBUG org.apache.zookeeper.server.quorum.Leader - outstanding is 0
sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.976 [CommitProcessor:3] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间：2023-09-05T19:45:23.976 进入 execution(DataTree.ProcessTxnResult org.apache.zookeeper.server.ZooKeeperServer.processTxn(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.976 [CommitProcessor:3] DEBUG org.apache.zookeeper.common.PathTrie - data11
14:45:23.976 [CommitProcessor:3] DEBUG org.apache.zookeeper.server.DataTree - Digests are matching for Zxid: 200000002, Digest in log and actual tree: 4248210059
时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾

分析

步骤二：

leader提交Proposal

从follower 获取请求 Leader.submitLearnerRequest
- LearnerHandler 线程处理follower的请求
- 调用了ProposalRequestProcessor.processRequest

ProposalRequestProcessor 使用的zxid进行提交数据

leader 通过 quorum.Leader.propose 提交提议

Leader.sendPacket 给发送提案包

void sendPacket(QuorumPacket qp) {
synchronized (forwardingFollowers) {
    for (LearnerHandler f : forwardingFollowers) {
        f.queuePacket(qp);
    }
}
}

调用syncRequestProcessor.processRequest

SyncRequestProcessor 将已提交日志同步到磁盘

步骤四:

判断是否成功，发送commit请求

LearnerHandler 接收到follwers的Leader.ACK请求
- Leader.processAck

Leader.tryToCommit方法尝试提交，判断Proposal是否可以提交，核心逻辑大于半数Ack则提交

public boolean hasAllQuorums() {
    for (QuorumVerifierAcksetPair qvAckset : qvAcksetPairs) {
        if (!qvAckset.getQuorumVerifier().containsQuorum(qvAckset.getAckset())) {
            return false;
        }
    }
    return true;
}
  public boolean containsQuorum(Set<Long> ackSet) {
    return (ackSet.size() > half);
}

发送Leader.COMMIT信息

步骤五：

处理提交请求

zk.commitProcessor.commit(p.request);
- commitProcessor
finalRequestProcessor
- 执行ZKDatabase.addCommittedProposal

转载自:https://juejin.cn/post/7275559201007403023