likes
comments
collection
share

Zookeeper Zab一致性协议全流程详细解读

作者站长头像
站长
· 阅读数 30

Zab协议 的全称是 Zookeeper Atomic Broadcast （Zookeeper原子广播）

Zookeeper的Zab协议代码十分复杂，涉及多个实例和多线程的调用,通过Aop日志分析才弄明白。

ZAB协议流程

请求流程

Zookeeper Zab一致性协议全流程详细解读

核心步骤：

步骤一： 将请求转发给leader，如果客户端所连的服务端是follwer节点

步骤二： leader发送Proposal，给所有follwer节点

步骤三： 处理leader的Proposal，将提交放到pendingTxns，并且发送Ack请求

步骤四: 判断是否成功，发送commit请求,收到大于一半节点的ack则可以提交

步骤五： 所有节点处理提交请求，修改数据库

具体处理逻辑在下面节点分析有更清楚的描述

查看节点状态

powershell命令

echo -----node1------;  echo stat | nc 127.0.0.1 2181 ;  echo -------node2-----; echo stat | nc 127.0.0.1 12181 ;  echo -------node3--------; echo stat | nc 127.0.0.1 22181 ;

状态结果

-----node1------
Zookeeper version: 3.9.1-SNAPSHOT-73ee39c034ea6eba2d46b0308c0fed68293e30c1-dirty, built on 2023-09-05 11:42 UTC
Clients:
 /127.0.0.1:12594[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: follower
Node count: 5

-------node2-----
Zookeeper version: 3.9.1-SNAPSHOT-73ee39c034ea6eba2d46b0308c0fed68293e30c1-dirty, built on 2023-09-05 11:42 UTC
Clients:
 /127.0.0.1:12596[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 014:
Connections: 1
Outstanding: 0
Zxid: 0x200000000
Mode: follower
Node count: 5

-------node3--------
Zookeeper version: 3.9.1-SNAPSHOT-73ee39c034ea6eba2d46b0308c0fed68293e30c1-dirty, built on 2023-09-05 11:42 UTC
Clients:
 /127.0.0.1:12598[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x200000000
Mode: leader
Node count: 5
Proposal sizes last/min/max: -1/-1/-1

分析

  • 客户端连的是node1节点
  • node3是leader节点

执行分析

客户端执行,通过日志分析服务端的执行逻辑

create  /data11  1

日志通过aspectJ打印

public aspect ZookeeperAspect
{
    pointcut myPointCut(): execution ( public  * * (org.apache.zookeeper.server.Request));
    before() : myPointCut()
    {
        System.out.println("时间："+ java.time.LocalDateTime.now()+" 进入 " + thisJoinPoint  );
       if(thisJoinPoint.getArgs()!=null&&thisJoinPoint.getArgs().length>0){
          System.out.print(" 参数  ");
       for(Object  obj: thisJoinPoint.getArgs())
       {
          System.out.print(obj);
       }
        System.out.println(" 参数结尾  ");
        }
    }
}

node1

日志

时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.ZooKeeperServer.submitRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.ZooKeeperServer.enqueueRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.RequestThrottler.submitRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.ZooKeeperServer.submitRequestNow(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.quorum.FollowerRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(Request org.apache.zookeeper.server.quorum.QuorumZooKeeperServer.checkUpgradeSession(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(boolean org.apache.zookeeper.server.ZooKeeperServer.authWriteRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  14:45:23.960 [FollowerRequestProcessor:1] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11

时间：2023-09-05T19:45:23.973 进入 execution(void org.apache.zookeeper.server.SyncRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.ZKDatabase.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.SendAckRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.commit(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:12181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间：2023-09-05T19:45:23.976 进入 execution(DataTree.ProcessTxnResult org.apache.zookeeper.server.ZooKeeperServer.processTxn(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.common.PathTrie - data11
14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.DataTree - Digests are matching for Zxid: 200000002, Digest in log and actual tree: 4248210059
时间：2023-09-05T19:45:23.977 进入 execution(void org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾

分析

步骤一：

将请求转发给leader

  • 通过ZooKeeperServer.processPacket 获取客户端的请求然后
    • 调用followerRequestProcessor.processRequest
  • FollowerRequestProcessor处理,将请求提交给leader
    • zks.getFollower().request(request); 请求提交给leader
  • 然后使用CommitProcessor进行处理

步骤三：

处理leader的Proposal

  • QuorumPeer线程运行执行follower.followLeader();
  • 通过follower.processPacket读取到leader的提案
    • Leader.PROPOSAL类型
  • followerZooKeeperServer.logRequest添加到pendingTxns
  • SyncRequestProcessor 将已提交日志同步到磁盘
  • SendAckRequestProcessor 提交到磁盘后发送Ack日志

步骤五：

处理提交请求

  • 与Proposal类似，使用 follower.processPacket处理
    • Leader.COMMIT类型
    • 处理pendingTxns的请求
  • commitProcessor
  • finalRequestProcessor
    • 执行ZKDatabase.addCommittedProposal

node2

日志

时间：2023-09-05T19:45:23.973 进入 execution(void org.apache.zookeeper.server.SyncRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.ZKDatabase.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.SendAckRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.commit(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:12181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间：2023-09-05T19:45:23.976 进入 execution(DataTree.ProcessTxnResult org.apache.zookeeper.server.ZooKeeperServer.processTxn(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾  14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.common.PathTrie - data11
14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.DataTree - Digests are matching for Zxid: 200000002, Digest in log and actual tree: 4248210059
时间：2023-09-05T19:45:23.977 进入 execution(void org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾

分析

步骤三：

处理leader的Proposal

  • QuorumPeer线程运行执行follower.followLeader();
  • 通过follower.processPacket读取到leader的提案
    • Leader.PROPOSAL类型
  • followerZooKeeperServer.logRequest添加到pendingTxns
  • SyncRequestProcessor 将已提交日志同步到磁盘
  • SendAckRequestProcessor 提交到磁盘后发送Ack日志

步骤五：

处理提交请求

  • 与Proposal类似，使用 follower.processPacket处理
    • Leader.COMMIT类型
    • 处理pendingTxns的请求
  • commitProcessor
  • finalRequestProcessor
    • 执行ZKDatabase.addCommittedProposal

node3

日志

时间：2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.quorum.Leader.submitLearnerRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.quorum.LeaderZooKeeperServer.submitLearnerRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.PrepRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾  14:45:23.963 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.SessionTrackerImpl - Checking session 0x100064ba8b80000
14:45:23.963 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.PrepRequestProcessor - Processing ACL: 31,s{'world,'anyone}
14:45:23.965 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.ZooKeeperServer - Permission requested: 4 
14:45:23.965 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.ZooKeeperServer - ACLs for node: [31,s{'world,'anyone}
14:45:23.965 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.ZooKeeperServer - Client credentials: ['ip,'127.0.0.1
14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.PrepRequestProcessor - Digest got from data tree is: 1371985504
14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.PrepRequestProcessor - Digest got from outstandingChanges is: 1442819401
时间：2023-09-05T19:45:23.972 进入 execution(void org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.972 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间：2023-09-05T19:45:23.972 进入 execution(Leader.Proposal org.apache.zookeeper.server.quorum.Leader.propose(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.quorum.Leader - Proposing:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间：2023-09-05T19:45:23.972 进入 execution(void org.apache.zookeeper.server.SyncRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.972 进入 execution(boolean org.apache.zookeeper.server.ZKDatabase.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.972 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.972 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnLog.append(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.974 进入 execution(void org.apache.zookeeper.server.quorum.AckRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.commit(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.976 [LearnerHandler-/127.0.0.1:12551] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.Leader.ToBeAppliedRequestProcessor.processRequest(Request))
 参数  14:45:23.976 [LearnerHandler-/127.0.0.1:12565] DEBUG org.apache.zookeeper.server.quorum.Leader - outstanding is 0
sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.976 [CommitProcessor:3] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间：2023-09-05T19:45:23.976 进入 execution(DataTree.ProcessTxnResult org.apache.zookeeper.server.ZooKeeperServer.processTxn(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾  14:45:23.976 [CommitProcessor:3] DEBUG org.apache.zookeeper.common.PathTrie - data11
14:45:23.976 [CommitProcessor:3] DEBUG org.apache.zookeeper.server.DataTree - Digests are matching for Zxid: 200000002, Digest in log and actual tree: 4248210059
时间：2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(Request))
 参数  sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾

分析

步骤二：

leader提交Proposal

  • 从follower 获取请求 Leader.submitLearnerRequest
    • LearnerHandler 线程处理follower的请求
    • 调用了ProposalRequestProcessor.processRequest
  • ProposalRequestProcessor 使用的zxid进行提交数据
    • leader 通过 quorum.Leader.propose 提交提议
    • Leader.sendPacket 给发送提案包 
      void sendPacket(QuorumPacket qp) {
synchronized (forwardingFollowers) {
    for (LearnerHandler f : forwardingFollowers) {
        f.queuePacket(qp);
    }
}
}
    • 调用syncRequestProcessor.processRequest
  • SyncRequestProcessor 将已提交日志同步到磁盘

步骤四:

判断是否成功，发送commit请求

  • LearnerHandler 接收到follwers的Leader.ACK请求

    • Leader.processAck

  • Leader.tryToCommit方法尝试提交，判断Proposal是否可以提交，核心逻辑大于半数Ack则提交

    public boolean hasAllQuorums() {
    for (QuorumVerifierAcksetPair qvAckset : qvAcksetPairs) {
        if (!qvAckset.getQuorumVerifier().containsQuorum(qvAckset.getAckset())) {
            return false;
        }
    }
    return true;
}
  public boolean containsQuorum(Set<Long> ackSet) {
    return (ackSet.size() > half);
}

  • 发送Leader.COMMIT信息

步骤五：

处理提交请求

  • zk.commitProcessor.commit(p.request);
    • commitProcessor
  • finalRequestProcessor
    • 执行ZKDatabase.addCommittedProposal
转载自:https://juejin.cn/post/7275559201007403023
评论
请登录