[clustering] when gateway lost connection to zookeeper, it didn't update ZookeeperNonBlockingLock's acquired or lockPath field, make a target get dup scribe when the gateway restored connection to zk

[yangyu66]

Bug description
Init setup:
two gateway instances (a and b) on different servers subscribe to the same target (with the same config), only one(a) gets the lock from zk and starts subscribing.

Produce bug:
when gateway a lost connection to Zookeeper, it didn't update its own ZookeeperNonBlockingLock's "acquired" or "lockPath" field, making a target get dup subscribe when the gateway restored connection to Zk

Step to reproduce
Steps to reproduce the behavior.

1. My config is

{
  "request": {
    "default": {
      "subscribe": {
        "prefix": {
        },
        "subscription": [
          {
            "path": {
              "elem": [
                {
                  "name": "interfaces"
                },
                {
                  "name": "interface",
                  "key": {
                    "name": "Ethernet100"
                  }
                },
                {
                  "name": "state"
                },
                {
                  "name": "counters"
                },
                {
                  "name": "in-discards"
                }
              ]
            },
            "mode": 2,
            "sample_interval": 30000000000,
            "heartbeat_interval": 30000000000
          }
        ]
      }
    }
  },
  "target": {
    "demo-router:8080": {
      "addresses": [
        "demo-router:8080"
      ],
      "credentials": {
        "username": "xx",
        "password": "xx"
      },
      "request": "default",
      "meta": {
        "NoTLS": "yes"
      }
    }
  }
}

3. on one server(a), Type './gnmi-gateway -EnableGNMIServer -ServerTLSCert=server.crt -ServerTLSKey=server.key -TargetLoaders=json -TargetJSONFile=targets.json -Exporters=kafka -ExporterKafkaTopic=xxx -ExporterKafkaBrokers=kafka:9092 -ExporterKafkaLogging -ZookeeperHosts zk:2181'

4. on another server(b), Type './gnmi-gateway -EnableGNMIServer -ServerTLSCert=server.crt -ServerTLSKey=server.key -TargetLoaders=json -TargetJSONFile=targets.json -Exporters=kafka -ExporterKafkaTopic=xxx -ExporterKafkaBrokers=kafka:9092 -ExporterKafkaLogging -ZookeeperHosts zk:2181'

5. Then break the connection between server a and zk

6. Server b starts subscribing to the target. (expected)

7. Restore connection between server a and zk

8. Server a starts subscribing to the target (bug)

Expected behavior
in step 8, Server a should not start subscribing to the target

Output or code snippets
in connections/zookeeper.go,
func (c *ZookeeperConnectionManager) eventListener(zkEvents <-chan zk.Event) {}, targetConfig.unlock(), , it should also clear/correct distributed lock status.

[yangyu66]

I think in

func (c *ZookeeperConnectionManager) eventListener(zkEvents <-chan zk.Event) {
....
case zk.StateDisconnected: ...
}

should add some code to clear DistributedLocker's "acquired" and "lockPath" fields, as
targetConfig.unlock() looks only to cancel the context to the target, but once the connection to zk is restored, it will get lock already acquired (even though with deadlock error, in connections/state.go t.config.Log.Error().Msgf("Target %s: error while trying to acquire lock: %v", t.name, err), get deadlock error but still mark lock acquired after connection to zk restored), and start subscribing to target.
I'm trying to create a pr to fix this issue, should publish soon