issue 235: kafka consumer and publisher for multiple brokers

[kotronis-te]

This PR adds support for multiple brokers using sarama ClusterAdmin interface.
It includes (currently in draft) docker-compose files to initiate a local installation with a kafka cluster and zookeeper.

Closes #235

[sbezverk]

Please fix possible panic.

[kotronis-te]

Hmmm I will turn to undraft when I get messages to kafka. I added sample prefixes to the XR's but I cannot seem to get sth in the single- or multi-broker case even with the initial gobmp container, looking at gobmp.parsed.unicast_prefix_v4 with console consumer. Any ideas on this?

[sbezverk]

@kotronis-te The issue was with kafka client/metadata configuration, see my test PR for diff.

[kotronis-te]

@kotronis-te The issue was with kafka client/metadata configuration, see my test PR for diff.

I am also getting the following on ios xr side:

xr-1  | RP/0/RP0/CPU0:Nov 29 08:33:14.447 UTC: smartlicserver[123]: %LICENSE-SMART_LIC-6-EXPORT_CONTROLLED : Usage of export controlled features is not allowed
xr-1  | RP/0/RP0/CPU0:Nov 29 08:33:14.479 UTC: smartlicserver[123]: %LICENSE-SMART_LIC-3-COMM_FAILED : Communications failure with the Cisco Smart Software Manager (CSSM) : Communications init failure

[kotronis-te]

Hmmm I think I am on to sth, the admin seems to want to create aux topics that have replication factor of 3 (minimum), which will not work in the single-broker case. Checking and will come back to this.

testbed-kafka-1-1  | [2023-11-29 08:44:04,397] INFO [Admin Manager on Broker 1]: Error processing create topic request CreatableTopic(name='__consumer_offsets', numPartitions=50, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='compression.type', value='producer'), CreateableTopicConfig(name='cleanup.policy', value='compact'), CreateableTopicConfig(name='segment.bytes', value='104857600')]) (kafka.server.ZkAdminManager)
testbed-kafka-1-1  | org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

In the meanwhile, I will integrate your changes.

[kotronis-te]

config updates in 3c7fbf3
kafka offset fix in 747c983

Now the main issue that remains is with the XRs, will consider using a different router like gobgp to test the setup.

[sbezverk]

@kotronis-te you can safely ignore these:

xr-1  | RP/0/RP0/CPU0:Nov 29 08:33:14.447 UTC: smartlicserver[123]: %LICENSE-SMART_LIC-6-EXPORT_CONTROLLED : Usage of export controlled features is not allowed
xr-1  | RP/0/RP0/CPU0:Nov 29 08:33:14.479 UTC: smartlicserver[123]: %LICENSE-SMART_LIC-3-COMM_FAILED : Communications failure with the Cisco Smart Software Manager (CSSM) : Communications init failure

we are using xrd in eval mode.

[kotronis-te]

@kotronis-te you can safely ignore these:

xr-1  | RP/0/RP0/CPU0:Nov 29 08:33:14.447 UTC: smartlicserver[123]: %LICENSE-SMART_LIC-6-EXPORT_CONTROLLED : Usage of export controlled features is not allowed
xr-1  | RP/0/RP0/CPU0:Nov 29 08:33:14.479 UTC: smartlicserver[123]: %LICENSE-SMART_LIC-3-COMM_FAILED : Communications failure with the Cisco Smart Software Manager (CSSM) : Communications init failure

we are using xrd in eval mode.

OK will ping when I switch the testbed I use to gobgp which I had tested also in the past

[sbezverk]

Not sure if it is worth it to switch to gobgp, as your change with a single node kafka is green, all expected messages for unicast v4 are received. Have nothing against using gobgp in future in ci/cd pipeline, just now let's try to keep less changes.

[kotronis-te]

Not sure if it is worth it to switch to gobgp, as your change with a single node kafka is green, all expected messages for unicast v4 are received. Have nothing against using gobgp in future in ci/cd pipeline, just now let's try to keep less changes.

No worries it will be a separate compose and configuration, used for testing. Will ping soon when tested.

[kotronis-te]

Fixed in 49044c9
seems to work like a charm

{"action":"add","router_hash":"87d86e0d3cf65f69738c696c5ab800b4","router_ip":"10.1.1.4","base_attrs":{"base_attr_hash":"37ec0eb8db18c3725ef01c705b9b9c85","origin":"incomplete","as_path":[1111],"as_path_count":1,"nexthop":"10.1.1.3","is_atomic_agg":false},"peer_hash":"934b7a4989b7ac8024ddaf4c50cd3185","peer_ip":"10.1.1.3","peer_type":0,"peer_asn":1111,"timestamp":"2023-11-29T09:40:40Z","prefix":"192.168.3.0","prefix_len":24,"is_ipv4":true,"origin_as":1111,"nexthop":"10.1.1.3","is_nexthop_ipv4":true,"is_adj_rib_in_post_policy":false,"is_adj_rib_out_post_policy":false,"is_loc_rib_filtered":false}

Opening the PR for review, let me know what to edit when available.

[kotronis-te]

@sbezverk I will turn to draft and work on the restructuring you propose so that this is better integrated with the CI.

[kotronis-te]

I am facing some issues with the current CI, will un-draft when resolved.

[kotronis-te]

@sbezverk I am running the CI/CD commands exactly as they are in the GH Action workflow:

    make container REGISTRY_NAME=localhost IMAGE_VERSION=cicd
    make validator
    docker pull sbezverk/xrd-control-plane:7.9.2
    docker compose -f ./build/unicastv4-bgp/test_bed.yml up --detach
   ./build/monitor_container.sh gobmp
   ./bin/validator --validate=true --kafka=127.0.0.1:9092 --msg-file=./testdata/validator/ipv4_unicast.msg

however even for the classic workflow I get:

I1220 14:20:59.326701   68933 kafka-consumer.go:35] NewKafkaConsumer
I1220 14:20:59.330829   68933 check.go:39] For message type 74, 3 messages found
I1220 14:20:59.331051   68933 check.go:76] Dictionary for topic type 74 contains 3 test messages
E1220 14:21:09.331936   68933 validator.go:96] timed out waiting for the test to complete

Any ideas? You had said at one point that you had verified the CI with the new changes, can you let me know what you did please and how the timeout was remedied?

This is blocking from testing also the other new workflows I have added, since I still get the timeout

[sbezverk]

@kotronis-te from the first look, it seems ok, but we need to test it in gihub cicd, try to add this code at the end of cicd.yml:

      - name: stop unicast v4 integration test
         run: docker compose -f ./build/unicastv4-bgp/test_bed.yml down     
      - name: starts kafka multi broker
        run: docker compose -f ./build/kafka-single-multi-broker/test_bed_multi_broker.yaml up --detach
      - name: check conection to kafka and bmp session with a router
        id: session2
        run: ./build/monitor_container.sh gobmp
        continue-on-error: true
      - name: Check on failures
        if: steps.session2.outcome != 'success'
        run: |
          docker ps -a
          docker logs gobmp
          docker logs xr-1
          exit 1 
      - name: stop kafka multi broker
        run: docker compose -f ./build/kafka-single-multi-broker/test_bed_multi_broker.yaml down
      - name: starts kafka single broker
        run: docker compose -f ./build/kafka-single-multi-broker/test_bed_one_broker.yaml up --detach
      - name: check conection to kafka and bmp session with a router
        id: session3
        run: ./build/monitor_container.sh gobmp
        continue-on-error: true
      - name: Check on failures
        if: steps.session3.outcome != 'success'
        run: |
          docker ps -a
          docker logs gobmp
          docker logs xr-1
          exit 1 
      - name: stop kafka single broker
        run: docker compose -f ./build/kafka-single-multi-broker/test_bed_one_broker.yaml down

Let's see if your scripts work in cicd.

[kotronis-te]

Closing, will re-initiate PR from other official repo for better software management.