Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reset-offsets: add --delete option #169

Closed
wants to merge 4 commits into from
Closed

Conversation

extemporalgenome
Copy link

@extemporalgenome extemporalgenome commented Dec 11, 2023

  • Support --delete option for reset-offsets command.
  • Support --before-earliest and --after-latest partition selection flags.
  • Simplifications and lint fixes.

@extemporalgenome extemporalgenome requested a review from a team as a code owner December 11, 2023 07:41
@extemporalgenome
Copy link
Author

Note, t is just a topicctl alias pointing at my own compilation of topicctl (from this branch) running in a kafka-topics container.

bash-5.0# t reset-offsets identified-messages kgillette-test --offset 0 --partitions 0,14,23
[2023-12-12 21:56:30]  INFO Checking cluster ID against version in cluster
[2023-12-12 21:56:32]  INFO This will reset the offsets for the following partitions in topic identified-messages for group kgillette-test:
------------+-------------
  PARTITION | NEW OFFSET  
------------+-------------
  0         | 0           
  14        | 0           
  23        | 0           
------------+-------------
[2023-12-12 21:56:32]  INFO Please ensure that all other consumers are stopped, otherwise the reset might be overridden.
OK to continue? (yes/no) yes
[2023-12-12 21:56:39]  INFO Success
bash-5.0# t get lags identified-messages kgillette-test 2>&1 | head -n 30
time="2023-12-12 21:58:15" level=info msg="Checking cluster ID against version in cluster"
time="2023-12-12 21:58:17" level=info msg="Group member lags:
------------+-----------+---------------+----------------------+---------------+----------------------+------------+-----------
  PARTITION | MEMBER ID | MEMBER OFFSET |     MEMBER TIME      | LATEST OFFSET |     LATEST TIME      | OFFSET LAG | TIME LAG  
------------+-----------+---------------+----------------------+---------------+----------------------+------------+-----------
  0         |           | 0             |                      | 27380405      | 2023-12-12T21:58:13Z | 27380405   |           
  1         |           | 27412567      | 2023-12-12T21:55:51Z | 27413939      | 2023-12-12T21:58:15Z | 1372       | 144s      
  2         |           | 27428998      | 2023-12-12T21:55:51Z | 27430375      | 2023-12-12T21:58:15Z | 1377       | 144s      
  3         |           | 27424885      | 2023-12-12T21:55:50Z | 27426290      | 2023-12-12T21:58:16Z | 1405       | 145s      
  4         |           | 27409576      | 2023-12-12T21:55:51Z | 27410979      | 2023-12-12T21:58:14Z | 1403       | 143s      
  5         |           | 27413931      | 2023-12-12T21:55:51Z | 27415343      | 2023-12-12T21:58:15Z | 1412       | 144s      
  6         |           | 27420568      | 2023-12-12T21:55:51Z | 27421977      | 2023-12-12T21:58:15Z | 1409       | 144s      
  7         |           | 27378912      | 2023-12-12T21:55:51Z | 27380324      | 2023-12-12T21:58:16Z | 1412       | 144s      
  8         |           | 27396003      | 2023-12-12T21:55:52Z | 27397415      | 2023-12-12T21:58:16Z | 1412       | 143s      
  9         |           | 27370922      | 2023-12-12T21:55:52Z | 27372334      | 2023-12-12T21:58:16Z | 1412       | 143s      
  10        |           | 27375087      | 2023-12-12T21:55:52Z | 27376499      | 2023-12-12T21:58:16Z | 1412       | 143s      
  11        |           | 27371684      | 2023-12-12T21:55:52Z | 27373096      | 2023-12-12T21:58:16Z | 1412       | 143s      
  12        |           | 27381118      | 2023-12-12T21:55:52Z | 27382531      | 2023-12-12T21:58:16Z | 1413       | 143s      
  13        |           | 27385899      | 2023-12-12T21:55:52Z | 27387301      | 2023-12-12T21:58:13Z | 1402       | 140s      
  14        |           | 0             |                      | 27396588      | 2023-12-12T21:58:13Z | 27396588   |           
  15        |           | 27370625      | 2023-12-12T21:55:52Z | 27372027      | 2023-12-12T21:58:13Z | 1402       | 140s      
  16        |           | 27373673      | 2023-12-12T21:55:53Z | 27375075      | 2023-12-12T21:58:13Z | 1402       | 140s      
  17        |           | 27373152      | 2023-12-12T21:55:53Z | 27374554      | 2023-12-12T21:58:13Z | 1402       | 140s      
  18        |           | 27378605      | 2023-12-12T21:55:53Z | 27380008      | 2023-12-12T21:58:13Z | 1403       | 140s      
  19        |           | 27379616      | 2023-12-12T21:55:53Z | 27381028      | 2023-12-12T21:58:16Z | 1412       | 142s      
  20        |           | 27382512      | 2023-12-12T21:55:45Z | 27383915      | 2023-12-12T21:58:13Z | 1403       | 148s      
  21        |           | 27383199      | 2023-12-12T21:55:54Z | 27384601      | 2023-12-12T21:58:15Z | 1402       | 140s      
  22        |           | 27358526      | 2023-12-12T21:55:54Z | 27359928      | 2023-12-12T21:58:15Z | 1402       | 141s      
  23        |           | 0             |                      | 27371349      | 2023-12-12T21:58:15Z | 27371349   |           
  24        |           | 27393145      | 2023-12-12T21:55:54Z | 27394537      | 2023-12-12T21:58:14Z | 1392       | 140s      
bash-5.0# t reset-offsets identified-messages kgillette-test --delete --before-earliest
[2023-12-12 21:58:51]  INFO Checking cluster ID against version in cluster
[2023-12-12 21:58:53]  INFO This will delete the offsets for the following partitions in topic identified-messages for group kgillette-test:
-------------
  PARTITION  
-------------
          0  
         14  
         23  
-------------
OK to continue? (yes/no) yes
[2023-12-12 21:58:57]  INFO Success
bash-5.0# t get lags identified-messages kgillette-test 2>&1 | head -n 30
time="2023-12-12 21:59:19" level=info msg="Checking cluster ID against version in cluster"
time="2023-12-12 21:59:22" level=info msg="Group member lags:
------------+-----------+---------------+----------------------+---------------+----------------------+------------+-----------
  PARTITION | MEMBER ID | MEMBER OFFSET |     MEMBER TIME      | LATEST OFFSET |     LATEST TIME      | OFFSET LAG | TIME LAG  
------------+-----------+---------------+----------------------+---------------+----------------------+------------+-----------
  0         |           | -1            |                      | 27381327      | 2023-12-12T21:59:19Z | 27381328   |           
  1         |           | 27412567      | 2023-12-12T21:55:51Z | 27414879      | 2023-12-12T21:59:20Z | 2312       | 209s      
  2         |           | 27428998      | 2023-12-12T21:55:51Z | 27431300      | 2023-12-12T21:59:19Z | 2302       | 208s      
  3         |           | 27424885      | 2023-12-12T21:55:50Z | 27427246      | 2023-12-12T21:59:21Z | 2361       | 210s      
  4         |           | 27409576      | 2023-12-12T21:55:51Z | 27411911      | 2023-12-12T21:59:19Z | 2335       | 207s      
  5         |           | 27413931      | 2023-12-12T21:55:51Z | 27416275      | 2023-12-12T21:59:19Z | 2344       | 207s      
  6         |           | 27420568      | 2023-12-12T21:55:51Z | 27422913      | 2023-12-12T21:59:19Z | 2345       | 207s      
  7         |           | 27378912      | 2023-12-12T21:55:51Z | 27381274      | 2023-12-12T21:59:20Z | 2362       | 208s      
  8         |           | 27396003      | 2023-12-12T21:55:52Z | 27398371      | 2023-12-12T21:59:21Z | 2368       | 208s      
  9         |           | 27370922      | 2023-12-12T21:55:52Z | 27373287      | 2023-12-12T21:59:20Z | 2365       | 208s      
  10        |           | 27375087      | 2023-12-12T21:55:52Z | 27377449      | 2023-12-12T21:59:20Z | 2362       | 207s      
  11        |           | 27371684      | 2023-12-12T21:55:52Z | 27374048      | 2023-12-12T21:59:20Z | 2364       | 208s      
  12        |           | 27381118      | 2023-12-12T21:55:52Z | 27383471      | 2023-12-12T21:59:20Z | 2353       | 207s      
  13        |           | 27385899      | 2023-12-12T21:55:52Z | 27388233      | 2023-12-12T21:59:19Z | 2334       | 206s      
  14        |           | -1            |                      | 27397521      | 2023-12-12T21:59:19Z | 27397522   |           
  15        |           | 27370625      | 2023-12-12T21:55:52Z | 27372959      | 2023-12-12T21:59:19Z | 2334       | 206s      
  16        |           | 27373673      | 2023-12-12T21:55:53Z | 27376025      | 2023-12-12T21:59:20Z | 2352       | 207s      
  17        |           | 27373152      | 2023-12-12T21:55:53Z | 27375485      | 2023-12-12T21:59:19Z | 2333       | 205s      
  18        |           | 27378605      | 2023-12-12T21:55:53Z | 27380954      | 2023-12-12T21:59:19Z | 2349       | 206s      
  19        |           | 27379616      | 2023-12-12T21:55:53Z | 27381974      | 2023-12-12T21:59:21Z | 2358       | 207s      
  20        |           | 27382512      | 2023-12-12T21:55:45Z | 27384859      | 2023-12-12T21:59:19Z | 2347       | 214s      
  21        |           | 27383199      | 2023-12-12T21:55:54Z | 27385533      | 2023-12-12T21:59:19Z | 2334       | 205s      
  22        |           | 27358526      | 2023-12-12T21:55:54Z | 27360872      | 2023-12-12T21:59:19Z | 2346       | 205s      
  23        |           | -1            |                      | 27372291      | 2023-12-12T21:59:19Z | 27372292   |           
  24        |           | 27393145      | 2023-12-12T21:55:54Z | 27395479      | 2023-12-12T21:59:19Z | 2334       | 205s      

@extemporalgenome
Copy link
Author

This is superseded by #172, which is intended to be followed by other PRs as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant