Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat[Storagetool]: cluster state ledger (CSL) file search support #558

Draft
wants to merge 54 commits into
base: main
Choose a base branch
from

Conversation

alexander-e1off
Copy link
Collaborator

Enhance storage tool to search records in cluster state ledger (CSL) file.

To search records in CSL file, only CSL file path (--csl-file option) must be passed, journal file path (--journal-path/--journal-path` options must not be present.

Scenarios of BMQStorageTool usage for CSL file

Output summary for CSL file

Example:

bmqstoragetool --csl-file=<path> --summary

Output records from the beginning of CSL file

Example:

bmqstoragetool --csl-file=<path> --csl-from-begin

NOTE: by default search is done from the latest snapshot.

Search and otput only desired record types in CSL file

Example:

bmqstoragetool --csl-file=<path> --csl-record-type=snapshot --csl-record-type=update

NOTE: snapshot, update, commit and ack are supported. Without this option all record types are selected.

Search and otput records details in CSL file

Example:

bmqstoragetool --csl-file=<path> --details

The same search filters could be applied to above scenarios as for journal file search (except --guid filter).

Example of short result output

[ recordType = SNAPSHOT electorTerm = 401 sequenceNumber = 25 timestamp = 1706022776 ][ logId = 37146FB620 offset = 23492436 ]
[ recordType = UPDATE electorTerm = 401 sequenceNumber = 27 timestamp = 1706029885 ][ logId = 37146FB620 offset = 23503460 ]
1 snapshot record(s) found.
1 update record(s) found.
No commit record(s) found.
No ack record(s) found.

Example of detail result output

============================
SNAPSHOT Record, offset: 808
    LogId                : E1AA1EC6E5
    ElectorTerm          : 1
    SequenceNumber       : 1
    HeaderWords          : 8
    LeaderAdvisoryWords  : 38
    Timestamp            : 26NOV2024_11:03:35.000000
    Epoch                : 1732619015
[
  choice = [
    leaderAdvisory = [
      sequenceNumber = [
        electorTerm = 1
        sequenceNumber = 1
      ]
      partitions = [
        [
          partitionId = 0
          primaryNodeId = 0
          primaryLeaseId = 4
        ]
      ]
      queues = [
        [
          uri = "bmq://bmq.test.persistent.fanout/my1"
 key = [ 44D469EF30 ]          partitionId = 0
          appIds = [
            [
              appId = "bar"
 appKey = [ 9FC222C73E ]            ]
            [
              appId = "baz"
 appKey = [ 935B5C5FC4 ]            ]
            [
              appId = "foo"
 appKey = [ 489D904F25 ]            ]
          ]
        ]
      ]
    ]
  ]
]

==========================
COMMIT Record, offset: 992
    LogId                : E1AA1EC6E5
    ElectorTerm          : 1
    SequenceNumber       : 2
    HeaderWords          : 8
    LeaderAdvisoryWords  : 10
    Timestamp            : 26NOV2024_11:03:35.000000
    Epoch                : 1732619015
[
  choice = [
    leaderAdvisoryCommit = [
      sequenceNumber = [
        electorTerm = 1
        sequenceNumber = 2
      ]
      sequenceNumberCommitted = [
        electorTerm = 1
        sequenceNumber = 1
      ]
    ]
  ]
]

===========================
UPDATE Record, offset: 1064
    LogId                : E1AA1EC6E5
    ElectorTerm          : 1
    SequenceNumber       : 3
    HeaderWords          : 8
    LeaderAdvisoryWords  : 34
    Timestamp            : 26NOV2024_11:03:59.000000
    Epoch                : 1732619039
[
  choice = [
    queueAssignmentAdvisory = [
      sequenceNumber = [
        electorTerm = 1
        sequenceNumber = 3
      ]
      queues = [
        [
          uri = "bmq://bmq.test.persistent.fanout/my2"
 key = [ 676EC94E89 ]          partitionId = 0
          appIds = [
            [
              appId = "foo"
 appKey = [ 40D805FBE4 ]            ]
            [
              appId = "bar"
 appKey = [ 8377729B23 ]            ]
            [
              appId = "baz"
 appKey = [ 0C996AF949 ]            ]
          ]
        ]
      ]
    ]
  ]
]

1 snapshot record(s) found.
1 update record(s) found.
1 commit record(s) found.
No ack record(s) found.

Example of summary result output

1 snapshot record(s) found.

2 update record(s) found, including:
    queueAssignmentAdvisory  : 1
    queueUnassignedAdvisory  : 1

3 commit record(s) found.

No ack record(s) found.

2 Queues found:
[ uri = "bmq://bmq.test.persistent.fanout/my2" key = [ 676EC94E89 ]
 partitionId = 0 appIds = [ [ appId = "foo" appKey = [ 40D805FBE4 ] ] [ appId = "bar" appKey = [ 8377729B23 ] ] [ appId = "baz" appKey = [ 0C996AF949 ] ] ] ]
[ uri = "bmq://bmq.test.persistent.fanout/my1" key = [ 44D469EF30 ]
 partitionId = 0 appIds = [ [ appId = "bar" appKey = [ 9FC222C73E ] ] [ appId = "baz" appKey = [ 935B5C5FC4 ] ] [ appId = "foo" appKey = [ 489D904F25 ] ] ] ]

Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant