Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional progress information and other enhancements for restore-dump command #910

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Commits on Sep 27, 2024

  1. Added ability to show extra details during restore process

    When working with a larger restore using the `restore-dump` command by default there is only the "Restoring database ..." indicator by default which provides minimal insight into how things are progressing.
    
    On the other hand, if you choose to add the `--debug` flag the amount of information can then be too much as the large `INSERT` queries are then also included in the output.
    
    The new `--show-details` flag optionally allows for users to see how things are going with their restore more easily.
    
    Additionally, a `--start-from` flag was added that can be provided with a table name. This allows a user to skip earlier tables and start the import from a later point easily without having to create a separate copy of their dump folder and manually remove those files.
    
    The new `--allow-different-destination` flag primarily helps with simplifying the process of taking a folder that was created for one database, e.g. the files contain a prefix for "first-database", and allows you to restore into a second database without having to adjust the database prefix on all of those existing files.
    
    Also incorporated a check to prevent an issue for custom generated files where users might accidentally provide an `INSERT` query that is larger than 16777216 bytes. This provides some useful feedback rather than the `pkt` error that was being received previously.
    orware committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    b0b03a2 View commit details
    Browse the repository at this point in the history
  2. Adjust connection pool IDs to start from 1

    From what I can tell this made no functional change and mainly allowed for the output I was providing with the new `--show-details` flag to allow showing information such as "Thread 1" rather than "Thread 0" without having to manually add one to the value elsewhere.
    orware committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    0fec054 View commit details
    Browse the repository at this point in the history
  3. Code cleanup plus additional flags

    BuildKite was failing due to the way I was passing the helper used for printing so I made some adjustments and simplified how that worked.
    
    Afterward I added in two more optional flags: `--schema-only` and `--data-only`.
    
    If both are used at the same time then that's essentially the equivalent of a normal "restore" where both are being included.
    
    The new `--start-from` parameter can also be used together with these too.
    orware committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    f050789 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. Adjusted --start-from to --starting-table and --ending-table

    I was thinking about this some more and it makes sense to allow for this option to be more comprehensive and allow not just >= a starting table to be included but a <= an ending table too or being able to provide a more limited starting/ending table range so I made some adjustments.
    
    This removes the `--start-from` option and replaces it with the `--starting-table` and `--ending-table` options which can be used separately or together.
    orware committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    eeb14a4 View commit details
    Browse the repository at this point in the history
  2. Address BuildKite Errors

    Should address these two that were noted:
    [2024-09-28T00:50:36Z] internal/cmd/database/restore.go:75:10: error strings should not be capitalized (ST1005)
    [2024-09-28T00:50:36Z] internal/cmd/database/restore.go:75:10: error strings should not end with punctuation or newlines (ST1005)
    orware committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    731c3a6 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Add max-query-size optional parameter

    Rather than having the 16777216 bytes limit be completely hard coded I thought it might be better to allow for it to be adjusted in rare cases since it may be possible that it is sometimes configured differently.
    
    Generally though, the default 16 MiB limit will be the default one encountered by our users.
    
    Note that this option is mainly for the quick length check performed prior to running a query and doesn't / cannot adjust the system configured message size limit for queries.
    orware committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    1421bf6 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Removing code that shuffles the processing order of the data files

    I'm not sure of the original intention for the shuffling code but I think allowing the data files to be processed more or less in numeric order will be of benefit to users with the other new options added within this branch.
    orware committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    bdef4dd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8467444 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. Configuration menu
    Copy the full SHA
    10437ea View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. Configuration menu
    Copy the full SHA
    ab81e9b View commit details
    Browse the repository at this point in the history
  2. Added show details output information for views

    Now that #932 was merged in a little earlier I was able to incorporate some extra show details output specific for views into this PR now.
    
    The current `canRestoreSchema()` has also been wrapped around the portion of code that restores the view definitions. In the future we may want to differentiate between restoring tables vs. views only but for now they are both considered to be part of the schema so I'm using the same check for both at the moment which should work ok for now.
    orware committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    bd304c5 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. Add additional context cancellation location

    While an improved ability to detect Ctrl+C context cancellation was added in #933 I was noticing in this branch that when using the `--show-details` output after pressing Ctrl+C I would still see a lot
    of extra output displayed about data files being processed that were actually getting skipped so this helps to add an extra cancellation check to avoid that unneeded output.
    orware committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    8426c12 View commit details
    Browse the repository at this point in the history