-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add optional progress information and other enhancements for restore-dump command #910
base: main
Are you sure you want to change the base?
Commits on Sep 27, 2024
-
Added ability to show extra details during restore process
When working with a larger restore using the `restore-dump` command by default there is only the "Restoring database ..." indicator by default which provides minimal insight into how things are progressing. On the other hand, if you choose to add the `--debug` flag the amount of information can then be too much as the large `INSERT` queries are then also included in the output. The new `--show-details` flag optionally allows for users to see how things are going with their restore more easily. Additionally, a `--start-from` flag was added that can be provided with a table name. This allows a user to skip earlier tables and start the import from a later point easily without having to create a separate copy of their dump folder and manually remove those files. The new `--allow-different-destination` flag primarily helps with simplifying the process of taking a folder that was created for one database, e.g. the files contain a prefix for "first-database", and allows you to restore into a second database without having to adjust the database prefix on all of those existing files. Also incorporated a check to prevent an issue for custom generated files where users might accidentally provide an `INSERT` query that is larger than 16777216 bytes. This provides some useful feedback rather than the `pkt` error that was being received previously.
Configuration menu - View commit details
-
Copy full SHA for b0b03a2 - Browse repository at this point
Copy the full SHA b0b03a2View commit details -
Adjust connection pool IDs to start from 1
From what I can tell this made no functional change and mainly allowed for the output I was providing with the new `--show-details` flag to allow showing information such as "Thread 1" rather than "Thread 0" without having to manually add one to the value elsewhere.
Configuration menu - View commit details
-
Copy full SHA for 0fec054 - Browse repository at this point
Copy the full SHA 0fec054View commit details -
Code cleanup plus additional flags
BuildKite was failing due to the way I was passing the helper used for printing so I made some adjustments and simplified how that worked. Afterward I added in two more optional flags: `--schema-only` and `--data-only`. If both are used at the same time then that's essentially the equivalent of a normal "restore" where both are being included. The new `--start-from` parameter can also be used together with these too.
Configuration menu - View commit details
-
Copy full SHA for f050789 - Browse repository at this point
Copy the full SHA f050789View commit details
Commits on Sep 28, 2024
-
Adjusted --start-from to --starting-table and --ending-table
I was thinking about this some more and it makes sense to allow for this option to be more comprehensive and allow not just >= a starting table to be included but a <= an ending table too or being able to provide a more limited starting/ending table range so I made some adjustments. This removes the `--start-from` option and replaces it with the `--starting-table` and `--ending-table` options which can be used separately or together.
Configuration menu - View commit details
-
Copy full SHA for eeb14a4 - Browse repository at this point
Copy the full SHA eeb14a4View commit details -
Should address these two that were noted: [2024-09-28T00:50:36Z] internal/cmd/database/restore.go:75:10: error strings should not be capitalized (ST1005) [2024-09-28T00:50:36Z] internal/cmd/database/restore.go:75:10: error strings should not end with punctuation or newlines (ST1005)
Configuration menu - View commit details
-
Copy full SHA for 731c3a6 - Browse repository at this point
Copy the full SHA 731c3a6View commit details
Commits on Sep 30, 2024
-
Add max-query-size optional parameter
Rather than having the 16777216 bytes limit be completely hard coded I thought it might be better to allow for it to be adjusted in rare cases since it may be possible that it is sometimes configured differently. Generally though, the default 16 MiB limit will be the default one encountered by our users. Note that this option is mainly for the quick length check performed prior to running a query and doesn't / cannot adjust the system configured message size limit for queries.
Configuration menu - View commit details
-
Copy full SHA for 1421bf6 - Browse repository at this point
Copy the full SHA 1421bf6View commit details
Commits on Oct 10, 2024
-
Removing code that shuffles the processing order of the data files
I'm not sure of the original intention for the shuffling code but I think allowing the data files to be processed more or less in numeric order will be of benefit to users with the other new options added within this branch.
Configuration menu - View commit details
-
Copy full SHA for bdef4dd - Browse repository at this point
Copy the full SHA bdef4ddView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8467444 - Browse repository at this point
Copy the full SHA 8467444View commit details
Commits on Oct 28, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 10437ea - Browse repository at this point
Copy the full SHA 10437eaView commit details
Commits on Nov 4, 2024
-
Configuration menu - View commit details
-
Copy full SHA for ab81e9b - Browse repository at this point
Copy the full SHA ab81e9bView commit details -
Added show details output information for views
Now that #932 was merged in a little earlier I was able to incorporate some extra show details output specific for views into this PR now. The current `canRestoreSchema()` has also been wrapped around the portion of code that restores the view definitions. In the future we may want to differentiate between restoring tables vs. views only but for now they are both considered to be part of the schema so I'm using the same check for both at the moment which should work ok for now.
Configuration menu - View commit details
-
Copy full SHA for bd304c5 - Browse repository at this point
Copy the full SHA bd304c5View commit details
Commits on Nov 7, 2024
-
Add additional context cancellation location
While an improved ability to detect Ctrl+C context cancellation was added in #933 I was noticing in this branch that when using the `--show-details` output after pressing Ctrl+C I would still see a lot of extra output displayed about data files being processed that were actually getting skipped so this helps to add an extra cancellation check to avoid that unneeded output.
Configuration menu - View commit details
-
Copy full SHA for 8426c12 - Browse repository at this point
Copy the full SHA 8426c12View commit details