Add optional progress information and other enhancements for restore-dump command #910
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When working with a larger restore using the `restore-dump` command by default there is only the "Restoring database ..." indicator by default which provides minimal insight into how things are progressing. On the other hand, if you choose to add the `--debug` flag the amount of information can then be too much as the large `INSERT` queries are then also included in the output. The new `--show-details` flag optionally allows for users to see how things are going with their restore more easily. Additionally, a `--start-from` flag was added that can be provided with a table name. This allows a user to skip earlier tables and start the import from a later point easily without having to create a separate copy of their dump folder and manually remove those files. The new `--allow-different-destination` flag primarily helps with simplifying the process of taking a folder that was created for one database, e.g. the files contain a prefix for "first-database", and allows you to restore into a second database without having to adjust the database prefix on all of those existing files. Also incorporated a check to prevent an issue for custom generated files where users might accidentally provide an `INSERT` query that is larger than 16777216 bytes. This provides some useful feedback rather than the `pkt` error that was being received previously.
From what I can tell this made no functional change and mainly allowed for the output I was providing with the new `--show-details` flag to allow showing information such as "Thread 1" rather than "Thread 0" without having to manually add one to the value elsewhere.
BuildKite was failing due to the way I was passing the helper used for printing so I made some adjustments and simplified how that worked. Afterward I added in two more optional flags: `--schema-only` and `--data-only`. If both are used at the same time then that's essentially the equivalent of a normal "restore" where both are being included. The new `--start-from` parameter can also be used together with these too.
I was thinking about this some more and it makes sense to allow for this option to be more comprehensive and allow not just >= a starting table to be included but a <= an ending table too or being able to provide a more limited starting/ending table range so I made some adjustments. This removes the `--start-from` option and replaces it with the `--starting-table` and `--ending-table` options which can be used separately or together.
|
I was thinking about this some more and it makes sense to allow for this option to be more comprehensive and allow not just those tables >= to a starting table to be included but also the tables <= to an ending table too as well as being able to provide a more specific starting/ending inclusive table range so I made some adjustments. The last commit removes the Examples:If not using the options at all then all tables should be included as they normally would. If the ending table provided comes before the starting table alphabetically then the restore will short circuit until that is corrected. This example skips the last two tables since the table range being provided will only include the first two: For the case above, running the same command without the use of The opposite however, starting from the |
Should address these two that were noted: [2024-09-28T00:50:36Z] internal/cmd/database/restore.go:75:10: error strings should not be capitalized (ST1005) [2024-09-28T00:50:36Z] internal/cmd/database/restore.go:75:10: error strings should not end with punctuation or newlines (ST1005)
Rather than having the 16777216 bytes limit be completely hard coded I thought it might be better to allow for it to be adjusted in rare cases since it may be possible that it is sometimes configured differently. Generally though, the default 16 MiB limit will be the default one encountered by our users. Note that this option is mainly for the quick length check performed prior to running a query and doesn't / cannot adjust the system configured message size limit for queries.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These changes add a few potentially useful capabilities to the
restore-dumpcommand.The first is the
--show-detailsflag, which helps provide a middle ground between the minimal default "Restoring database ..." output and the potentially too noisy--debugoutput, since that ends up outputting the largeINSERTqueries being executed as well to the terminal. This flag will output the names of the files that will be included within the restore folder and will also show progress processing the queries within the various data files.The--start-fromflag allows you to provide a table name which will then be used to skip earlier tables so that the restore will begin from the table name that is provided which can be helpful to avoid having to start a restore completely from the very first table again.(Replaced by the
--starting-tableand--ending-tableoptions described in the later comment down below.)The
--allow-different-destinationflag addresses #540 and adds extra flexibility since users would not be required to adjust the database name prefix embedded into the files within a dump folder for a different database and can simply enable this option and it will allow those files to be utilized for a database with a different name.The
--schema-onlyand--data-onlyflags can help address situations where you may have a folder containing both sets of files but only need one type. While a new folder containing only the files needed is an option this mainly helps add extra flexibility without needing to take that extra step.Example output:
The following two examples use an example database name of
metricsinto itsmainbranch.The first example shows using the
--schema-onlyflag:The second example demonstrates how using
--start-fromprevents files from the earlierconnectionstable from being included at all and allows things to start from the provided table name which happens to be calledplaceholder:To demonstrate the usage of
--allow-different-destinationfirst let's see the default behavior when attempting to restore a dump folder, created for themetricsdatabase, into a database namednot-metrics:As expected, the restore is unable to complete since the filenames within the dump folder have the embedded
metricsname which doesn't match with thenot-metricsname being passed into the command.But if we add the new
--allow-different-destinationflag in, this allows the restore to proceed into the differently named destination database.I didn't want to include a lot of lines showing data files being processed but below is a snippet showing the extra details that would be included towards the end of processing a file: