Add tool to migrate data from badger to mysql or postgresql

[maraino]

Description

This PR adds a tool that allows migrating data from BadgerDB to MySQL or PostgreSQL databases.

[joshdrake]

This looks good, just a few nits and two larger questions about runtime/performance:

• this does not batch inserts/sets in any way, is that possible (just from a brief perusal of the nosql package, doesn't seem so)?
• do we need to consider "resume" behavior, eg: could we seek the iterator to a given position (for each table) based on user input?

[maraino]

@joshdrake

this does not batch inserts/sets in any way, is that possible (just from a brief perusal of the nosql package, doesn't seem so)?

It is not actually possible to do batch inserts with nosql. It is actually possible to create transactions, but it might be problematic for large databases.

do we need to consider "resume" behavior, eg: could we seek the iterator to a given position (for each table) based on user input?

I'll take a look at this

[joshdrake]

lgtm

[hslatman]

Minor nits

[celevra]

thank you for your work!
i'm no go specialist and need a little help on how to compile the script:

login blub # go build
main.go:14:2: cannot find package "github.com/dgraph-io/badger/v2" in any of:
        /usr/lib/go-1.18/src/github.com/dgraph-io/badger/v2 (from $GOROOT)
        /root/go/src/github.com/dgraph-io/badger/v2 (from $GOPATH)
/usr/lib/go-1.18/src/github.com/smallstep/nosql/badger/v2/badger.go:12:2: cannot find package "github.com/dgraph-io/badger/v2/options" in any of:
        /usr/lib/go-1.18/src/vendor/github.com/dgraph-io/badger/v2/options (vendor tree)
        /usr/lib/go-1.18/src/github.com/dgraph-io/badger/v2/options (from $GOROOT)
        /root/go/src/github.com/dgraph-io/badger/v2/options (from $GOPATH)
/usr/lib/go-1.18/src/github.com/smallstep/nosql/postgresql/postgresql.go:13:2: cannot find package "github.com/jackc/pgx/v4" in any of:
        /usr/lib/go-1.18/src/vendor/github.com/jackc/pgx/v4 (vendor tree)
        /usr/lib/go-1.18/src/github.com/jackc/pgx/v4 (from $GOROOT)
        /root/go/src/github.com/jackc/pgx/v4 (from $GOPATH)
/usr/lib/go-1.18/src/github.com/smallstep/nosql/postgresql/postgresql.go:14:2: cannot find package "github.com/jackc/pgx/v4/stdlib" in any of:
        /usr/lib/go-1.18/src/vendor/github.com/jackc/pgx/v4/stdlib (vendor tree)
        /usr/lib/go-1.18/src/github.com/jackc/pgx/v4/stdlib (from $GOROOT)
        /root/go/src/github.com/jackc/pgx/v4/stdlib (from $GOPATH)

could you guide me in the right direction?

[celevra]

any news on how to use it?

[hslatman]

@celevra

Start by ensuring you have Go installed according to the instructions: https://go.dev/doc/install. Then follow the steps on https://github.com/smallstep/certificates/blob/master/CONTRIBUTING.md to get started with development. If you still hit issues with the dependencies not being found, try a go mod tidy in the directory of the migration tool