Skip to content

Commit

Permalink
Merge pull request #6032 from EnterpriseDB/mtk/schema_scope
Browse files Browse the repository at this point in the history 
Included schema scope options for `-allSchemas` and multiple schemas on Invoking MTK page
  • Loading branch information
@gvasquezvargas
gvasquezvargas authored Sep 27, 2024
2 parents e28ae35 + fa2108b commit 612bc3e
Show file tree
Hide file tree
Showing 6 changed files with 181 additions and 62 deletions.
2 changes: 1 addition & 1 deletion product_docs/docs/migration_toolkit/55/06_building_toolkit.properties_file.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Before executing Migration Toolkit commands, modify the `toolkit.properties` fil

!!! Note

Unless specified in the command line, Migration Toolkit expects the source database to be Oracle and the target database to be EDB Postgres Advanced Server. For any other source or target database, specify the `-sourcedbtype` or `-targetdbtype` options as described in [Migrating from a non-Oracle source database](/migration_toolkit/latest/07_invoking_mtk/#migrating-from-a-non-oracle-source-database).
Unless specified in the command line, Migration Toolkit expects the source database to be Oracle and the target database to be EDB Postgres Advanced Server. For any other source or target database, specify the `-sourcedbtype` or `-targetdbtype` options as described in [Migrating from a non-Oracle source database](/migration_toolkit/latest/07_invoking_mtk/#migrating-schemas-from-a-non-oracle-source-database).

For specifying a target database on BigAnimal, see [Defining a BigAnimal URL](#defining-a-biganimal-url).

Expand Down
16 changes: 8 additions & 8 deletions product_docs/docs/migration_toolkit/55/07_invoking_mtk/08_mtk_command_options.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,40 +43,40 @@ If you specify the `-offlineMigration` option in the command line, Migration Too
!!! Note
The following examples invoke Migration Toolkit in Linux. To invoke Migration Toolkit in Windows, use the `runMTK.bat` command instead of the `runMTK.sh` command.

To perform an offline migration of both schema and data, specify the `‑offlineMigration` keyword, followed by the schema name:
To perform an offline migration of both schema and data, specify the `‑offlineMigration` keyword, followed by the [schema scope](../07_invoking_mtk/#migrating-schemas):

```shell
./runMTK.sh -offlineMigration <schema_name>
./runMTK.sh -offlineMigration <schema_scope>
```

Each database object definition is saved in a separate file with a name derived from the schema name and object type in your home folder. To specify an alternative file destination, include a directory name after the `‑offlineMigration` option:
Each database object definition is saved in a separate file with a name derived from each schema name and object type in your home folder. To specify an alternative file destination, include a directory name after the `‑offlineMigration` option:

```shell
./runMTK.sh -offlineMigration <file_dest> <schema_name>
./runMTK.sh -offlineMigration <file_dest> <schema_scope>
```

To perform an offline migration of only schema objects (creating empty tables), specify the `‑schemaOnly` keyword in addition to the `‑offlineMigration` keyword when invoking Migration Toolkit:

```shell
./runMTK.sh -offlineMigration -schemaOnly <schema_name>
./runMTK.sh -offlineMigration -schemaOnly <schema_scope>
```

To perform an offline migration of only data, omitting any schema object definitions, specify the `‑dataOnly` keyword and the `‑offlineMigration` keyword when invoking Migration Toolkit:

```shell
./runMTK.sh -offlineMigration -dataOnly <schema_name>
./runMTK.sh -offlineMigration -dataOnly <schema_scope>
```

By default, data is written in COPY format. To write the data in a plain SQL format, include the `‑safeMode` keyword:

```shell
./runMTK.sh -offlineMigration -dataOnly -safeMode <schema_name>
./runMTK.sh -offlineMigration -dataOnly -safeMode <schema_scope>
```

By default, when you perform an offline migration that contains table data, a separate file is created for each table. To create a single file that contains the data from multiple tables, specify the `‑singleDataFile` keyword:

```shell
./runMTK.sh -offlineMigration -dataOnly -singleDataFile -safeMode <schema_name>
./runMTK.sh -offlineMigration -dataOnly -singleDataFile -safeMode <schema_scope>
```

!!! Note
Expand Down
186 changes: 143 additions & 43 deletions product_docs/docs/migration_toolkit/55/07_invoking_mtk/index.mdx
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,17 @@
---
title: "Invoking Migration Toolkit"

description: "Learn how to perform a migration with the Migration Toolkit."
deepToC: true
---

<div id="invoking_mtk" class="registered_link"></div>

After installing Migration Toolkit and specifying connection properties for the source and target databases in the [toolkit.properties file](../06_building_toolkit.properties_file), Migration Toolkit is ready to perform migrations.

!!!note
Ensure the operating system user account running the Migration Toolkit is the owner of the `toolkit.properties` file with a minimum of read permission on the file. See [Invalid file permissions](../09_mtk_errors/#invalid-file-permissions) for more information.
!!!

The Migration Toolkit executable is named `runMTK.sh` on Linux systems and `runMTK.bat` on Windows systems. On a Linux system, the executable is located in:

`/usr/edb/migrationtoolkit/bin`
Expand All @@ -17,68 +22,123 @@ On Windows, the executable is located in:

See [Migration Toolkit command options](08_mtk_command_options) for information on controlling details of the migration.

## Importing character data with embedded binary zeros (NULL characters)

Migration Toolkit supports importing a column with a value of NULL. However, Migration Toolkit doesn't support importing NULL character values (embedded binary zeros 0x00) with the JDBC connection protocol. If you're importing data that includes the NULL character, use the `-replaceNullChar` option to replace the NULL character with a single, non-NULL, replacement character.

!!! Note
If the following error appears upon invoking the Migration Toolkit, check the file permissions of the `toolkit.properties` file.
- MTK implicitly replaces NULL characters with an empty string.
- The `-replaceNullChar` option doesn't work with the `-copyViaDBLinkOra` option.

Once the data is migrated, use a SQL statement to replace the character specified by `-replaceNullChar` with binary zeros.

## Migrating schemas

When migrating schemas from the source database specified in the [toolkit.properties file](../06_building_toolkit.properties_file), you have to provide the `<schema_scope>` for the migration. You have these options:

- To migrate a single schema, specify the schema name.
- To migrate multiple schemas, provide a comma-delimited list of schema names.
- To migrate all the schemas, use the `-allSchemas` option.

These examples display how to execute the command depending on your operating system and schema scope.

| | On Linux | On Windows |
|------------------|--------------------------------------------------------------|---------------------------------------------------------------|
| Single schema | `$ ./runMTK.sh <schema_name>` | `> .\runMTK.bat <schema_name>` |
| Multiple schemas | `$ ./runMTK.sh <schema_name1>,<schema_name2>,<schema_name3>` | `> .\runMTK.bat <schema_name1>,<schema_name2>,<schema_name3>` |
| All schemas | `$ ./runMTK.sh -allSchemas` | `> .\runMTK.bat -allSchemas` |

### Migrating schemas from Oracle

```text
MTK-11015: The connection credentials file ../etc/toolkit.properties is
not secure and accessible to group/others users. This file contains
plain passwords and should be restricted to Migration Toolkit owner user
only.
Unless specified in the command line, Migration Toolkit expects the source database to be Oracle and the target database to be EDB Postgres Advanced Server. To migrate schemas, navigate to the executable and invoke the following command.

On Linux:

```shell
$ ./runMTK.sh <schema_scope>
```

The operating system user account running the Migration Toolkit must be the owner of the `toolkit.properties` file with a minimum of read permission on the file. In addition, there must be no permissions of any kind for group and other users. The following is an example of the recommended file permissions, where user enterprisedb is running the Migration Toolkit.
On Windows:

`-rw------- 1 enterprisedb enterprisedb 191 Aug 1 09:59 toolkit.properties`
```shell
> .\runMTK.bat <schema_scope>
```

## Importing character data with embedded binary zeros (NULL characters)
Where `<schema_scope>` is the [scope of schemas](#migrating-schemas) to migrate which can be a single schema, multiple schemas, or all schemas from the specified database.

Migration Toolkit supports importing a column with a value of NULL. However, Migration Toolkit doesn't support importing NULL character values (embedded binary zeros 0x00) with the JDBC connection protocol. If you're importing data that includes the NULL character, use the `-replaceNullChar` option to replace the NULL character with a single, non-NULL, replacement character.
<details><summary>Single schema example</summary>

!!! Note
- MTK implicitly replaces NULL characters with an empty string.
- The `-replaceNullChar` option doesn't work with the `-copyViaDBLinkOra` option.
The following example migrates a schema (table definitions and table content) named `HR` from an Oracle database to an EDB Postgres Advanced Server database.

Once the data is migrated, use a SQL statement to replace the character specified by `-replaceNullChar` with binary zeros.
On Linux:

## Migrating a schema from Oracle
```shell
$ ./runMTK.sh HR
```

Unless specified in the command line, Migration Toolkit expects the source database to be Oracle and the target database to be EDB Postgres Advanced Server. To migrate a complete schema on Linux, navigate to the executable and invoke the following command:
On Windows:

`$ ./runMTK.sh <schema_name>`
```shell
> .\runMTK.bat HR
```

To migrate a complete schema on Windows, navigate to the executable and invoke the following command:
</details>
<br/>

`> .\runMTK.bat <schema_name>`
<details><summary>Multiple schemas example</summary>

Where:
The following example migrates multiple schemas (named HR and ACCTG) from an Oracle database to a PostgreSQL database.

`<schema_name>` is the name of the schema in the source database specified in the `toolkit.properties` file that you want to migrate. You must include at least one `schema_name` value.
On Linux:

!!! Note
- When the default database user of a migrated schema is automatically migrated, the custom profile of the default database user is also migrated if a custom profile exists. A custom profile is a user-created profile. For example, custom profiles exclude Oracle profiles `DEFAULT` and `MONITORING_PROFILE`.
- PostgreSQL default rows are limited to 8 KB in size. This means that each table row must fit into a single 8 KB block, Otherwise, an error occurs indicating, for example, that we can create a table with 1600 columns of INT and insert data for all the columns. However, we can't do the same with BIGINT columns because INT is stored as 4 bytes, but each BIGINT requires more space (8 bytes). For more information, see [PostgreSQL Limits](https://www.postgresql.org/docs/14/limits.html) in the PostgreSQL documentation.
```shell
$ ./runMTK.sh -targetdbtype postgres HR,ACCTG
```

You can migrate multiple schemas by following the command name with a comma-delimited list of schema names.
On Windows:

On Linux, execute the following command:
```shell
> .\runMTK.bat -targetdbtype postgres HR,ACCTG
```

`$ ./runMTK.sh <schema_name1>,<schema_name2>,<schema_name3>`
</details>
<br/>

On Windows, execute the following command:
<details><summary>All schemas example</summary>

`> .\runMTK.bat <schema_name1>,<schema_name2>,<schema_name3>`
The following example migrates all schemas from the Oracle database specified in the [toolkit.properties file](../06_building_toolkit.properties_file) to an EDB Postgres Advanced Server database.

On Linux:

```shell
$ ./runMTK.sh -allSchemas
```

On Windows:

```shell
> .\runMTK.bat -allSchemas
```

</details>
<br/>

!!!note Notes
- When the default database user of a migrated schema is automatically migrated, the custom profile of the default database user is also migrated if a custom profile exists. A custom profile is a user-created profile. For example, custom profiles exclude Oracle profiles `DEFAULT` and `MONITORING_PROFILE`.
- PostgreSQL default rows are limited to 8 KB in size. This means that each table row must fit into a single 8 KB block, Otherwise, an error occurs indicating, for example, that we can create a table with 1600 columns of INT and insert data for all the columns. However, we can't do the same with BIGINT columns because INT is stored as 4 bytes, but each BIGINT requires more space (8 bytes). For more information, see [PostgreSQL Limits](https://www.postgresql.org/docs/14/limits.html) in the PostgreSQL documentation.
!!!

<div id="migrate_non_oracle" class="registered_link"></div>

## Migrating from a non-Oracle source database
### Migrating schemas from a non-Oracle source database

If you don't specify a source database type and a target database type, Postgres assumes the source database is Oracle and the target database is EDB Postgres Advanced Server.

To invoke Migration Toolkit, open a command window, navigate to the executable, and invoke the following command:

`$ ./runMTK.sh -sourcedbtype <source_type> -targetdbtype <target_type> [options, …] <schema_name>;`
```shell
$ ./runMTK.sh -sourcedbtype <source_type> -targetdbtype <target_type> [options, …] <schema_scope>;
```

`-sourcedbtype <source_type>`

Expand All @@ -102,26 +162,66 @@ To invoke Migration Toolkit, open a command window, navigate to the executable,
| EDB Postgres Advanced Server | enterprisedb |
| PostgreSQL | postgres or postgresql |

`<schema_name>`
`<schema_scope>`

`<schema_name>` is the name of the schema in the source database specified in the `toolkit.properties` file that you want to migrate. You must include at least one `<schema_name>` value.
Where `<schema_scope>` is the [scope of schemas](#migrating-schemas) to migrate which can be a single schema, multiple schemas, or all schemas from the specified database.

<details><summary>Single schema example</summary>

The following example migrates a schema (table definitions and table content) named `HR` from a MySQL database on a Linux system to an EDB Postgres Advanced Server host. The command includes the `‑sourcedbtype` and `-targetdbtype` options.

On Linux, use the following command:
On Linux:

```shell
$ ./runMTK.sh -sourcedbtype mysql -targetdbtype enterprisedb HR
```

On Windows:

```shell
> .\runMTK.bat -sourcedbtype mysql -targetdbtype enterprisedb HR
```

</details>
<br/>

<details><summary>Multiple schemas example</summary>

The following example migrates multiple schemas (named HR and ACCTG) from a MySQL database to a PostgreSQL database.

On Linux:

```shell
$ ./runMTK.sh -sourcedbtype mysql -targetdbtype postgres HR,ACCTG
```

`$ ./runMTK.sh -sourcedbtype mysql -targetdbtype enterprisedb HR`
On Windows:

On Windows, use the following command:
```shell
> .\runMTK.bat -sourcedbtype mysql -targetdbtype postgres HR,ACCTG
```

</details>
<br/>

`> .\runMTK.bat -sourcedbtype mysql -targetdbtype enterprisedb HR`
<details><summary>All schemas example</summary>

You can migrate multiple schemas from a source database by including a comma-delimited list of schemas at the end of the Migration Toolkit command. The following example migrates multiple schemas (named HR and ACCTG) from a MySQL database to a PostgreSQL database.
The following example migrates all schemas from a PostgreSQL database specified in the [toolkit.properties file](../06_building_toolkit.properties_file) to an EDB Postgres Advanced Server database.

On Linux, use the following command:
!!! Note
The `-allSchemas` parameter is supported only for Oracle, EDB Postgres Advanced Server, and PostgreSQL source databases.

`$ ./runMTK.sh -sourcedbtype mysql -targetdbtype postgres HR,ACCTG`
On Linux:

On Windows, use the following command:
```shell
$ ./runMTK.sh -sourcedbtype postgres -allSchemas
```

On Windows:

```shell
> .\runMTK.bat -sourcedbtype postgres -allSchemas
```

`> .\runMTK.bat -sourcedbtype mysql -targetdbtype postgres HR,ACCTG`
</details>
<br/>
10 changes: 5 additions & 5 deletions ...on_toolkit/55/07_invoking_mtk/mtk_command_options_in_file/creating_txt_file.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,20 +66,20 @@ Specifying an option both at the command line and in the text file causes the mi

## Order of processing

Migration Toolkit reads command line options and option files in the order you provide them when running the command. Ensure you add the [schema scope](executing_migration_with_txt/#provide-the-scope-for-the-schema-migration) (`schema_name` or `-allSchemas`) as the last parameter at the command line.
Migration Toolkit reads command line options and option files in the order you provide them when running the command. Ensure you add the [schema scope](../../07_invoking_mtk/#migrating-schemas) as the last parameter at the command line.

For example, if you run the following command, Migration Toolkit first recognizes the `-sourcedbtype oracle` option, and then reads the contents of `options_textfile` in order from top to bottom. The last parameter is the schema scope (`<schema_name>` or `-allSchemas`).
For example, if you run the following command, Migration Toolkit first recognizes the `-sourcedbtype oracle` option, and then reads the contents of `options_textfile` in order from top to bottom. The last parameter is the `<schema_scope>`.

```shell
runMTK.sh -sourcedbtype oracle -optionsFile options_textfile schema_name
runMTK.sh -sourcedbtype oracle -optionsFile options_textfile <schema_scope>
```

Using an options file means that you can employ different syntaxes to perform a migration where parameters are executed in the same way. The following alternatives perform the same migration.

**Alternative 1**

```shell
runMTK.sh -sourcedbtype oracle -optionsFile <options_textfile> <schema_name>
runMTK.sh -sourcedbtype oracle -optionsFile <options_textfile> <schema_scope>
```

Where the content of the `<options_textfile>` file is:
Expand All @@ -92,7 +92,7 @@ dataOnly
**Alternative 2**

```shell
runMTK.sh -sourcedbtype oracle -optionsFile <options_textfile> -dataOnly <schema_name>
runMTK.sh -sourcedbtype oracle -optionsFile <options_textfile> -dataOnly <schema_scope>
```

Where the content of the `<options_textfile>` file is:
Expand Down
Loading

1 comment on commit 612bc3e

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 Published on https://edb-docs-staging.netlify.app as production
🚀 Deployed on https://66f6a1b56d45dc3d15fff6f1--edb-docs-staging.netlify.app

Please sign in to comment.