Support multiple operations acting on the same object in one migration #239
Comments
|
This was raised in the specific case of creating a table then adding an index to it in #203. |
andrew-farries
added a commit
that referenced
this issue
Nov 6, 2024
Add a `--skip-validation` flag to the `start` command.
If set, migration validation is skipped.
For example, this table has a `NOT NULL` `name` field:
```json
{
"name": "01_create_table",
"operations": [
{
"create_table": {
"name": "products",
"columns": [
{
"name": "id",
"type": "serial",
"pk": true
},
{
"name": "name",
"type": "varchar(255)",
"nullable": false
}
]
}
}
]
}
```
starting this migration will fail as the `name` field is already `NOT
NULL`:
```json
{
"name": "02_set_nullable",
"operations": [
{
"alter_column": {
"table": "products",
"column": "name",
"nullable": false,
"up": "name",
"down": "name"
}
}
]
}
```
```
Error: migration is invalid: column "name" on table "products" is NOT NULL
```
But if the start command is invoked with `--skip-validation` then the
start command succeeds.
This is part of #239
andrew-farries
added a commit
that referenced
this issue
Nov 6, 2024
Allow the `add_column` operation to add a column to a table that was
created by an operation earlier in the same migration.
The following migration would previously have failed to start:
```json
{
"name": "43_multiple_ops",
"operations": [
{
"create_table": {
"name": "players",
"columns": [
{
"name": "id",
"type": "serial",
"pk": true
},
{
"name": "name",
"type": "varchar(255)",
"check": {
"name": "name_length_check",
"constraint": "length(name) > 2"
}
}
]
}
},
{
"add_column": {
"table": "players",
"column": {
"name": "rating",
"type": "integer",
"comment": "hello world",
"check": {
"name": "rating_check",
"constraint": "rating > 0 AND rating < 100"
},
"nullable": false
}
}
}
]
}
```
As of this PR, the migration can be started.
The above migration does not validate yet, but it can be started
successfully with the `--skip-validation` flag to the `start` command.
Part of #239
andrew-farries
added a commit
that referenced
this issue
Nov 6, 2024
Allow the `create_index` operation to add an index to a table that was
created by an operation earlier in the same migration.
The following migration would previously have failed to start:
```json
{
"name": "43_multiple_ops",
"operations": [
{
"create_table": {
"name": "players",
"columns": [
{
"name": "id",
"type": "serial",
"pk": true
},
{
"name": "name",
"type": "varchar(255)",
"check": {
"name": "name_length_check",
"constraint": "length(name) > 2"
}
}
]
}
},
{
"create_index": {
"name": "idx_player_name",
"table": "players",
"columns": [
"name"
]
}
}
]
}
```
As of this PR the migration can be started.
The above migration does not validate yet, but it can be started
successfully with the --skip-validation flag to the start command.
Part of #239
andrew-farries
added a commit
that referenced
this issue
Nov 7, 2024
Allow the `create_index` operation to add an index to a column that was
created by an operation earlier in the same migration.
The following migration would previously have failed to start:
```json
{
"name": "43_multiple_ops",
"operations": [
{
"create_table": {
"name": "players",
"columns": [
{
"name": "id",
"type": "serial",
"pk": true
},
{
"name": "name",
"type": "varchar(255)",
"check": {
"name": "name_length_check",
"constraint": "length(name) > 2"
}
}
]
}
},
{
"add_column": {
"table": "players",
"column": {
"name": "rating",
"type": "integer",
"comment": "hello world",
"check": {
"name": "rating_check",
"constraint": "rating > 0 AND rating < 100"
},
"nullable": false
}
}
},
{
"create_index": {
"name": "idx_player_rating",
"table": "players",
"columns": [
"rating"
]
}
}
]
}
```
As of this PR the migration can be started.
The above migration does not validate yet, but it can be started
successfully with the `--skip-validation` flag to the `start` command.
Part of #239
This was referenced Nov 7, 2024
andrew-farries
added a commit
that referenced
this issue
Nov 8, 2024
Update the schema during `create_table` operation validation so that
validation of subsequent operations in the same migration can see the
new table.
This means that migrations that add a new table and then perform some
other operation, like adding a column, on that table can be validated
because the new table is visible to the `add_column` operation during
it's validation phase.
This means that the following migration is now able to validate:
```json
{
"name": "43_multiple_ops",
"operations": [
{
"create_table": {
"name": "players",
"columns": [
{
"name": "id",
"type": "serial",
"pk": true
},
{
"name": "name",
"type": "varchar(255)",
"check": {
"name": "name_length_check",
"constraint": "length(name) > 2"
}
}
]
}
},
{
"add_column": {
"table": "players",
"column": {
"name": "rating",
"type": "integer",
"comment": "hello world",
"check": {
"name": "rating_check",
"constraint": "rating > 0 AND rating < 100"
},
"nullable": false
}
}
}
]
}
```
Previously, the new table would not have been visible to the
`add_column` operation and its validation would have failed.
In order to make this work the schema needs to be re-read from the
target database in between validation and migration start, as the
previous assumption that validation would not make changes to the
in-memory schema representation no longer holds.
Part of #239
andrew-farries
added a commit
that referenced
this issue
Nov 8, 2024
Update the schema during `add_column` operation validation so that
validation of subsequent operations in the same migration can see the
new column.
This means that migrations that add a new column and then perform some
other operation, like adding an index, on that column can be validated
because the new column is visible to the `create_index` operation during
it's validation phase.
This means that the following migration is now able to validate:
```json
{
"name": "43_multiple_ops",
"operations": [
{
"add_column": {
"table": "players",
"column": {
"name": "rating",
"type": "integer",
"comment": "hello world",
"check": {
"name": "rating_check",
"constraint": "rating > 0 AND rating < 100"
},
"nullable": false
}
}
},
{
"create_index": {
"name": "idx_player_rating",
"table": "players",
"columns": [
"rating"
]
}
}
]
}
```
Previously, the new column would not have been visible to the
`create_index` operation and its validation would have failed.
This PR does for the `add_column` operation what #455 did for the
`create_table` operation.
Part of #239
kvch
pushed a commit
to kvch/pgroll
that referenced
this issue
Nov 11, 2024
…o#449) Allow the `add_column` operation to add a column to a table that was created by an operation earlier in the same migration. The following migration would previously have failed to start: ```json { "name": "43_multiple_ops", "operations": [ { "create_table": { "name": "players", "columns": [ { "name": "id", "type": "serial", "pk": true }, { "name": "name", "type": "varchar(255)", "check": { "name": "name_length_check", "constraint": "length(name) > 2" } } ] } }, { "add_column": { "table": "players", "column": { "name": "rating", "type": "integer", "comment": "hello world", "check": { "name": "rating_check", "constraint": "rating > 0 AND rating < 100" }, "nullable": false } } } ] } ``` As of this PR, the migration can be started. The above migration does not validate yet, but it can be started successfully with the `--skip-validation` flag to the `start` command. Part of xataio#239
kvch
pushed a commit
to kvch/pgroll
that referenced
this issue
Nov 11, 2024
…o#451) Allow the `create_index` operation to add an index to a table that was created by an operation earlier in the same migration. The following migration would previously have failed to start: ```json { "name": "43_multiple_ops", "operations": [ { "create_table": { "name": "players", "columns": [ { "name": "id", "type": "serial", "pk": true }, { "name": "name", "type": "varchar(255)", "check": { "name": "name_length_check", "constraint": "length(name) > 2" } } ] } }, { "create_index": { "name": "idx_player_name", "table": "players", "columns": [ "name" ] } } ] } ``` As of this PR the migration can be started. The above migration does not validate yet, but it can be started successfully with the --skip-validation flag to the start command. Part of xataio#239
kvch
pushed a commit
to kvch/pgroll
that referenced
this issue
Nov 11, 2024
…#454) Allow the `create_index` operation to add an index to a column that was created by an operation earlier in the same migration. The following migration would previously have failed to start: ```json { "name": "43_multiple_ops", "operations": [ { "create_table": { "name": "players", "columns": [ { "name": "id", "type": "serial", "pk": true }, { "name": "name", "type": "varchar(255)", "check": { "name": "name_length_check", "constraint": "length(name) > 2" } } ] } }, { "add_column": { "table": "players", "column": { "name": "rating", "type": "integer", "comment": "hello world", "check": { "name": "rating_check", "constraint": "rating > 0 AND rating < 100" }, "nullable": false } } }, { "create_index": { "name": "idx_player_rating", "table": "players", "columns": [ "rating" ] } } ] } ``` As of this PR the migration can be started. The above migration does not validate yet, but it can be started successfully with the `--skip-validation` flag to the `start` command. Part of xataio#239
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Migrations are composed of arbitrarily many operations, but with the limitation that a later operation in a migration cannot act on an object that has been created or modified by an earlier operation.
This limitiation severely impacts the usefulness of having multiple operations per migration.
It should be possible to compose arbitrary operations without this restriction.
The text was updated successfully, but these errors were encountered: