Client should push migrations in small batches

[kostasb]

Describe the bug
When the client pushes a very large schema (i.e. xata schema upload with >30 tables) it is prone to hitting gateway timeout.

Migrations or schema update operations should be pushed in iterative batches, with a best practice size of no more than 10 operations per request.

Relates to internal issue https://github.com/xataio/xata/issues/3353

[SferaDev]

The ~10 operations is a limitation on the backend with synchronous schema changes. With the integration of pgroll we will be able to do async schema changes, and this won't be an issue.

We cannot chunk the schema, as dependencies should be ordered (for example links, or circular refs). So we won't attempt to fix it client side, and instead wait for the new APIs to land.

[urso]

The idea would be to not push all migration files at once. e.g. only apply up to 10 files per API call. The files should not be split up into operations. But right now if users push many files with many tables in them then the gateway will timeout the operation.

[SferaDev]

@urso For this command, we are doing:

    const { edits } = await xata.api.migrations.compareBranchWithUserSchema({
      workspace,
      region,
      database,
      branch,
      schema
    });

    await xata.api.migrations.applyBranchSchemaEdit({ workspace, region, database, branch, edits });

So it's very tricky to try to manipulate the backend suggested edits.

[urso]

I see.