Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bump common-streams to 0.6.0 #374

Merged
merged 4 commits into from
May 9, 2024
Merged

Bump common-streams to 0.6.0 #374

merged 4 commits into from
May 9, 2024

Commits on May 9, 2024

  1. Bump common-streams to 0.6.0

    @istreeter @pondzix
    istreeter authored and pondzix committed May 9, 2024
    Configuration menu
    Copy the full SHA
    0944b60 View commit details
    Browse the repository at this point in the history

  2. Handle too many columns - part 2 

    We already handle one type of exception for too many columns case with
message like:

```Too many columns (10067) in schema. Only 10000 columns are allowed```

It turns out BQ can fail with different one in this case, like:

```Too many total leaf fields: 10001, max allowed field count: 10000```

so we have to handle it as well.
    @pondzix
    pondzix committed May 9, 2024
    Configuration menu
    Copy the full SHA
    8338639 View commit details
    Browse the repository at this point in the history

  3. Allow for delay in Writer discovering new columns 

    After the loader alters the table to add new columns, it immediately
opens a new Writer and expects the Writer to be aware of the new
columns. However, we have found the Writer might get opened with no
awareness of the newly added columns. Presumbably because of the async
nature of BigQuery's architecture.

This fix works by retrying opening the writer until eventually it should
get opened with awareness of the new columns
    @istreeter @pondzix
    istreeter authored and pondzix committed May 9, 2024
    Configuration menu
    Copy the full SHA
    771f821 View commit details
    Browse the repository at this point in the history

  4. Require alter table when schema is evolved for contexts 

    When schema evolves, e.g. when new nested field is added to entity, loader should explicity try to alter underlying BigQuery schema. It works correctly for self-describing events (unstruct columns), but as it turns in case of contexts it doesn't modify schema, alter is skipped, what results in bad data. This commit should fix this problem.
    @pondzix
    pondzix committed May 9, 2024
    Configuration menu
    Copy the full SHA
    c1951fe View commit details
    Browse the repository at this point in the history