Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ingest/glue): use resource_links in catalog correctly #1

Closed
wants to merge 14 commits into from

Commits on Oct 21, 2024

  1. Configuration menu
    Copy the full SHA
    16f9075 View commit details
    Browse the repository at this point in the history
  2. fix(ingestion/glue): manage table names from resource_links from near…

    …est catalog correctly
    
    Changes:
    
    If resource links are detected, re-use database names from the current/nearest catalog.
    Otherwise external resource names are picked out instead of the aliased ones.
    This will cause an incoherent situation when creating full table names later in DataHub.
    
    Refactoring:
    
    - database filtering is move to an upper step in `get_databases()` using JMESPath notation
    - `get_tables_from_database()` signature changed to support database aliases
    - use an explicit source_config check but it is useless actually (filtering has already been done)
    
    Credits to @sl-fredericdalessandro for his preliminary patchset to support LakeFormation in Glue ingestion.
    aviv-julienjehannet committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    14826fe View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    24a4f87 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c09f13b View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Configuration menu
    Copy the full SHA
    fb6a5b8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6c55511 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f21c8bc View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    cdb23ac View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    bab8274 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    7f6f18c View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    7ba77a9 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    b7958c4 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    48f4b1a View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    36c45b5 View commit details
    Browse the repository at this point in the history