DataHub v0.8.33

Release Highlights

User Experience

Refreshed the ML Entity page to match the feel of all other entity types; improved ML lineage functionality

Ingestion Improvements

• Airflow Improvements - as demoed in March Town Hall
  • Add support to capture Airflow execution runs from lineage backend
  • Introduce new High level API for generating dataflow/job/dataprocessinstance
• MS SQL ingestion now captures table & column descriptions
• Trino platform support for Great Expectations
• New Presto-on-Hive ingestion source
• BigQuery ingestion now supports extraction of usage info from audit logs
• Fix to Looker ingestion to extract Explore Views from join names
• Fix to Tableau ingestion to avoid duplicating schema in URNs for upstream tables
• Simplify & annotate Redshift Usage source

Full Commit Log

• feat(gms): Expose kafka listener concurrency as a GMS setting by @jjoyce0510 in #4536
• feat(ingest): add option for external Spark cluster by @kevinhu in #4571
• fix(upgrade): Renaming kafka producer since it clashes with spring-internal by @dexter-mh-lee in #4573
• feat(GraphQL): Add data platform query to GraphQL API by @jjoyce0510 in #4574
• build(ui): Fix Windows UI lint by @mattmatravers in #4556
• doc: make note prominent on quickstart by @anshbansal in #4558
• fix(protobuf) minor bugfixes for protobuf by @leifker in #4553
• feat(docs) Improves docs around developing datahub, removes deprecated docs on building metadata service by @pedro93 in #4552
• chore: cleanup extra file by @anshbansal in #4541
• feat(snowflake): reduce permissions provisioned by default by @anshbansal in #4543
• fix(ingestion): Redshift usage refactoring - simplify, annotate, fix bugs by @rslanka in #4572
• fix(graphql): Adding PRE FabricType to GraphQL by @jjoyce0510 in #4582
• feat(search) - add DATETIME FieldType by @aditya-radhakrishnan in #4407
• fix(tableau): fix for incorrect schema returned by tableau api for sn… by @mayurinehate in #4577
• chore: update default cli for managed ingestion by @anshbansal in #4581
• feat(okta) - add support for filtering/searching when ingesting Okta groups and users by @aditya-radhakrishnan in #4586
• doc(snowflake): add example of table pattern by @anshbansal in #4580
• fix(doc): try to fix broken link by @daha in #4593
• fix(bigquery): incorrect lineage when views are present by @anshbansal in #4568
• feat(metadata-service): Supporting a configurable Authorizer Chain by @jjoyce0510 in #4584
• fix(search): Make sure home page and search pages are consistent by @dexter-mh-lee in #4588
• fix(browse): Reduce browse aggregation size by @dexter-mh-lee in #4601
• doc: add page for handling deprecations, breaking changes etc. by @anshbansal in #4590
• docs(GraphQL): fix typo by @Falci in #4605
• feat(search): Add SearchScore annotation to use fields for search ranking by @dexter-mh-lee in #4596
• feat(ingestion): Redshift Usage Source - simplify OperationalStats workunit generation. by @rslanka in #4585
• feat(tableau): add some logic to normalize table names in tableau by @gabe-lyons in #4609
• fix: urlencode slash in urns too by @daha in #4527
• fix(bigquery): fix lineage bug, improve docs, add dataset filter config by @anshbansal in #4607
• fix(protobuf) fix test instabilitity by @leifker in #4612
• fix(ui): Fix dashboard tags display by @jjoyce0510 in #4611
• feat(ui): Adding GraphQL queries to fetch entity deprecation status by @jjoyce0510 in #4614
• feat(ingest): enable connection string for all sqlalchemy datasources by @ms32035 in #4508
• fix(docs): add grant statements for redshift-ingestion by @Abhiram98 in #4559
• chore: fix lint and remove incorrect integration mark from unit tests by @anshbansal in #4621
• feat: adding gradle, pip cache via gh cache, docker cache via dockerhub by @anshbansal in #4387
• doc(scheduling): make it easier to find ui ingestion by @anshbansal in #4610
• feat(glue): add CatalogId parameter for cross-account access by @BoyuanZhangDE in #4608
• doc(cli): add env variables and options for ingest command by @anshbansal in #4598
• fix(ingest): Restricting pytest docker version to <0.12 by @treff7es in #4639
• fix(cypress) - add waits for cypress search test to remove flakiness by @aditya-radhakrishnan in #4640
• Revert "feat: adding gradle, pip cache via gh cache, docker cache via dockerhub" by @dexter-mh-lee in #4637
• feat(search): Only reindex if the mappings for an existing field changed by @dexter-mh-lee in #4629
• feat: add presto-on-hive metadata ingestion source by @jchen0824 in #4625
• feat(ingest): add trino platform for great expectations by @ms32035 in #4594
• fix(kafka): Stop overriding kafka registry props with empty values by @jsotelo in #4604
• [model]: Dataprocess instance entity to model datajob/jobflow runs by @treff7es in #4459
• feat(ingest): add Urn python library for DataJob, DataFlow, Domain and Tag by @tc350981 in #4618
• fix(ingestion): ensure source/sink reports are always logged by @anshbansal in #4592
• fix(ingestion): extract explore views from join name in Looker by @dyanarose in #4627
• feat(ingestion): Enable lower-casing of the name part of dataset urn if env variable is set. by @rslanka in #4649
• feat: Enable the ingestion of bigquery audit logs to parse usage info… by @tha23rd in #4441
• fix(ingest): Fix snowflake KEY_PAIR auth by @mkamalas in #4638
• fix(home): Fix issue where some browse cards are missing by @dexter-mh-lee in #4652
• fix(tableau): avoid duplicate schema in URNs for upstream tables by @maaaikoool in #4645
• feat(ingest): capture MSSQL table+column descriptions by @kevinhu in #4579
• feat(ml): bringing ml screens up to date w/ the modern ui layout & improving ml lineage by @gabe-lyons in #4651
• (feat:airflow) Add support to capture airflow executions + high level dataflow/jobs api by @treff7es in #4615
• fix(ingestion): add missing workunit ids by @anshbansal in #4657
• fix(ingestion): Adding missing init.py by @anshbansal in #4659
• fix(bigquery-usage): missing dependency by @anshbansal in #4661
• feat(cypress) - add cypress dashboard view to CI by @aditya-radhakrishnan in #4654
• feat(autocomplete): show fully qualified name in autocomplete by @gabe-lyons in #4663
• feat(ingestion) dbt: Fixing issue with strip_user_ids_from_email and adding owner_naming_pattern by @arunvasudevan in #4587
• fix(sqlparser): fix sqlparser breaking due to # sign by @anshbansal in #4662
• fix(ingestion): validate datasource in Tableau connector, before creating its upstream by @nandacamargo in #4613
• Added Relative Routing on the Users & Groups screen by @Ankit-Keshari-Vituity in #4664
• fix(airflow): Not importing emitters directly to eliminate unneeded dependency by @treff7es in #4668
• docs: remove ingestion source summary table by @maggiehays in #4670
• feat(ml): some machine learning followups by @gabe-lyons in #4669
• fix(search): Fix urn component settings by @dexter-mh-lee in #4672
• fix(ingestion): update example recipes by @anshbansal in #4660
• feat(theming): set custom logo without rebuilding by @gabe-lyons in #4674
• feat(data-platform): Add platform entities for the connectors we support by @dexter-mh-lee in #4676
• refactor(authorization): Add authorizedActor function to Authorizer interface by @dexter-mh-lee in #4678
• docs(tags) - add tags usage guide by @aditya-radhakrishnan in #4677
• fix(cli):Supress printing variables to logs during ingestion failure by @atulsaurav in #4566
• fix(docs): Improving Add Users Doc by @jjoyce0510 in #4679
• Fix/modal validations by @ShubhamThakre in #4673

New Contributors

• @Falci made their first contribution in #4605
• @ms32035 made their first contribution in #4508
• @jchen0824 made their first contribution in #4625
• @dyanarose made their first contribution in #4627
• @mkamalas made their first contribution in #4638
• @atulsaurav made their first contribution in #4566

Full Changelog: v0.8.32...v0.8.33