Release Highlights

Potential Downtime

This release introduces substantial improvements to search ranking which require reindexing indices.

During the reindexing:

a system-update job will set indices to read-only and create a backup/clone of each index
new components will be prevented from start-up until the reindex completes
Helm deployments will go into read-only mode and new ingestion runs will fail

This process can take anywhere from 5 minutes to multiple hours; as a rough estimate, please expect it to take 1 hour for every 2.3 million entities. After the reindex is complete, please check your ingestion run to re-run any that did not complete.

User Experience

New Search and Browse Experience

We have some really exciting improvements to the DataHub user experience in this release! The new search and browse experience, which was first made available in the previous release behind a feature flag, is now on by default. Check out our release notes for v0.10.5 to get more information and documentation on this new Browse experience.

Learn all about the new Search and Browse experience!

Improvements to Search

In addition to the ranking changes mentioned above, this release includes changes to the highlighting of search entities to understand why they match your query. You can also sort your results alphabetically or by last updated times, in addition to relevance. In this release, we suggest a correction if your query has a typo in it.

See the Search improvements in action!

Manage Home Page Posts

In this release we now enable you to create and delete pinned announcements on your DataHub homepage! If you have the “Manage Home Page Posts” platform privilege you’ll see a new section in settings called “Home Page Posts” where you can create and delete text posts and link posts that your users see on the home page.

OpenAPI Endpoints Expanded

OpenAPI entity and aspect endpoints expanded to improve developer experience when using this API with additional aspects to be added in the near future.

Metadata ingestion

Added support for Confluent S3 Sink Connector, extracting stored procedures and jobs from mssql, and snowflake shares. Additionally, sql parsing source now converts query logs into CLL and usage.

Developer Experience

The CLI now supports recursive deletes.

Versioned documentation

Starting from this release, we support versioned documentation on the datahub docs site! Select the version you’re on and browse docs specifically at that version.

Performance Improvements

Batching of default aspects on initial ingestion (SQL)
Improvements to multi-threading. Ingestion recipes, if previously reduced to 1 thread, can be restored to the 15 thread default.
Gradle 7 upgrade moderately improves build speed
DataHub Ingestion slim images reduced in size by 2GB+

Important Bug Fixes

Glue Schema Registry fixed

Deprecation Notice

MAE Events are no longer produced. MAE events have been deprecated for over a year.

What's Changed

feat(ingest/presto-on-hive): enable partition key for presto-on-hive by @zheyu001 in #8380
feat(classification): allow parallelisation to reduce time by @mayurinehate in #8368
feat(ingest): Add metabase database id to platform instance mapping by @k-popov in #8359
feat(ingest): add ability to read other method types than GET for OAS ingest recipes by @jsmilkstein in #8303
fix(ingest): fix data platform urn in dataset_urn_to_key and dataset_key_to_urn by @Masterchen09 in #8209
fix(ingest/s3): wrong sorting in case of multi-partition key by @anshbansal in #8536
fix(ingest/presto): fix presto on hive test failures by @hsheth2 in #8548
Cypress test for managing groups by @kkorchak in #8520
feat(ingest/kafka-connect): add support for Confluent S3 Sink Connector by @tusharm in #8298
Variable rename - Allows deselection of members in add members modal for a group by @Sukeerthi31 in #8529
fix(ingest/s3): catch no such bucket exception instead of failing by @anshbansal in #8549
fix(ingest): add tableau sqlglot dep by @hsheth2 in #8552
fix(ingetion/mssql): convert dataset urns to lowercase by @siddiquebagwan in #8551
Fix flaky add_user smoke test by @kkorchak in #8471
feat(ci): use docker registry cache by @hsheth2 in #8544
fix(glue): restore glue configurations by @RyanHolstien in #8533
build(release): Update files for 0.10.5 release by @iprentic in #8556
docs(release): Update updating-datahub.md for 0.10.5 release by @iprentic in #8557
feat(ingestion/snowflake): use user email-id in urn generation for top users stat by @siddiquebagwan in #8513
docs(development.md): Minor grammatical error by @PauloGoncalvesLima in #8558
fix(usage): Update index lifecycle policy to not delete old datahub usage events by @iprentic in #8565
fix(ui): Simplify background color for Entity Health Status popover by @jjoyce0510 in #8559
fix: add --write args on pre-commit prettier by @yoonhyejin in #8560
docs(observe): Add feature doc for Freshness Assertions by @jjoyce0510 in #8547
docs(updating): add details on Unified Search & Browse experience by @maggiehays in #8568
fix: fix features section by @yoonhyejin in #8571
feat(ingest): allow lower freq profiling based on date of month/day of week by @anshbansal in #8489
fix(stats): default to 3 months by @anshbansal in #8566
fix(aspect): count query only for relevant aspect index by @iprentic in #8569
feat(quickstart): bump quickstart start periods more by @hsheth2 in #8573
Origin/cypress test for managing policies by @kkorchak in #8554
feat(ui) Show source documentation when editing entity documentation by @chriscollins3456 in #8516
fix(ingest): handle redaction of configs with int keys by @hsheth2 in #8545
fix(ingest/snowflake): maintain qualified name casing, do not lowercase by @mayurinehate in #8574
feat(docs): add github repo links to readme and docs by @yoonhyejin in #8422
feat(ebean): Add metric in ebean aspect DAO for failed tries, as well as failed operation… by @iprentic in #8576
refactor(search) Use search across multiple-entities API, deprecate Aggregator classes by @iprentic in #8498
feat(siblings): dont show multiple platform icons if the siblings are ghost nodes by @gabe-lyons in #8543
docs(lineage): Add description to make_lineage_mce by @eboneil in #8596
doc(ingest/log): failure log at pipeline level document by @anshbansal in #8591
Dataset ownership test by @kkorchak in #8583
doc(release): release notes for 0.2.10 by @anshbansal in #8599
docs(release): fix typo by @anshbansal in #8600
feat(ui): apply views to: domains, containers, terms by @eboneil in #8572
feat(search): embedded view dropdown by @joshuaeilers in #8598
fix(ingest/file): remove entity_type_counts and aspect_counts by @hsheth2 in #8586
fix(ingest): use hive pure_sasl variant by @hsheth2 in #8570
Feat(ingest/ldap)fix list index out of range error by @alplatonov in #8525
harden autocomplete test by @joshuaeilers in #8603
feat(ui/graphql) Add ability to sort search results from search results page by @chriscollins3456 in #8595
fix(ingest): Add client_certificate_path for rest client cert instead of ca_certif… by @mkamalas in #8581
refactor(graphql): extract code into metadata-io part 1 by @anshbansal in #8607
docs(ingest): update s3 and gcs doc with concept mapping by @mayurinehate in #8575
Fix(ingestion/clickhouse) move to two tier sqlalchemy by @alplatonov in #8300
fix(cypress): attempt to fix autocomplete test by @joshuaeilers in #8619
fix(cleanup): cleanup of 2 sub-modules by @anshbansal in #8616
docs(ingsetion/csv-enricher): fix sample csv mentioned in Docstrings by @siddiquebagwan in #8432
feat(ingest): allow relative start time config by @mayurinehate in #8562
fix(ingest/airflow): make inlets work again by @hsheth2 in #8631
feat(ingest/s3): Adding option to pass in any spark config property to s3 source by @treff7es in #8621
feat(impact analysis): allow deep linking of url params in impact analysis by @gabe-lyons in #8617
feat(ui) Display combined sibling results in search + 2 minor updates by @chriscollins3456 in #8602
feat(ui) Display consistent search results in embedded searches by @chriscollins3456 in #8597
feat(ingest): Add DataHub source by @asikowitz in #8561
fix(ingest/okta): fix event_loop RuntimeError with nested asyncio by @skrydal in #8637
fix(ingest/kafka): use SchemaReference properties instead of dict access by @Deepankarkr in #8615
feat(ingestion/ldap): flag to ingest ldap users with email instead of username by @Deepankarkr in #8606
Combine siblings in autocomplete by @joshuaeilers in #8610
fix(ingest): avoid mutable defaults in powerbi dataclass by @hsheth2 in #8609
chore(spring): upgrade minor versions of spring components by @david-leifker in #8627
docs(quickstart): quickstart documentation, clarification on production by @david-leifker in #8628
feat(datahub-ingestion): refactor datahub ingestion slim images by @david-leifker in #8515
bug(8584): emit data_platform_instance aspect if the config has platform_instance by @jinlintt in #8585
chore(snappy): fix snappy version constraint by @david-leifker in #8629
chore(hazelcast): update hazelcast version by @david-leifker in #8633
feat(graphql) Support exists operator in GraphQL Search API by @jjoyce0510 in #8652
[fix] [health ui] Removing ghost 0 for health signals on search cards by @jjoyce0510 in #8587
fix(data products): removing data products filter in search as its not indexed on entity documents by @gabe-lyons in #8650
feat(ingest/bigquery): add tag to BigQuery clustering columns by @ANich in #8495
fix(ingest/snowflake): fix usage enum bug by @hsheth2 in #8649
feat(ingest/dbt-cloud): use job-based graphql queries by @hsheth2 in #8647
Add and remove documentation and link for dataset by @kkorchak in #8604
Lineage column level test by @kkorchak in #8641
tests(search): search golden tests by @eboneil in #8605
Add test case for dataset deprecation test by @kkorchak in #8646
docs(ingest/kafka-connect): add details on platform instance mapping by @mayurinehate in #8654
docs(ingest/airflow): add capture_executions to docs by @hsheth2 in #8662
Fix a few view select issues by @joshuaeilers in #8670
feat(search): Add word gram analyzer for name fields by @iprentic in #8611
fix(docker): misc docker fixes by @david-leifker in #8677
tests(search): more golden tests by @eboneil in #8683
test(ingest/vertica): Skip integration test failing CI; support arm Macs by @asikowitz in #8694
ci: add needs_artifact_download output for ingestion image by @hsheth2 in #8695
logs(ingestion/unity): Hide stack trace on sql parse failure logs by @asikowitz in #8657
feat(ingestion/powerbi): support multiple tables as upstream in native SQL parsing by @siddiquebagwan-gslab in #8592
build(ingest): Bump pydantic pin by @asikowitz in #8660
remove(ingest/snowflake): Remove legacy snowflake lineage by @asikowitz in #8653
fix(ingest/ldap): Handle case when 'objectClass' not in attrs by @asikowitz in #8658
fix(ui) Remove new Role entity from searchable entity types by @chriscollins3456 in #8655
fix(java) Use alias for name search sorting and fix missing mappings by @chriscollins3456 in #8648
feat(ui) Create page for managing home page posts by @chriscollins3456 in #8707
fix(ingest/powerbi): add sqlglot python dep by @hsheth2 in #8704
ci(ingest): make ingestion caching rules correct by @hsheth2 in #8685
fix(cleanup): cleanup of 1 sub-module by @anshbansal in #8678
fix(policies): fix concurrent modification exception by @RyanHolstien in #8681
fix(ingest/bigquery): Add config option to create DataPlatformInstance, default off by @asikowitz in #8659
feat(ingest/looker): Record observed lineage timestamps for Looker and LookML sources by @ANich in #7735
feat(ingest/mssql): load jobs and stored procedures by @RChygir in #5363
fix(ingestion/kafka-connect): update retrieval of database name in Debezium SQL Server by @Starkie in #8608
feat(ingest/snowflake): tables from snowflake shares as siblings by @mayurinehate in #8531
feat(ingest/sql-queries): Add sql queries source, SqlParsingBuilder, sqlglot_lineage performance optimizations by @asikowitz in #8494
highlight matched fields in search results by @joshuaeilers in #8651
Add links to glossary term cards without counts by @joshuaeilers in #8705
fix non sibling document links by @joshuaeilers in #8724
refactor(policies): Rename edit all privilege to edit entity by @jjoyce0510 in #8722
feat(java/ui) Add search suggestions to our search experience by @chriscollins3456 in #8710
fix(cypress) Fix login.js cypress test by @chriscollins3456 in #8719
Fixes for faling login.js and managing_groups.js Cypress tests by @kkorchak in #8725
fix(kafka-setup): remove dependency confluent docker utils by @lix-mms in #8715
docs(docs): add native versioning by @yoonhyejin in #8714
config(ingest/rest): Update rest sink defaults to retry more often by @asikowitz in #8729
chore(jackson): update to released version of jackson by @david-leifker in #8674
fix(examples): fix typo in business glossary bootstrap yml by @mayurinehate in #8703
fix(schemaRegistry): change api servlet check to only apply to internal to fix glue support by @RyanHolstien in #8693
fix(ingest): stateful redundant run skip handler by @mayurinehate in #8467
fix(superset): get alternate platform value if sqlalchemy_uri param is missing by @akhil7philip in #8667
feat(ingest): support writing configs to files by @hsheth2 in #8696
feat(search): De-duplicate scale factors across entities by @iprentic in #8718
test(lineage): Add test for scroll across lineage by @iprentic in #8728
feat(ingest/metabase): detect source table for cards sourced from other cards by @k-popov in #8577
(ingestion) bug fix: emit platform instance aspect for dataset in Databricks ingestion by @jinlintt in #8671
feat(config): Turn on new search & browse experience by default by @iprentic in #8737
chore(ingest/s3) Bump Deequ and Pyspark version by @treff7es in #8638
docs(ingest/openapi): Downgrade status from CERTIFIED to INCUBATING by @asikowitz in #8736
feat(health): Adding Entity Health Status to the Lineage Graph View by @jjoyce0510 in #8739
build(ingest): Pin mypy-boto3-sagemaker directly by @asikowitz in #8746
feat(ingest/datahub): Improvements, bug fixes, and docs by @asikowitz in #8735
docs(obseve): Adding Volume Assertion Guide by @jjoyce0510 in #8706
fix(ingest/okta): Removed code closing okta's event_loop by @skrydal in #8675
fix(highlight): disable full name highlight by @joshuaeilers in #8750
fix(ui): hide pages from web crawlers by @hsheth2 in #8738
docs: add index pages for feature/deployment guides by @hsheth2 in #8723
feat(docs): move versioned_sidebars to static-assets by @yoonhyejin in #8743
docs(observe): DataHub Operation freshness assertion guide by @zmcnellis in #8749
feat(cli): support recursive deletes by @hsheth2 in #8709
fix(ingest/bigquery): Handle null view_definition; remove view definition hash ids by @asikowitz in #8747
feat(ingest/usage): Make cumulative query character limit configurable by @asikowitz in #8751
fix(ingest/athena): Fixing db container id by @treff7es in #8689
feat(systemMetadata): add pipeline names to system metadata by @hsheth2 in #8684
ci: separate airflow build and test by @mayurinehate in #8688
fix(ingest/athena): fix container linting by @hsheth2 in #8761
fix(datahub-frontend) Give permission for start.sh so it can run by @rtekal in #8594
feat(sql-parser): schema-aware output column casing by @hsheth2 in #8760
fix(ingest/bigquery): Filter out fine grained lineage with no upstreams by @asikowitz in #8758
feat(iceberg): Upgrade Iceberg ingestion source to pyiceberg 0.4.0 by @cccs-eric in #8357
Allow frontend to use http proxy by @githendrik in #8691
docs(observe): Dataset Profile volume assertion guide by @zmcnellis in #8764
docs:fix broken img links under managed-datahub by @yoonhyejin in #8769
fix:small typo on graphql tutorial by @yoonhyejin in #8741
refactor(build): upgrade to gradle 7 & guava update by @david-leifker in #8745
fix(siblings): space icons out by @joshuaeilers in #8767
chore(build): upgrade gradle wrapper by @hsheth2 in #8776
feat(EntityService): batched transactions and ebean updates by @david-leifker in #8456
fix(frontend): Fix"Logout with OIDC not working" by @FirKys in #8773
docs:upgrade docusaurus version by @yoonhyejin in #8770
fix:change global graph url to static-assets by @yoonhyejin in #8742
doc(tests): fix endpoint param to push results by @anshbansal in #8783
fix(elastic): improve error handling for profiling by @anshbansal in #8785
chore(analytics): bump version by @joshuaeilers in #8786
docs(session): add documentation for session token duration and fix default by @RyanHolstien in #8791
fix(ingest/datahub): Support postgres; build(postgres): Modernize postgres docker setup by @asikowitz in #8762
feat(airflow-plugin): add package type information by @mayurinehate in #8795
feat(systemMetadata): Adding a lastRunId field system metadata by @jjoyce0510 in #8672
added support for group-owners in dataflow entities by @dnks23 in #8154
fix(ingest/tableau): fix tableau native CLL for snowflake, add type annotations by @mayurinehate in #8779
fix(ingest/bigquery): fix partition and median queries for profiling by @mayurinehate in #8778
docs: add datahub source to integrations page by @hsheth2 in #8787
chore(ingest): upgrade sqlglot fork by @hsheth2 in #8775
docs: minor fix on versioning navbar and dropdown by @jeffmerrick in #8790
feat(ingest): drop sql_metadata parser by @hsheth2 in #8765
fix(ingest): drop wrap_aspect_as_workunit method by @hsheth2 in #8766
feat(search): Also de-duplicate the field queries based on field names by @iprentic in #8788
feat(openapi): entity endpoints & analytics raw by @david-leifker in #8537
docs(db-retention): update with default setting by @david-leifker in #8797
fix(custom-search): fix custom search to be able to use unquoted query by @david-leifker in #8805
feat: add feedback widget by @yoonhyejin in #8732
fix(gms): Fixed Recently Viewed section for users with '@' in the URN. by @skrydal in #8754
fix(spark-test): upgrade gradle and fix spark smoke test by @david-leifker in #8777

New Contributors

@zheyu001 made their first contribution in #8380
@jsmilkstein made their first contribution in #8303
@tusharm made their first contribution in #8298
@PauloGoncalvesLima made their first contribution in #8558
@Deepankarkr made their first contribution in #8615
@ANich made their first contribution in #8495
@siddiquebagwan-gslab made their first contribution in #8592
@RChygir made their first contribution in #5363
@Starkie made their first contribution in #8608
@akhil7philip made their first contribution in #8667
@zmcnellis made their first contribution in #8749
@githendrik made their first contribution in #8691
@FirKys made their first contribution in #8773
@dnks23 made their first contribution in #8154

Full Changelog: v0.10.5...v0.11.0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v0.11.0