Acryl DataHub v0.8.19.0

Release Highlights

• Add support for definitions in Mode connector
• Handle long queries in usage connector
• Fix issue where top users would sometimes have malformed email addresses

What's Changed

• datahub-project#3714 @grumbler fix(superset): handle dashboards without charts (datahub-project#3713)
• datahub-project#3729 @varunbharill feat(ingest): adding utilities methods to DataHubGraph class.
• datahub-project#3723 @dexter-mh-lee fix(docker): fix issues with base64 not working on some platforms
• datahub-project#3702 @lvicentesanchez fix(ingest): add source.config.connection.schema_registry_config to SchemaRegistryClient creation
• datahub-project#3706 @sgomezvillamor fix(ingestion): adds missing port to the connection bootstrap
• datahub-project#3725 @treff7es feat(ingest): trim long sql queries in usage connector
• datahub-project#3712 @mayurinehate fix(ingest): update trino source get_table_comment to handle not found error
• datahub-project#3726 @treff7es fix(ingest): get mysql geotypes properly
• datahub-project#3727 @treff7es fix(profiler): fix division by zero in pct_unique calculation
• datahub-project#3708 @treff7es docs(redshift): adding svv_table privilege requirement to redshift source
• datahub-project#3717 @varunbharill fix(docs): datahub cli docs have the wrong case for corpuser
• datahub-project#3721 @gabe-lyons fix(mode): support definitions in mode query
• datahub-project#3720 @gabe-lyons fix 0 artifact when there are no search recommendations

Acryl DataHub v0.8.18.1

DataHub Release 0.8.18 is here! See the full DataHub release notes here.

Release Highlights

1. Redshift Lineage: Out-of-the-box support for ingesting Dataset->Dataset lineage from Redshift system tables. Includes Tables, Views, and COPY from S3

   1. Video Demo

2. Apache Nifi Connector (Beta): Integration with Apache Nifi to extract DataJobs and DataFlows! Read the source docs here. This source is currently incubating in beta.

3. Mode Connector (Beta): Integration with Mode Analytics to extract reports, charts, and more! Read the source docs here. This source is currently incubating in beta.

4. Glossary Term Transformer: Allows users to add tags or glossary terms to entities based on a regex match filter (Shoutout to community Member ecooklin!)

5. Bug Fixes:

   1. [bigquery] Fix handling of partitioned & snapshotted tables for lineage usage, and basic table indexing.
   2. Add config to specify ca certificate path for datahub-rest sink
   3. [snowflake] Handling for special characters in snowflake databases and schemas.
   4. [cli] Fix deletes by search cannot accept auth token

What's Changed

• fix(bigquery-usage): Fix filters and event joining logic. by @varunbharill in datahub-project#3610
• feat(cli): adding a put command and docs by @swaroopjagadish in datahub-project#3614
• feat(profiler): dynamically combine queries by @hsheth2 in datahub-project#3572
• fix(ingest): rectify platform urn in kafka connect source by @mayurinehate in datahub-project#3624
• fix(ci): pin great expectations version by @swaroopjagadish in datahub-project#3629
• fix(delete): Remove logic that adds an invalid filter for platform field by @dexter-mh-lee in datahub-project#3619
• feat(cfg): Add configurable GCP log page size by @jjoyce0510 in datahub-project#3556
• fix(ingestion): Add config to specify ca certificate path for datahub-rest sink by @dexter-mh-lee in datahub-project#3632
• fix(ingest): workaround great-expectations compatibility issue by @hsheth2 in datahub-project#3634
• fix(ingestion): Handling for special characters in snowflake databases and schemas. by @rslanka in datahub-project#3635
• fix(ingestion): Enhance supported and unsupported base_objects_accessed for Snowflake Usage by @rslanka in datahub-project#3608
• fix(mongodb): enable version check for document size filter. by @varunbharill in datahub-project#3644
• fix(cli): fix for deletion cli by @anshbansal in datahub-project#3653
• feat(ingest): add --strict-warnings option by @hsheth2 in datahub-project#3665
• fix(ingestion): black formatting by @hsheth2 in datahub-project#3676
• feat(ingestion): anonymous usage stats by @kevinhu in datahub-project#3668
• feat(delete): support deleting by search w/ tokens by @gabe-lyons in datahub-project#3684
• refactor(profiling): clean up SQL query analysis by @hsheth2 in datahub-project#3674
• fix(snowflake): support geo types by @gabe-lyons in datahub-project#3686
• feat(profiler): add query combiner report statistics by @hsheth2 in datahub-project#3678
• feat(transformer) Adds glossary terms transformer by @ecooklin in datahub-project#3657
• feat(ingest): add nifi source in metadata-ingestion by @mayurinehate in datahub-project#3681
• feat(bigquery): support snapshot and partition tables in bigquery ingest & lineage by @gabe-lyons in datahub-project#3695
• fix(ingest): refactor urn deletion by @kevinhu in datahub-project#3694
• feat(ingestion): Add lineage support for Redshift source by @gabe-lyons in datahub-project#3697
• fix(ingestion): Disable query parser failure reporting to Datahub in redshift lineage by default by @treff7es in datahub-project#3699
• fix(nifi): add env in nifi config, add unit tests, fix nifi doc by @mayurinehate in datahub-project#3703
• feat(mode): add mode analytics ingestion source by @gabe-lyons in datahub-project#3710

Acryl DataHub v0.8.17.7

Release Highlights

• Mode analytics ingestion source

Changelog

• datahub-project#3710 @gabe-lyons feat(mode): add mode analytics ingestion source
• datahub-project#3703 @mayurinehate fix(nifi): add env in nifi config, add unit tests, fix nifi doc
• datahub-project#3707 @treff7es docs(redshift): Adding requirements for redshift permissions
• datahub-project#3687 @anshbansal docs(airflow): add some troubleshooting for error

Acryl DataHub v0.8.17.6

Release Highlights

• Add support for lineage in Redshift source
• (new) Apache Nifi source
• Fixes for BigQuery source related to partitioned tables

ChangeLog

• datahub-project#3699 @treff7es fix(ingest): disable query parser failure reporting to datahub in redshift lineage by default
• datahub-project#3697 @gabe-lyons feat(ingestion): Add lineage support for Redshift source
• datahub-project#3698 @gabe-lyons fix(ingest): revert accidental change to example recipe file_to_datahub_rest.yml
• datahub-project#3689 @anshbansal fix(docker): fix for local docker compose for M1
• datahub-project#3694 @kevinhu refactor(ingest): cli deletion function
• datahub-project#3695 @gabe-lyons feat(ingest): bigquery - support snapshot and partition tables during ingest & lineage
• datahub-project#3681 @mayurinehate feat(ingest): add nifi source
• datahub-project#3693 @jjoyce0510 fix(deletes): Fixing system metadata index deletes

Full Changelog: v0.8.17.5...v0.8.17.6

Acryl DataHub v0.8.17.5

Highlights

• Better reporting of performance of data profiling
• Adds glossary terms transformer
• Fixes for snowflake
• Fixes for delete cli using tokens
• cli usage stats

ChangeLog

• datahub-project#3657 @ecooklin feat(ingest): adds glossary terms transformer
• datahub-project#3678 @hsheth2 feat(profiler): add query combiner report statistics
• datahub-project#3686 @gabe-lyons fix(snowflake): support geo types
• datahub-project#3596 @dexter-mh-lee fix(elasticsearch): Use auth tokens to authorize curl requests in dockerize
• datahub-project#3683 @jjoyce0510 fix(auth): Tag Create Privileges were incorrectly using Dataset Create privileges
• datahub-project#3674 @hsheth2 refactor(profiling): clean up SQL query analysis
• datahub-project#3679 @anshbansal docs(business glossary): fix specification of the file
• datahub-project#3685 @jeffmerrick docs: change roadmap link in docs
• datahub-project#3684 @gabe-lyons feat(delete): support deleting by search w/ tokens
• datahub-project#3669 @anshbansal docs(scheduling): re-arrange docs related to scheduling, lineage, CLI
• datahub-project#3668 @kevinhu feat(ingestion): anonymous usage stats
• datahub-project#3675 @dexter-mh-lee fix(kafka): Add back changes to centralize kafka config
• datahub-project#3677 @gabe-lyons fix(build): correcting m1 preflight check
• datahub-project#3676 @hsheth2 fix(ingestion): black formatting
• datahub-project#3667 @swaroopjagadish feat(metadata-model): adding metadata model doc generation and upload…
• datahub-project#3660 @dexter-mh-lee fix(search): Improve search and recs performance
• datahub-project#3665 @hsheth2 feat(ingest): add --strict-warnings option
• datahub-project#3662 @anshbansal fix(docs): fix build on m1
• datahub-project#3659 @dexter-mh-lee refactor: remove unused PDL files
• datahub-project#3654 @pedro93 fix(test): Adds docker engine configuration checks before running docker-based tests for metadata-io

Full Changelog: v0.8.17.4...v0.8.17.5

Acryl DataHub v0.8.17.4

Highlights

• Fixes mongodb compatibility with versions <4.4
• Fixes snowflake lineage for accounts with and without access history
• Fixes bug in delete command in cli

ChangeLog

• datahub-project#3653 @anshbansal fix(cli): fix response handling for deletion cli
• datahub-project#3650 @bartlomiejolma docs: correct title of postgres gms
• datahub-project#3649 @jjoyce0510 docs(graphql): adding additional GraphQL docs
• datahub-project#3652 @treff7es feat(build): Preflight script for metadata ingestion on m1
• datahub-project#3643 @gabe-lyons fix(ui): fix bug where key/value toggle would appear on schema tabs with no fields
• datahub-project#3646 @gabe-lyons fix(ui): fix issue where markdown links are unclickable
• datahub-project#3647 @swaroopjagadish feat(build): add incremental builds for python
• datahub-project#3648 @maggiehays docs: update to DataHub Adopter logos & Townhall details
• datahub-project#3644 @varunbharill fix(ingest): mongodb enable version check for document size filter.
• datahub-project#3640 @swaroopjagadish fix(docs): small fixes to docs and docker images for custom metadata …
• datahub-project#3434 @EnricoMi feat(quickstart): simplify docker generate and compare script
• datahub-project#3608 @rslanka fix(ingest): enhance supported and unsupported base_objects_accessed for snowflake
• datahub-project#3639 @gabe-lyons docs(business glossary): document the business glossary file format
• datahub-project#3597 @gabe-lyons feat(autorender): Auto render aspects that don't have frontend components in the UI

Acryl DataHub v0.8.17.3

Highlights

• Adds support for CA certificates for rest sink
• Fixes snowflake ingestion with special characters in databases and schemas
• Works around issues with latest great-expectations versions

Changelog

• datahub-project#3638 @jjoyce0510 fix(ui): fixing ownership bug
• datahub-project#3635 @rslanka fix(ingestion): Handling for special characters in snowflake databases and schemas.
• datahub-project#3634 @hsheth2 fix(ingest): workaround great-expectations compatibility issue
• datahub-project#3632 @dexter-mh-lee feat(ingest): add ca_certificate_path to rest sink

Full Changelog: v0.8.17.2...v0.8.17.3

Acryl DataHub v0.8.17.2

What's Changed

• Ingestion
  • Fix for kafka-connect source producing invalid platform urn
  • Improvements to BigQuery source
  • Improvements to profiling (dynamically combining queries)

Full ChangeLog

• datahub-project#3631 @dexter-mh-lee fix(ui): fix issue with recently viewed and most popular recs not showing up
• datahub-project#3556 @jjoyce0510 feat(ingest): Add configurable GCP log page size to bigquery sources
• datahub-project#3582 @adriangb fix(tests): fix typo in test name
• datahub-project#3626 @claudio-benfatto fix(metadata-service): fix debug logging in MAE producer
• datahub-project#3630 @shirshanka feat(metadata-service): support for custom model extensions without forks
• datahub-project#3619 @dexter-mh-lee fix(delete): remove logic that adds an invalid filter for platform field
• datahub-project#3629 @swaroopjagadish fix(ci): pin great expectations version
• datahub-project#3617 @varunbharill fix(redshift-usage): Add docs for redshift usage ingestion.
• datahub-project#3620 @dexter-mh-lee fix(search): Fix issue when query is empty
• datahub-project#3627 @serefacet docs(okta): Added Okta Logout Settings
• datahub-project#3624 @mayurinehate fix(ingest): rectify platform urn in kafka connect source
• datahub-project#3618 @jeffmerrick fix(docs-website): fix incorrect managed url
• datahub-project#3621 @jjoyce0510 fix(metadata-service): fix Profiling NPE + misc improvements
• datahub-project#3606 @jjoyce0510 doc(components): Adding DataHub components overview
• datahub-project#3572 @hsheth2 feat(profiler): dynamically combine queries
• datahub-project#3611 @gabe-lyons adding es logo

Full Changelog: v0.8.17.1...v0.8.17.2

Acryl DataHub v0.8.17.1

Release Highlights (Python)

• Fixes an issue with BigQuery usage source introduced in 0.8.17.0
• Adds a new command put to the datahub cli

What's Changed

• datahub-project#3614 @swaroopjagadish feat(cli): adding a put command and docs
• datahub-project#3610 @varunbharill fix(bigquery-usage): fix filters and event joining logic.
• datahub-project#3591 @aseembansal-gogo fix(docs): move transformers to be under metadata ingestion
• datahub-project#3488 @jeffmerrick docs:remove hubspot form and instead link to acryldata.io
• datahub-project#3598 @jjoyce0510 feat(auth): Metadata Service Authentication!
• datahub-project#3605 @arunvasudevan Remove GMS GraphQL Service
• datahub-project#3603 @jjoyce0510 feat(boot): Ingest datahub root user info on boot
• datahub-project#3600 @robscriva fix(ui): add default monospace font to font-family in the entity tabs

Full Changelog: v0.8.17.0...v0.8.17.1

Acryl DataHub v0.8.17.0

What's Changed

• fix(pypi): fixing package metadata to reflect source and changelog co… by @shirshanka in #3

• datahub-project#3595 @dexter-mh-lee fix(browse): fix browse response size issue
• datahub-project#3593 @swaroopjagadish feat(model): adding a field to capture unmodeled field level properties
• datahub-project#3590 @mayurinehate feat(ingest): add bigquery sink connector lineage in kafka connect source
• datahub-project#3594 @jeffmerrick docs: add docs nav links
• datahub-project#3592 @remisalmon fix(ingest): fix dbt source platform when disable_dbt_node_creation is False
• datahub-project#3584 @rslanka feat(ingestion): Adds --dry-run and --preview options to datahub ingest command.

Full Changelog: v0.8.16.12...v0.8.17.0

Known Issues

• The bigquery-usage ingestion plugin will fail to retrieve the log entries if there are no allow/deny patterns specified for the table_pattern config parameter. Fix will be available in the next release.