Trino ingestion with profiling enabled is not working as expected with table filter enabled

[anilreddygollapalli]

Describe the bug
When running trino ingestion with specific table name mentioned under filter --> tables --> table name(fully qualified), pipeline is not considering the filter specified.

To Reproduce
Steps to reproduce the behavior:

1. Create new trino ingestion pipeline with some profiling parameters
2. specify schema and table name under filter
3. Run the ingestion pipeline.
4. When you check the ingestion logs, you will not be able to see the table name under where condition while pulling profiling data.

Expected behavior
it should be able to query against the table specified under filter section.

Screenshots
NA

Desktop (please complete the following information):

• OS: Windows
• Browser: chrome
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[jjoyce0510]

Hi there - we use the profilingPattern to include and exclude assets for profiling..

Can you confirm you are using this field?

[anilreddygollapalli]

we will test this and let you know.

[anilreddygollapalli]

Hello Tamas,

Even after adding the below parameters to filter a specific table, it is not taking into consideration.

Reference : https://datahubproject.io/docs/generated/ingestion/sources/trino

    profile_pattern:
        allow:
            - test_tbl
    schema_pattern:
        allow:
            - test_db
    table_pattern:
        allow:
            - test_tbl
    profiling:
        enabled: true
        profile_table_level_only: false
        include_field_distinct_count: true
        include_field_distinct_value_frequencies: true
        include_field_histogram: true
        include_field_max_value: true
        include_field_mean_value: true
        include_field_median_value: true
        include_field_min_value: true
        include_field_null_count: true
        include_field_quantiles: true
        include_field_sample_values: true
        include_field_stddev_value: true

this record found in the ingestion logs and it showing as view_pattern instead of table_pattern, but when we checked the analytics tab, we dont see that table ingested with profiling info.

[2024-11-13 17:12:27,452] INFO {datahub.ingestion.source.sql.sql_config:113} - Applying table_pattern {'allow': ['test_tbl']} to view_pattern.

Please look into this issue.