Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] dbt source rendering: Use . instead of _ between schema & table #1308

Open
1 task
corsettigyg opened this issue Nov 7, 2024 · 3 comments
Open
1 task
Labels
area:rendering Related to rendering, like Jinja, Airflow tasks, etc dbt:source Primarily related to dbt source command or functionality enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone
Milestone

Comments

@corsettigyg
Copy link
Contributor

Description

Currently in cosmos, sources are rendered as emptyOperators with a task_id of {schema}_{table}_source

Image

The main issue with this naming convention is that the usual convention of naming tables with the schema uses dot (.) instead of underscore (_), and by using underscore in the naming of our tasks, it is not possible to properly identify where the schema name ends and where the table name begins.

Use case/motivation

By having a logical separation of the schema and the table, it is easy to extend cosmos and add extra features to the sources. At my company we are adding sensors before the sources, but the process becomes way harder because of something as simple as using the _ instead of . in the name of the task.

Related issues

No response

Are you willing to submit a PR?

  • Yes, I am willing to submit a PR!
@corsettigyg corsettigyg added enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone labels Nov 7, 2024
@dosubot dosubot bot added area:rendering Related to rendering, like Jinja, Airflow tasks, etc dbt:source Primarily related to dbt source command or functionality labels Nov 7, 2024
@tatiana tatiana added this to the Cosmos 1.8.0 milestone Nov 11, 2024
@tatiana
Copy link
Collaborator

tatiana commented Nov 11, 2024

@corsettigyg, thanks for your feedback! Would this address the issue if we were to use __ instead of _ as a separator between schema & table?

@corsettigyg
Copy link
Contributor Author

@tatiana totally

@tatiana
Copy link
Collaborator

tatiana commented Nov 20, 2024

I just discussed this with @pankajkoti and @pankajastro, and I was concerned about making this change backwards compatible.

The preferred approach is to make it backwards compatible by having yet another configuration—and having a ticket to make the default __ in Cosmos 2.0. We'll squeeze in this change in Cosmos 1.8. The configuration could be within RenderConfig.

@corsettigyg, would you by any chance have bandwith of contributing to this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:rendering Related to rendering, like Jinja, Airflow tasks, etc dbt:source Primarily related to dbt source command or functionality enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone
Projects
None yet
Development

No branches or pull requests

2 participants