[Bug] DbtVirtualenvBaseOperator uses system-wide dbt instead of virtualenv-specific in v1.7.0 #1246
Labels
area:execution
Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc
bug
Something isn't working
execution:virtualenv
Related to Virtualenv execution environment
triage-needed
Items need to be reviewed / assigned to milestone
Astronomer Cosmos Version
Other Astronomer Cosmos version (please specify below)
If "Other Astronomer Cosmos version" selected, which one?
1.7.0
dbt-core version
1.8.7
Versions of dbt adapters
No response
LoadMode
DBT_LS_MANIFEST
ExecutionMode
VIRTUALENV
InvocationMode
SUBPROCESS
airflow version
2.4.6
Operating System
Google Cloud Composer (Linux-based)
If a you think it's an UI issue, what browsers are you seeing the problem on?
No response
Deployment
Google Cloud Composer
Deployment details
No response
What happened?
In version 1.7.0, the dbt command is being executed using the local node's Python path instead of the virtualenv path. This causes the operator to use the system-wide dbt installation rather than the one installed in the virtualenv.
The operator should use the Python path from the created virtualenv to execute dbt commands, ensuring that the correct (virtualenv-specific) version of dbt is used.
Relevant log output
How to reproduce
DbtVirtualenvBaseOperator
in Cloud Composer.Anything else :)?
This bug was introduced by #1200, which I created. I apologize for this bug.
I noticed that
self._py_bin
becomesNone
in theself.run_subprocess
method, so the command does not change during task execution.It appears that this issue is caused by
self
being bound to a different instance within theself.run_subprocess
method. I added the following logger at various points to confirm that the instance id differs at the time of run_subprocess:The logs show:
I'm wondering if this might be related to DAG pickle of Airflow.
I believe this may have been the direct cause of the bug reported in #958. When I investigated in version 1.5.0, I also found that properties were disappearing within the
run_subprocess
method.To address this, I plan to change the
invoke_dbt
method to be treated as a property, ensuring that therun_subprocess
method of the currently active instance is referenced, rather than directly assigning an instance method toinvoke_dbt
in__init__
.Are you willing to submit PR?
Contact Details
No response
The text was updated successfully, but these errors were encountered: