on-run-end hook take a long time to finish

[papost]

Hey folks,

I am trying to initialize and use the elementary package (it's awesome), but I am skeptical if we are going to use it. Even If I run a single view model with no transformations & calculations, it takes a long time to finish the on-run-end hook of elementary. Although it is a great tool, this problem will waste a lot of our time while we are developing new models. Am I missing something? Does anyone have the same issue as me? Waiting for your answers!

[oravi]

Hi @papost !

Thanks for reaching out and sorry to hear that it takes a long time for you to run. Please let me share some context on what happens during the on-run-end hook and then we can think together what might be the issue.

on-run-end is responsible for the following -

• Upload metadata about all the resources in your project into plain tables in your warehouse. For example, all your model configurations and settings are being uploaded into dbt_models, all the tests configurations are being uploaded to dbt_tests , etc. We do have a caching mechanism here that will automatically detect if there's no change in the configuration - nothing will be re-uploaded during the next dbt runs. Only if you change a model setting for example, we will reupload the metadata into dbt_models.

  • For some huge projects this could take a relatively long time (a few minutes) but only for the first run. This can be also disabled using a flag and then you can trigger this upload on a different schedule. I can share details about it if we understand that this is the issue in your case.

• Upload run results about all the models or tests that participated in this run. This part uploads details like execution times for each model / test, execution status for each model / test, errors and traceback for each failed model / test, and more.

  • Usually this is not the heavy part and must be uploaded as the result is only available for the current run. That said, we have a flag to disable this as well if really needed.

• Upload invocation details about the command invocation. Details like what selector were used during the run, what was the command and more.

  • Usually this part is very lightweight as it's only one insert query.

Another thing to note - some of our users enable this metadata and run results collection only in prod so it doesn't slow down their local work. You can see the following to learn how to configure it -

models:
  elementary:
    +schema: "elementary"
    +enabled: "{{ target.name == 'prod' }}"