Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support Passing in {{ target.user }} for BQ Query Comments #1110

Open
3 tasks done
iamtodor opened this issue Feb 20, 2024 · 9 comments
Open
3 tasks done

[Feature] Support Passing in {{ target.user }} for BQ Query Comments #1110

iamtodor opened this issue Feb 20, 2024 · 9 comments
Labels
enhancement New feature or request

Comments

@iamtodor
Copy link

iamtodor commented Feb 20, 2024

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Hey folks!

I have a question in regards to Big query job comment(job-label): recently we upgraded to 1.4.9 and I would like to utilize this API https://docs.getdbt.com/reference/project-configs/query-comment#bigquery-include-query-comment-items-as-job-labels
I walked thru the doc and I was able to have the default one with the config:

query-comment:
  job-label: True

Our pipeline process looks like this: Airflow task calls dbt run from python operator. Obviously, we have plenty of tasks in the DAG. The business task is to gather information in regard to how many bytes are processed per hour per DAG. For this, I use the built-in region-us.INFORMATION_SCHEMA.JOBS from BigQuery. This is where labels come into play. As I mentioned previously python operator runs dbt task, and from the Python task I am able to find out the DAG name. Hence I am capable of putting the DAG name into dbt run command the way, for instance, we pass vars so I would like to ask/understand whether it's possible, and if so, then how?

I found that I also can add a custom comment in the way https://docs.getdbt.com/reference/project-configs/query-comment#append-a-custom-comment:

query-comment:
  comment: "run by {{ target.user }} in dbt"
  append: True

but dbt knows nothing about Airflow, while Airflow knows about dbt.

If I miss something I would be glad to clarify it. And, if I am moving towards the wrong direction I would be pleased to be corrected :)

I asked the same question in slack https://getdbt.slack.com/archives/C99SNSRTK/p1708089270625579 , but got no attention

Describe alternatives you've considered

Seems like there are no alternatives

Who will this benefit?

BigQuery users who would like to distinguish jobs per DAG

Are you interested in contributing this feature?

There is a chance, with all the context and pitfalls to be provided

Anything else?

Nope

@iamtodor iamtodor added enhancement New feature or request triage labels Feb 20, 2024
@dbeatty10 dbeatty10 added the bytes_processed Issues related to cost tracking in BigQuery label May 30, 2024
Copy link
Contributor

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

@github-actions github-actions bot added the Stale label Nov 27, 2024
@iamtodor
Copy link
Author

Please don't make it stale

@github-actions github-actions bot removed the Stale label Nov 28, 2024
@amychen1776
Copy link

Hello - I'm going to close out this issue because this is neither a bug or feature request. Please ask these questions in our community related sites like Discourse or Slack.

As an aside, you might want to take a look at environmental variables through - that might work since you need to supply this via an yml file config.

@amychen1776 amychen1776 closed this as not planned Won't fix, can't repro, duplicate, stale Jan 6, 2025
@iamtodor
Copy link
Author

iamtodor commented Jan 6, 2025

@amychen1776 could you please let me know what I should do to make this issue a feature request?

@amychen1776
Copy link

@iamtodor were you able to get it working with an env_var?

@iamtodor
Copy link
Author

iamtodor commented Jan 7, 2025

@amychen1776 nope, I wasnt

@amychen1776 amychen1776 changed the title Pass (modify/append) BigQuery job label as a CLI flag [Feature] Support Passing in {{ target.user }} for BQ Query Comments Jan 7, 2025
@amychen1776 amychen1776 removed triage bytes_processed Issues related to cost tracking in BigQuery labels Jan 7, 2025
@amychen1776
Copy link

I'll adjust the issue then

@amychen1776 amychen1776 reopened this Jan 7, 2025
@iamtodor
Copy link
Author

iamtodor commented Jan 7, 2025

@amychen1776 thank you!

@amychen1776
Copy link

@iamtodor My coworker just flagged that there is a way around this using a macro which would be the way to get target.user https://docs.getdbt.com/reference/project-configs/query-comment#advanced-use-a-macro-to-generate-a-comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants