Compute model adjustments
BigQuery queries can use different compute models based on:
On-demand (default): Billed per data processed
Project-level assignment: Billing project assigned to a reservation
Query-level assignment: Query with specific reservation (overrides project settings)
These levels provide flexibility to optimize compute models across your workloads.
Pipeline Level Assignment
Masthead optimizes compute at the pipeline level, delivering significant savings with minimal effort. Ready-to-use solutions are available for common orchestration technologies to assign compute models across all jobs.
dbt
Use the official dbt-reservations package for model-level reservation assignment.
Dataform
Use the official dataform-reservations package for action-level reservation assignment.
Airflow
reservation_id = 'RESERVATION_ID'
sql_with_overridden_reservation = f"""SET @@reservation='{reservation_id}';
{your_sql}
"""
BigQueryInsertJobOperator(
task_id="run_query",
configuration={
"query": {
"query": sql_with_overridden_reservation
}
},
...
)
Project Level Assignment
For tools that run BigQuery jobs without job-level compute control (e.g., Looker), use a dedicated project and a project-level assignment.
Migration steps:
Assign the project to a reservation with job type
QUERY
Grant BigQuery Job User role to the service account
Update Billing Project ID in the tool's BigQuery connection (if unavailable, adjust
Project ID
instead)Verify functionality
Fully qualified references for datasets and tables are required (starting with project_id
where the data resides). Otherwise BigQuery will be looking for the table in the billing project.
Done! Queries from this tool now use the assigned reservation capacity.
Last updated