Dataform reservations

Overview

Action-level assignment optimizes BigQuery resource usage by assigning compute reservations to Dataform actions based on Masthead recommendations. This system enables businesses to efficiently manage their BigQuery costs and resource allocation with minimal manual intervention.

Key Benefits

• Cost optimization: Automatically route high-compute workloads to reserved slots and high-data volume workloads to on-demand capacity

• Automated reassignment: Once configured, reservations are applied automatically based on action categorization

Usage example with publish action

SQLX

config {
  type: "table",
  schema: "my_schema",
}

pre_operations {
  ${reservations.reservation_setter(ctx)}
}

SELECT * FROM source_table

JS

publish('my_table', {
    type: 'table',
    schema: 'my_schema',
}).preOps(ctx => `
${reservations.reservation_setter(ctx)}
`).query(ctx => `
SELECT * FROM source_table
`);

Reservations configuration

The reservation configuration object defines how actions are assigned to reservations:

const RESERVATION_CONFIG = [
  {
    tag: 'standard_reservation',
    reservation: 'projects/{project}/locations/{location}/reservations/{name}',
    actions: [
      'project.dataset.table'
    ]
  },
  {
    tag: 'on_demand',
    reservation: 'none',
    actions: [
      'project.action_name'
    ]
  }
]

Masthead provides the RESERVATION_CONFIG object with intelligent action categorization:

• High-Slots: Large data volume operations requiring consistent compute

• Low-Slots: Standard ETL processes with flexible timing requirements

• On-Demand: Computation-intensive operations with variable resource needs

You can review it regularly manually or develop an automated process using programmatic insights access.

The recommended pipeline model is estimated based on:

• Usage Patterns: Historical query performance and frequency

• Cost Analysis: BigQuery slot usage and billing data

• Resource Availability: Current reservation capacity and utilization

Monitoring and Optimization

To ensure optimal performance:

• Monitor performance: Track BigQuery job execution and costs in real-time

• Review categorization: Regularly check action assignments in Compute Cost Insights

• Adjust allocations: Fine-tune reservation assignments based on usage patterns and cost analysis

Troubleshooting

If you encounter issues with reservation assignment:

1. Check compilation: Verify Dataform compilation completes without errors

2. Validate configuration: Ensure action names in RESERVATION_CONFIG match exactly

3. Monitor assignments: Review BigQuery job details to confirm reservation usage

4. Get support: Contact Masthead support for configuration optimization assistance