Masthead Data
  • Introduction
    • What is Masthead Data?
    • Data Anomaly Detection
    • Pipeline and Model Observability
    • Data Quality Scans
    • Metadata management: Column-level lineage and Data Dictionary
  • GET STARTED
    • Quickstart
  • Data Products
    • Data Products
    • Domains
  • COST INSIGHTS
    • Pipeline Costs
    • Storage Costs
  • INTEGRATIONS
    • Slack
    • Looker
    • PagerDuty
    • Power BI
  • Resources
    • Release Notes
    • Support
Powered by GitBook
On this page
  1. COST INSIGHTS

Pipeline Costs

Last updated 14 days ago

Masthead's Cost Recommendations identify actionable optimizations to improve cloud spend and ROI for data products.

The costs are estimates based on past usage. The costs saving estimates are intended to help to compare and prioritize cost saving recommendations. Credits or discounts aren't included neither in costs nor savings calculated.

Pipeline Compute Cost Insights

Masthead analyzes BigQuery compute resource consumption and provides recommendations based on the pipelines and resource usage patterns. These recommendations help leverage the most cost-effective (e.g. on-demand or capacity pay-as-you-go).

Compute Billing Models: BigQuery offers different compute billing models that impact costs. Masthead analyzes usage to determine the optimal model. More details in .

Masthead sees switching between On-Demand and BigQuery Editions Standard (pay-as-you-go) as the most common and biggest cost optimization opportunity among our customers. We currently focus on this type of analysis and recommendation in the product. Please to discuss further opportunities regarding compute model alternatives.

Check recommendations .

Alternative Pricing for a Project

Masthead analyzes and aggregates all compute resources used within a project to recommend the most cost-effective compute pricing model. We consider overall project usage patterns and potential savings by switching between On-Demand and Editions Standard models.

Pipeline-Specific Compute Cost Optimization

Masthead provides a granular view of compute costs, grouped by pipeline. This allows to optimize costs at the pipeline level by considering moving them under BigQuery Standard Editions model based on slot-consumption. Filter by labels to analyze specific workloads.

Recommendation: Review pipeline-specific recommendations on the Costs Insights page and consider adjusting pipeline compute configurations.

Optimize Compute Costs for Dead-end Tables

Masthead identifies tables with no downstream consumption, tagging them as "Dead-End Tables". This allows to analyze related pipelines and reclaim resources.

Recommendation:

  1. Review compute usage associated with dead-end tables.

  2. Verify the tables have no consumers downstream

  3. Adjust or disable related pipelines to reduce unnecessary compute spend.

Check overall compute usage and see project level recommendation .

Read more about .

Check pipeline cost recommendations on page:

Check recommendations for pipelines with dead-end tables on :

on Costs Insights page
combining different compute billing models
Pipeline costs
Costs Insights page
BigQuery compute billing models
Google Cloud documentation
contact us
on the dashboard
Costs recommendations on Dashboard page
Alternative model plan for pipelines
Pipelines with dead-end tables