Masthead Data
  • Introduction
    • What is Masthead Data?
    • Data Anomaly Detection
    • Pipeline and Model Observability
    • Data Quality Scans
    • Metadata management: Column-level lineage and Data Dictionary
  • GET STARTED
    • Quickstart
  • COST INSIGHTS
    • Pipeline Costs
    • Storage Costs
  • INTEGRATIONS
    • Slack
    • Looker
    • PagerDuty
    • Power BI
  • Resources
    • Release Notes
    • Support
Powered by GitBook
On this page
  1. COST INSIGHTS

Pipeline Costs

Last updated 1 month ago

Masthead's Cost Recommendations identify actionable optimizations to improve cloud spend and ROI for data products.

The costs are estimates based on past usage. The costs saving estimates are intended to help to compare and prioritize cost saving recommendations. Credits or discounts aren't included neither in costs nor savings calculated.

Pipeline Compute Cost Insights

Masthead analyzes BigQuery compute resource consumption and provides recommendations based on the pipelines and resource usage patterns. These recommendations help leverage the most cost-effective BigQuery compute billing models (e.g. on-demand or capacity pay-as-you-go).

Compute Billing Models: BigQuery offers different compute billing models that impact costs. Masthead analyzes usage to determine the optimal model. More details in Google Cloud documentation.

Masthead sees switching between On-Demand and BigQuery Editions Standard (pay-as-you-go) as the most common and biggest cost optimization opportunity among our customers. We currently focus on this type of analysis and recommendation in the product. Please contact us to discuss further opportunities regarding compute model alternatives.

Check recommendations on the dashboard.

Alternative Pricing for a Project

Masthead analyzes and aggregates all compute resources used within a project to recommend the most cost-effective compute pricing model. We consider overall project usage patterns and potential savings by switching between On-Demand and Editions Standard models.

Check overall compute usage and see project level recommendation on Costs Insights page.

Pipeline-Specific Compute Cost Optimization

Masthead provides a granular view of compute costs, grouped by pipeline. This allows to optimize costs at the pipeline level by considering moving them under BigQuery Standard Editions model based on slot-consumption. Filter by labels to analyze specific workloads.

Recommendation: Review pipeline-specific recommendations on the Costs Insights page and consider adjusting pipeline compute configurations.

Read more about combining different compute billing models.

Check pipeline cost recommendations on Costs Insights page:

Optimize Compute Costs for Dead-end Tables

Masthead identifies tables with no downstream consumption, tagging them as "Dead-End Tables". This allows to analyze related pipelines and reclaim resources.

Recommendation:

  1. Review compute usage associated with dead-end tables.

  2. Verify the tables have no consumers downstream

  3. Adjust or disable related pipelines to reduce unnecessary compute spend.

Check recommendations for pipelines with dead-end tables on Costs Insights page:

Costs recommendations on Dashboard page
Alternative model plan for pipelines
Pipelines with dead-end tables