Integrate using Terraform

To connect Masthead with your BigQuery data warehouse, you'll need to create a few integration resources in your Google Cloud project.

Google account running resources creation should have Owner permissions for the Google Cloud project. Or the following roles are required:

Deploy the resources

1

Add Masthead agent module to your Terraform project

module "masthead-agent" {
  source  = "masthead-data/masthead-agent/google"
  
  project_id = "YOUR_PROJECT_ID"
  
  # Enable modules for the used services
  enable_modules = {
    bigquery      = true
    dataform      = true
    dataplex      = true
    analytics_hub = true
  }
}

More details about the module on Terraform Registry.

2

Apply the new resource changes

Update your dependencies:

terraform init

Plan and verify the resource changes:

terraform plan

Apply the changes to your cloud project:

terraform apply
3

Continue integration on the UI

Go back to Deployment page, click Manual Deployment and enter the project ID.

Click Check permissions and connect.

You will be forwarded to the overview dashboard for your newly integrated project.

We'll notify you once all the metadata is collected and the insights are ready.

Export retrospective logs

Get observability insights within a few hours after project integration. This can be enabled by exporting the retrospective logs to Masthead for analysis.

1

Masthead will provide Storage Object Creator permission to your service account to write the exported retrospective logs into Masthead's Cloud Storage bucket.

The account must have the following permissions in your project:

2

Copy the code, update the variables and run the command to start the export operation:

  • PROJECT_ID - your project ID

  • YYYY-MM-DD - export start date

gcloud logging copy _Default storage.googleapis.com/masthead_retro_logs_{PROJECT_ID} \
--location=global \
--log-filter='protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob" OR protoPayload.methodName="google.cloud.bigquery.v2.TableService.InsertTable" OR protoPayload.methodName="google.cloud.bigquery.v2.JobService.Query" resource.type="bigquery_table" OR resource.type="bigquery_dataset" OR resource.type="bigquery_project" timestamp > "{YYYY-MM-DD}T00:00:00.0Z"'

How to batch and route logs retrospectively

Check the progress of the started operation:

gcloud logging operations describe OPERATION_ID \
--location=global --project=PROJECT_ID
3

Inform Masthead team

Your Masthead project will use the exported data to generate the compute and cost insights without the need to wait for weeks.

Last updated