Integrate using Terraform

To connect Masthead with your BigQuery data warehouse, you'll need to create a few integration resources in your Google Cloud project.

An account running an installation should have Owner role, or alternatively, the following roles are required in the Google Cloud project:

• Logging Admin(roles/logging.admin)

• Pub/Sub Admin (roles/pubsub.admin)

• Project IAM Admin (roles/resourcemanager.projectIamAdmin)

Deploy the resources

1

Add Masthead agent module to your IaC project

Terraform

module "masthead-agent" {
  source  = "masthead-data/masthead-agent/google"

  project_id = "YOUR_PROJECT_ID"

  # Enable modules for the used services
  enable_modules = {
    bigquery      = true
    dataform      = true
    dataplex      = true
    analytics_hub = true
  }
}

More details about the module on Terraform Registry.

Pulumi

2

Apply the new resource changes

Terraform

terraform init
terraform plan
terraform apply

Pulumi

3

Continue integration on the UI

Go back to Deployment page, click Verify custom deployment, enter the project ID and click Check permissions and connect.

4

Configuration Completed

Masthead installation will create your workspace and open your dashboard once setup is completed.

5

Workspace is ready

You'll receive an email notification with the link to the workspace. We'll notify you once all the metadata is collected and the insights are ready.

All done! Thank you for completing installation!

Export retrospective logs

Masthead allows you to get observability insights within a few hours after project integration.

By default Masthead uses the Private Logs Viewer role to export 30 days of retrospective logs automatically. If you set enable_privatelogviewer_role = false in Terraform module, quick insights can be enabled by exporting the retrospective logs to Masthead for analysis:

1

Masthead will provide Storage Object Creator permission to your service account to write the exported retrospective logs into Masthead's Cloud Storage bucket.

The account must have the following permissions in your project:

• Logging Admin(roles/logging.admin)

• Logs Configuration Writer(roles/logging.configWriter)

2

Copy the code, update the variables and run the command to start the export operation:

• PROJECT_ID - your project ID

• YYYY-MM-DD - export start date, 30 days ago

gcloud logging copy _Default storage.googleapis.com/masthead_retro_logs_{PROJECT_ID} \
--location=global \
--log-filter='protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob" OR protoPayload.methodName="google.cloud.bigquery.v2.TableService.InsertTable" OR protoPayload.methodName="google.cloud.bigquery.v2.JobService.Query" resource.type="bigquery_table" OR resource.type="bigquery_dataset" OR resource.type="bigquery_project" timestamp > "{YYYY-MM-DD}T00:00:00.0Z"'

How to batch and route logs retrospectively.

Check the progress of the started operation:

gcloud logging operations describe OPERATION_ID \
--location=global --project=PROJECT_ID

3

Inform Masthead team

Your Masthead account will use the exported data to generate the compute and cost insights without the need to wait for weeks.

VPC-SC Configuration

If your Google Cloud project is within a VPC service perimeter, you must configure ingress and egress policies to ensure that the necessary resources are accessible to Masthead Data.

Policies configuration suggestion

restricted_services = [
  "bigquery.googleapis.com",
  "pubsub.googleapis.com",
  "logging.googleapis.com",
  "dataplex.googleapis.com",
  "analyticshub.googleapis.com"
]

ingress_policies {
  ingress_from {
    identities = [
      "serviceAccount:[email protected]",
      "serviceAccount:[email protected]",
      "serviceAccount:[email protected]",
      "serviceAccount:[email protected]"
    ]
    sources {
      resource = "projects/431544431936" # masthead-prod
    }
  }

  ingress_to {
    resources = ["*"]
    roles = [
      "roles/bigquery.metadataViewer",
      "roles/bigquery.resourceViewer",
      "roles/logging.privateLogViewer",
      "roles/pubsub.subscriber",
      "roles/analyticshub.viewer",
      "projects/YOUR_PROJECT_ID/roles/analyticsHubSubscriptionViewer",

    ]
  }
}

egress_policies {
  egress_to {
    resources = ["*"]
    roles = [
      "roles/bigquery.metadataViewer",
      "roles/bigquery.resourceViewer"
    ]
  }

  egress_from {
    identities = [
      "serviceAccount:[email protected]",
    ]
    sources {
      resource = "projects/431544431936" # masthead-prod
    }
  }
}