Integrate using Terraform
To connect Masthead with your BigQuery data warehouse, you'll need to create a few integration resources in your Google Cloud project.
Deploy the resources
Add Masthead agent module to your Terraform project
module "masthead-agent" {
source = "masthead-data/masthead-agent/google"
project_id = "YOUR_PROJECT_ID"
# Enable modules for the used services
enable_modules = {
bigquery = true
dataform = true
dataplex = true
analytics_hub = true
}
}
More details about the module on Terraform Registry.
Continue integration on the UI
Go back to Deployment page, click Verify custom deployment, enter the project ID and click Check permissions and connect.
Export retrospective logs
Masthead allows you to get observability insights within a few hours after project integration.
By default Masthead uses the Private Logs Viewer role to export 30 days of retrospective logs automatically. If you set enable_privatelogviewer_role = false
in Terraform module, quick insights can be enabled by exporting the retrospective logs to Masthead for analysis:
Masthead will provide Storage Object Creator
permission to your service account to write the exported retrospective logs into Masthead's Cloud Storage bucket.
The account must have the following permissions in your project:
Logging Admin (
roles/logging.admin
)Logs Configuration Writer (
roles/logging.configWriter
)
Copy the code, update the variables and run the command to start the export operation:
PROJECT_ID - your project ID
YYYY-MM-DD - export start date, 30 days ago
gcloud logging copy _Default storage.googleapis.com/masthead_retro_logs_{PROJECT_ID} \
--location=global \
--log-filter='protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob" OR protoPayload.methodName="google.cloud.bigquery.v2.TableService.InsertTable" OR protoPayload.methodName="google.cloud.bigquery.v2.JobService.Query" resource.type="bigquery_table" OR resource.type="bigquery_dataset" OR resource.type="bigquery_project" timestamp > "{YYYY-MM-DD}T00:00:00.0Z"'
Last updated