SaaS Manual Resource Creation (Google Cloud + BigQuery)

Add Masthead agent module to your Terraform project

module "masthead-agent" {
  source  = "masthead-data/masthead-agent/google"
  version = "~> 0.2.0"
  
  project_id = "YOUR_PROJECT_ID"
  
  # Enable only specific modules
  enable_modules = {
    bigquery      = true
    dataform      = false
    dataplex      = true
    analytics_hub = true
  }
}

More details about the module on Terraform Registry.

Apply the new resource changes

Update your dependencies:

terraform init

Plan and verify the resource changes:

terraform plan

Apply the changes to your cloud project:

terraform apply

Continue integration on the UI

Go back to Deployment page, click Manual Deployment and enter the project ID again.

Then click Check permissions and connect.

Select Google Cloud project

Choose a project where BigQuery datasets are located and you would like Masthead Solution to monitor them. Use this project to create resources further.

Copy the project ID, you will need it further during the the Manual deployment option.

Create Pub/Sub resources

Under this selected project, navigate to Pub/Sub.

1. Click Create Topic. Set topic id masthead-topic. (unmark checkbox Add a default subscription)

1. Navigate into the newly created. Click Create Subscription

   
2. Set subscription id: masthead-agent-subscription

3. Set Cloud Pub/Sub topic ID: masthead-topic, then scroll down a little ...

4. Set Acknowledgement deadline: 60 seconds

   
5. Click Create and we are all done with Pub/Sub topic and Subscription.

Create Logs Router

Navigate to Logs Router from the Logging menu. Click Create Sink button in the upper right corner and fill in the required fields:

1. Name: masthead-agent-sink

1. Set Sink Destination: Cloud Pub/Sub topic -> choose from the dropdown newly created Pub/Sub topic masthead-topic

1. Choose logs to include in the sink. Copy the text below and paste it into the filter.

protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR "google.cloud.bigquery.v2.JobService.InsertJob" 
OR "google.cloud.bigquery.v2.TableService.InsertTable" 
OR "google.cloud.bigquery.v2.JobService.Query" 
resource.type ="bigquery_table" OR resource.type ="bigquery_dataset" OR resource.type ="bigquery_project"

1. Hit Create Sink to complete creation

Grant Masthead Service Account roles

Grant [email protected] next roles:

• BigQuery Metadata Viewer

• BigQuery Resource Viewer

• Pub/Sub Subscriber

Grant Masthead Service Account to quickly onboard from retrospective data

The Masthead platform can gain insights in a few hours by parsing retrospective logs and creating a data model around them. To do so, please grant Service Account [email protected] a Private Logs Viewer role.

1. Navigate to IAM & Admin -> IAM

2. Click on GRANT ACCESS button on the top left of the screen

3. Specify [email protected] in the New principals field

4. Click on Select a role and type Private Logs Viewer. Select found Private Logs Viewer role.

5. Click SAVE

This will enable Masthead Agent to look up only recently produced events in the Google Cloud that corresponds to the filter defined in the Cloud Logs Sink.

Continue integration on the UI

Go back to Deployment page, click Manual Deployment and enter the project ID again.

Then click Check permissions and connect.