SaaS Manual Resource Creation (Google Cloud + BigQuery)

To enable Masthead Solution in your BigQuery data warehouse, it is required to create the next resources in your Google Cloud.

An account running an installation should have Owner permissions for the Google Cloud project.

OR if it has an Editor role, the next additional permissions are required:

• Logging Admin

• Project IAM Admin

Manage resources with Terraform

1

Add Masthead agent module to your Terraform project

module "masthead-agent_bigquery" {
  source  = "masthead-data/masthead-agent/google//modules/bigquery"
  project_id = "PROJECT_ID"
}

More details about the module on Terraform Registry.

2

Apply the new resource changes

Update your dependencies:

terraform init

Plan and verify the resource changes:

terraform plan

Apply the changes to your cloud project:

terraform apply

3

Continue integration on the UI

Go back to Deployment page, click Manual Deployment and enter the project ID again.

Then click Check permissions and connect.

You will be forwarded to the overview dashboard for your newly integrated project.

Manual resource creation

1

Select Google Cloud project

Choose a project where BigQuery datasets are located and you would like Masthead Solution to monitor them. Use this project to create resources further.

Copy the project ID, you will need it further during the the Manual deployment option.

2

Create Pub/Sub resources

Under this selected project, navigate to Pub/Sub.

1. Click Create Topic. Set topic id masthead-topic. (unmark checkbox Add a default subscription)

1. Navigate into the newly created. Click Create Subscription

   
2. Set subscription id: masthead-agent-subscription

3. Set Cloud Pub/Sub topic ID: masthead-topic, then scroll down a little ...

4. Set Acknowledgement deadline: 60 seconds

   
5. Click Create and we are all done with Pub/Sub topic and Subscription.

3

Create Logs Router

Navigate to Logs Router from the Logging menu. Click Create Sink button in the upper right corner and fill in the required fields:

1. Name: masthead-agent-sink

1. Set Sink Destination: Cloud Pub/Sub topic -> choose from the dropdown newly created Pub/Sub topic masthead-topic

1. Choose logs to include in the sink. Copy the text below and paste it into the filter.

protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR "google.cloud.bigquery.v2.JobService.InsertJob" 
OR "google.cloud.bigquery.v2.TableService.InsertTable" 
OR "google.cloud.bigquery.v2.JobService.Query" 
resource.type ="bigquery_table" OR resource.type ="bigquery_dataset" OR resource.type ="bigquery_project"

1. Hit Create Sink to complete creation

4

Grant Masthead Service Account roles

Grant [e-mail] next roles:

• BigQuery Metadata Viewer

• BigQuery Resource Viewer

• Pub/Sub Subscriber

5

Grant Masthead Service Account to quickly onboard from retrospective data

The Masthead platform can gain insights in a few hours by parsing retrospective logs and creating a data model around them. To do so, please grant Service Account [e-mail] a Private Logs Viewer role.

1. Navigate to IAM & Admin -> IAM

2. Click on GRANT ACCESS button on the top left of the screen

3. Specify [e-mail] in the New principals field

4. Click on Select a role and type Private Logs Viewer. Select found Private Logs Viewer role.

5. Click SAVE

This will enable Masthead Agent to look up only recently produced events in the Google Cloud that corresponds to the filter defined in the Cloud Logs Sink.

6

Continue integration on the UI

Go back to Deployment page, click Manual Deployment and enter the project ID again.

Then click Check permissions and connect.

You will be forwarded to the overview dashboard for your newly integrated project.