SaaS Manual Resource Creation (Google Cloud + BigQuery)

To enable Masthead Solution in your BigQuery data warehouse, it is required to create the next resources in your Google Cloud.

1. Select Google project

Choose a project where BigQuery datasets are located and you would like Masthead Solution to monitor them. Use this project to create resources further.

Copy the project ID; you would need it further during the installation while selecting the Manual deployment option.

2. Create Pub/Sub resources

Under this selected project, navigate to Pub/Sub

  1. Click Create Topic. Set topic id masthead-topic. (unmark checkbox Add a default subscription)

  1. Navigate into the newly created. Click Create Subscription

  1. Set subscription id: masthead-agent-subscription

  2. Set Cloud Pub/Sub topic ID: masthead-topic then scroll down a little ...

  3. Set Acknowledgement deadline: 60 seconds

  1. Click Create and we are all done with Pub/Sub topic and Subscription.

3. Create Logs Router

Navigate to Logs Router from the Logging menu. Click Create Sink button in the upper right corner and fill in the required fields:

  1. Name: masthead-agent-sink

  1. Set Sink Destination: Cloud Pub/Sub topic -> choose from the dropdown newly created Pub/Sub topic masthead-topic

  1. Choose logs to include in the sink. Copy the text below and paste it into the filter.

protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR "google.cloud.bigquery.v2.JobService.InsertJob" 
OR "google.cloud.bigquery.v2.TableService.InsertTable" 
OR "google.cloud.bigquery.v2.JobService.Query" 
resource.type ="bigquery_table" OR resource.type ="bigquery_dataset" OR resource.type ="bigquery_project"

  1. Hit Create Sink to complete creation

4. Create a custom role

To get metadata of BigQuery schema and its tables and views, navigate to IAM & Admin -> Roles menu.

  1. Click Create Role

  2. Set title: masthead_bq_schema_reader

  3. Description: Masthead BigQuery assets metadata reader

  4. Set Role Launch Satge: General Availability

  1. Click Add Permissions and add next permissions:

bigquery.datasets.get
bigquery.tables.get
bigquery.tables.list

  1. Hit Create to complete creation and click

5. Grant Masthead Service Account roles

Grant masthead-data@masthead-prod.iam.gserviceaccount.com next roles

masthead_bq_schema_reader
Pub/Sub Subscriber

6. Grant Masthead Service Account to quickly onboard from retrospective data

The Masthead platform can gain insights in a few hours by parsing retrospective logs and creating a data model around them. To do so, please grant Service Account retro-data@masthead-prod.iam.gserviceaccount.com a Private Logs Viewer role.

  • Navigate to IAM & Admin -> IAM

  • Click on "GRANT ACCESS" button on the top left of the screen

  • Specify retro-data@masthead-prod.iam.gserviceaccount.com in the New principals field

  • Click on "Select a role" and type Private Logs Viewer. Select found Private Logs Viewer role.

This will enable Masthead Agent to look up only recently produced events in the Google Cloud that correspond to the filter below.

protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" 
OR "google.cloud.bigquery.v2.JobService.InsertJob" 
OR "google.cloud.bigquery.v2.TableService.InsertTable" 
OR "google.cloud.bigquery.v2.JobService.Query"
resource.type ="bigquery_table" OR resource.type ="bigquery_dataset" OR resource.type ="bigquery_project"

Last updated