SaaS Manual Resource Creation (Google Cloud + BigQuery)

To enable Masthead Solution in your BigQuery data warehouse, it is required to create next resources in your Google Cloud.

1. Select Google project

Choose a project where BigQuery datasets are located and you would like Masthead Solution to monitor them. Use this project to create resources further.

Copy project ID, you would need it further during the installation while selecting Manual deployment option.

2. Create Pub/Sub resources

Under this selected project

  • Click Create Topic. Set topic id masthead-topic. (unmark checkbox Add a default subscription)

  • Navigate into the newly created. Click Create Subscription

  • Set subscription id: masthead-agent-subscription

  • Set Acknowledgement deadline: 60 seconds

3. Create Logs Router

Navigate to Logs Router from the Logging menu. Click Create Sink button and fill-in required fields:

  • Name: masthead-agent-sink

  • Set Sink Destination: Cloud Pub/Sub topic -> choose from the dropdown newly create Pub/Sub topic masthead-topic

  • Choose logs to include in sink. Copy below text and paste in filter.

protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR "google.cloud.bigquery.v2.JobService.InsertJob" 
OR "google.cloud.bigquery.v2.TableService.InsertTable" 
OR "google.cloud.bigquery.v2.JobService.Query" 
resource.type ="bigquery_table" OR resource.type ="bigquery_dataset" OR resource.type ="bigquery_project"

  • Complete creation and click Create Sink

4. Create a custom role

To get metadata of BigQuery schema and its tables and views, navigate to IAM & Admin -> Roles menu

  • Click Create Role

  • Set title: masthead_bq_schema_reader

  • Description: Masthead BigQuery assets metadata reader

  • Click Add Permissions and add next permissions:

bigquery.datasets.get
bigquery.tables.get
bigquery.tables.list

  • Complete creation and click Create

5. Grant Masthead Service Account roles

Grant masthead-data@masthead-prod.iam.gserviceaccount.com next roles

masthead_bq_schema_reader
Pub/Sub Subscriber

6. Grant Masthead Service Account to quickly onboard from retrospective data

Retrospective Log parserIf you would like to have data monitoring insights from day 1, you can enable Masthead to look up retrospective events. To do so, please grant Service Account retro-data@masthead-prod.iam.gserviceaccount.com a Private Logs Viewer role.

  • Navigate to IAM & Admin -> IAM

  • Click on "GRANT ACCESS" button on the top left of the screen

  • Specify retro-data@masthead-prod.iam.gserviceaccount.com in the New principals field

  • Click on "Select a role" and type Private Logs Viewer. Select found Private Logs Viewer role.

This will enable Masthead Agent to look up only recently produced events in the Google Cloud that correspond to the filter below. The look-up period is 30 days.

Here is a filter that is used for collection retrospective events:

protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" 
OR "google.cloud.bigquery.v2.JobService.InsertJob" 
OR "google.cloud.bigquery.v2.TableService.InsertTable" 
OR "google.cloud.bigquery.v2.JobService.Query"
resource.type ="bigquery_table" OR resource.type ="bigquery_dataset" OR resource.type ="bigquery_project"

Last updated