Masthead Data
  • Introduction
    • What is Masthead Data?
    • Data Anomaly Detection
    • Pipeline and Model Observability
    • Data Quality Scans
    • Metadata management: Column-level lineage and Data Dictionary
  • GET STARTED
    • Quickstart
  • COST INSIGHTS
    • Pipeline Costs
    • Storage Costs
  • INTEGRATIONS
    • Slack
    • Looker
    • PagerDuty
    • Power BI
  • Resources
    • Release Notes
    • Support
Powered by GitBook
On this page
  1. GET STARTED

SaaS Manual Resource Creation (Google Cloud + BigQuery)

Last updated 6 days ago

To enable Masthead Solution in your BigQuery data warehouse, it is required to create the next resources in your Google Cloud.

A user running an installation should have Owner permissions for the Google Cloud project.

Or if the user has an Editor role, the next additional permissions are required:

  • Logging Admin

  • Project IAM Admin

  • Role Administrator

1

Select Google Cloud project

Choose a project where BigQuery datasets are located and you would like Masthead Solution to monitor them. Use this project to create resources further.

Copy the project ID, you will need it further during the the Manual deployment option.

2

Create Pub/Sub resources

Under this selected project, navigate to Pub/Sub.

  1. Click Create Topic. Set topic id masthead-topic. (unmark checkbox Add a default subscription)

  1. Navigate into the newly created. Click Create Subscription

  2. Set subscription id: masthead-agent-subscription

  3. Set Cloud Pub/Sub topic ID: masthead-topic, then scroll down a little ...

  4. Set Acknowledgement deadline: 60 seconds

  5. Click Create and we are all done with Pub/Sub topic and Subscription.

3

Create Logs Router

Navigate to Logs Router from the Logging menu. Click Create Sink button in the upper right corner and fill in the required fields:

  1. Name: masthead-agent-sink

  1. Set Sink Destination: Cloud Pub/Sub topic -> choose from the dropdown newly created Pub/Sub topic masthead-topic

  1. Choose logs to include in the sink. Copy the text below and paste it into the filter.

protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR "google.cloud.bigquery.v2.JobService.InsertJob" 
OR "google.cloud.bigquery.v2.TableService.InsertTable" 
OR "google.cloud.bigquery.v2.JobService.Query" 
resource.type ="bigquery_table" OR resource.type ="bigquery_dataset" OR resource.type ="bigquery_project"

  1. Hit Create Sink to complete creation

4

Grant Masthead Service Account roles

Grant masthead-data@masthead-prod.iam.gserviceaccount.com next roles:

  • BigQuery Metadata Viewer

  • BigQuery Resource Viewer

  • Pub/Sub Subscriber

5

Grant Masthead Service Account to quickly onboard from retrospective data

The Masthead platform can gain insights in a few hours by parsing retrospective logs and creating a data model around them. To do so, please grant Service Account retro-data@masthead-prod.iam.gserviceaccount.com a Private Logs Viewer role.

  1. Navigate to IAM & Admin -> IAM

  2. Click on GRANT ACCESS button on the top left of the screen

  3. Specify retro-data@masthead-prod.iam.gserviceaccount.com in the New principals field

  4. Click on Select a role and type Private Logs Viewer. Select found Private Logs Viewer role.

  5. Click SAVE

This will enable Masthead Agent to look up only recently produced events in the Google Cloud that corresponds to the filter defined in the Cloud Logs Sink.