Environment sizing

To estimate the scale of your infrastructure, gather two key metrics from your Google Cloud environment:

• the total number of tables
• and the expected volume of BigQuery logs that Masthead processes.

1. Total number of tables

Use Cloud Asset Inventory to easily count BigQuery tables across multiple regions.

Use the Google Cloud console

1. Open Cloud Asset Inventory in the Google Cloud console.
2. Apply the filter Resource type = bigquery.Table to view the total count.

Use the gcloud command-line tool

Run the following command, replacing <YOUR-PROJECT-ID>. For multiple projects, either sum the results or change --scope to your folder/organization level:

gcloud asset search-all-resources \
    --scope="projects/<YOUR-PROJECT-ID>" \
    --asset-types="bigquery.googleapis.com/Table" \
    --format="value(name)" | wc -l

2. Total number of BigQuery logs

Masthead uses Google Cloud Logging to process metadata and analyze your BigQuery environment. To estimate the expected log volume:

1. Go to the Logs Explorer in the Google Cloud console.

2. Refine the logs scope to ensure that you select the Google Cloud projects to integrate with Masthead.

3. Set the time range in the top right corner to Last 30 days.

4. In the query builder, apply the following log filter:

   (
     protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR
     protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.ReadRows" OR
     protoPayload.methodName="google.cloud.bigquery.v2.JobService.GetQueryResults" OR
     protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob" OR
     protoPayload.methodName="google.cloud.bigquery.v2.JobService.Query" OR
     protoPayload.methodName="google.cloud.bigquery.v2.TableDataService.List" OR
     protoPayload.methodName="google.cloud.bigquery.v2.TableService.InsertTable"
   ) AND (
     resource.type="bigquery_table" OR
     resource.type="bigquery_dataset" OR
     resource.type="bigquery_project"
   )

5. Click Run query.

6. Under the Histogram panel and preceding the log list there is X results - a total number of matched log entries over the selected period.

7. Note the total count of matched logs and share this information with the Masthead team.