Environment sizing

To help us understand the scale of your infrastructure, we need to gather two key metrics from your Google Cloud environment:

• the total number of tables
• and the expected volume of BigQuery logs Masthead will process.

1. Total number of tables

Use Cloud Asset Inventory to easily count BigQuery tables across multiple regions.

Option A: Cloud Console

1. Open Cloud Asset Inventory in the Google Cloud Console.
2. Apply the filter Resource type = bigquery.Table to view the total count.

Option B: gcloud CLI

Run the following command, replacing <YOUR-PROJECT-ID>. For multiple projects, either sum the results or change --scope to your folder/organization level:

gcloud asset search-all-resources \
    --scope="projects/<YOUR-PROJECT-ID>" \
    --asset-types="bigquery.googleapis.com/Table" \
    --format="value(name)" | wc -l

2. Total number of BigQuery logs

Masthead uses Google Cloud Logging to process metadata and analyze your BigQuery environment. To estimate the expected log volume:

1. Go to the Logs Explorer in the Google Cloud Console.

2. Refine logs scope to ensure you have selected the Google Cloud project(s) that will be integrated with Masthead.

3. Set the time range in the top right corner to Last 30 days.

4. In the query builder, apply the following log filter:

   (
     protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows" OR
     protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryWrite.ReadRows" OR
     protoPayload.methodName="google.cloud.bigquery.v2.JobService.GetQueryResults" OR
     protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob" OR
     protoPayload.methodName="google.cloud.bigquery.v2.JobService.Query" OR
     protoPayload.methodName="google.cloud.bigquery.v2.TableDataService.List" OR
     protoPayload.methodName="google.cloud.bigquery.v2.TableService.InsertTable"
   ) AND (
     resource.type="bigquery_table" OR
     resource.type="bigquery_dataset" OR
     resource.type="bigquery_project"
   )

5. Click Run query.

6. Under the Histogram panel and above the log there is X results - a total number of matched log entries over the selected period.

7. Note the total count of matched logs and share it with us.