BigQuery Buildkite Plugin
This Buildkite plugin can be used to deploy tables/views schemas to BigQuery
A Buildkite plugin for deploying tables/views schemas to BigQuery.
Using the plugin
If the version number is not provided then the most recent version of the plugin will be used. Do not use version number as master
or any branch names.
Simple
steps:
- plugins:
- wayfair-incubator/bigquery#v1.2.2:
gcp_project: gcp-us-project
dataset_schema_directory: schemas/gcp-us-project/dataset
Configuration
Required
gcp_project
(required, string)
The full name of the GCP project you want to deploy.
Example: gcp-us-project
dataset_schema_directory
(required, string)
The directory in your repository where are you storing the schemas for your tables and views.
Example: gcp-us-project/dataset_name
prod_build_branch
(optional, string)
Default branch for the repo.
Example: main
Default: master
execute_only_changed_files
(optional, boolean)
Whether to execute just the files that changed in a directory or not.
Example: true
Default: true
fail_pipeline_on_first_exception
(optional, boolean)
Whether to fail pipeline as soon as there is one failure.
Example: true
Default: true
Secret
This plugin expects GCP_SERVICE_ACCOUNT
is placed as environment variable. Make sure to store it securely!
env:
gcp_service_account: '{"email": ""}'
Example
Basic
The following pipeline will deploy all the schemas to the gcp-us-project
living under directory/project/
Schemas
This plugin uses GBQ to deploy to Google BigQuery. GBQ now supports specifying partitions with the schema as well.
To leverage this you need to nest your JSON table schema in a dictionary. An example for the same is given below. Library supports Time and Range based partitioning along with Clustering.
All the configuration options can be found here.
{
"partition": {
"type": "range",
"definition": {
"field": "ID",
"range": {
"start": 1,
"end": 100000,
"interval": 10
}
}
},
"clustering": [
"ID"
],
"schema": [
{
"name": "ID",
"type": "INTEGER",
"mode": "REQUIRED"
}
]
}
Contributing
See the Contributing Guide for additional information.
To execute tests locally (requires that docker
and docker-compose
are installed):
docker-compose run test
Credits
This plugin was originally written by Jash Parekh for Wayfair.