Cloud Data Pipeline

Minimal GCP pipeline: CSV files in Cloud Storage, queried via a BigQuery external table (schema autodetected from headers). Infrastructure is defined with Terraform.

Prerequisites

• Terraform >= 1.0
• Google Cloud SDK (gcloud auth application-default login)
• A GCP project with billing enabled

Quick start

cd infra/terraform
cp terraform.tfvars.example terraform.tfvars   # set project, region, bucket_name, dataset_id, table_id
terraform init
terraform apply

Terraform uploads a sample CSV to gs://<bucket>/data/input.csv. Add your own CSVs at the bucket root (*.csv) or under data/ — BigQuery reads both locations.

Query in BigQuery (use terraform output for dataset and table IDs):

SELECT *, _FILE_NAME
FROM `<project>.<dataset>.<table>`
LIMIT 20;

Layout

data/sample/     Sample CSV (uploaded to GCS as data/input.csv)
infra/terraform/ GCS bucket, BigQuery dataset & external table
docs/            Optional notes (e.g. Data Studio)