Skip to content

README.md

Latest commit

 

History

History
161 lines (99 loc) · 3.68 KB

README.md

File metadata and controls

161 lines (99 loc) · 3.68 KB

akoflow-driver-gcs-uploader

A container-first Google Cloud Storage (GCS) uploader designed to run entirely inside a Docker environment.

This project provides:

  • A Python CLI tool to recursively upload files to GCS
  • Full support for endpoint override (local emulator or real GCP)
  • Docker + Docker Compose setup
  • Makefile automation
  • Zero dependency on host Python or local SDKs

The storage backend can be: - A local Fake GCS Server (for development) - Real Google Cloud Storage (production)

Project Structure

akoflow-driver-gcs-uploader/
│
├── docker/
│   └── local/
│       ├── .env
│       ├── docker-compose.yml
│       └── Dockerfile
├── upload_and_move.py
├── Dockerfile
├── docker-compose.yaml
├── Makefile
├── requirements.txt
└── README.md

Requirements

  • Docker
  • Docker Compose

No local Python installation is required.

How It Works

The upload_and_move.py script:

  • Recursively scans a directory (relative to CWD)
  • Excludes configurable directory names
  • Uploads files to a GCS bucket
  • Supports bucket creation
  • Supports dry-run mode
  • Supports custom storage endpoint override

All execution happens inside the app container.

Environment Configuration

The project uses an .env file via Docker Compose.

Example:

GOOGLE_CLOUD_PROJECT=local-project

For real GCP usage:

GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
GOOGLE_CLOUD_PROJECT=your-project-id

Storage Endpoint Configuration

The system supports endpoint override:

STORAGE_ENDPOINT=http://gcs-emulator:4443

If STORAGE_ENDPOINT is empty, the script connects to real Google Cloud Storage.

Build and Run

Build images

make build

Start environment

make up

Stop environment

make down

Script CLI Options

--bucket               Destination bucket (required)
--prefix               Object prefix
--source-dir           Directory to scan (default: .)
--project              Project override
--credentials          Service account JSON path
--storage-endpoint     Storage endpoint override
--exclude-dir          Directory exclusion (repeatable)
--create-bucket        Create bucket if missing
--dry-run              Preview upload only

Common Commands

Run uploader:

make run GCS_BUCKET=my-bucket

Dry run:

make run-dry GCS_BUCKET=my-bucket

Create bucket:

make create-bucket GCS_BUCKET=my-bucket

List buckets:

make list-buckets

List objects:

make list-objects GCS_BUCKET=my-bucket

Clean bucket:

make clean-bucket GCS_BUCKET=my-bucket

Production Usage

To use real Google Cloud Storage:

  1. Remove or unset STORAGE_ENDPOINT
  2. Provide valid credentials via GOOGLE_APPLICATION_CREDENTIALS
  3. Set the correct GOOGLE_CLOUD_PROJECT

Example:

make run GCS_BUCKET=my-bucket STORAGE_ENDPOINT=

License

MIT