Skip to content

README.md

Latest commit

 

History

History
308 lines (217 loc) · 5.24 KB

README.md

File metadata and controls

308 lines (217 loc) · 5.24 KB

akoflow-driver-s3-uploader

A lightweight, containerized CLI utility for recursively uploading files to Amazon S3 (or LocalStack S3) with flexible configuration via CLI arguments and environment variables.

This project is designed for reproducible execution inside Docker, making it suitable for local development, CI pipelines, and technical demonstrations.

Overview

akoflow-driver-s3-uploader recursively scans a specified directory and uploads all eligible files to a destination S3 bucket.

Key Features

  • Recursive file discovery
  • Configurable source directory
  • Custom S3 prefix support
  • Environment variable fallback for credentials
  • LocalStack compatibility
  • Dockerized execution
  • Dry-run mode for safe previews
  • Deterministic, container-based runtime

Project Structure

akoflow-driver-s3-uploader/
│
├── docker/
│   └── local/
│       ├── .env
│       ├── docker-compose.yml
│       └── Dockerfile
│
├── venv/
│
├── .env
├── docker-compose.yml
├── Dockerfile
├── Makefile
├── README.md
├── requirements.txt
└── upload_and_move.py

How It Works

  1. The script resolves the source directory relative to the current working directory.
  2. It recursively scans all files.
  3. Excluded directories are ignored.
  4. S3 object keys are constructed using an optional prefix.
  5. Files are uploaded using boto3.
  6. A summary report is printed.

CLI Arguments

Required

  • --bucket
    Destination S3 bucket name.

Optional

  • --prefix
    S3 key prefix (folder path inside the bucket).
    Example: uploads/

  • --source-dir
    Directory to scan recursively.
    Resolved relative to the current working directory.
    Default: .

  • --endpoint-url
    Custom S3 endpoint (e.g., LocalStack).
    Example: http://localstack:4566

  • --region
    AWS region.

  • --access-key
    AWS access key ID.

  • --secret-key
    AWS secret access key.

  • --session-token
    AWS session token (temporary credentials).

  • --exclude-dir
    Directory name to exclude (repeatable).
    Example:

    --exclude-dir .git --exclude-dir node_modules

  • --dry-run
    Preview mode. Prints what would be uploaded without performing uploads.

Environment Variables

If CLI credentials are not provided, the script reads:

  • AWS_ENDPOINT_URL
  • AWS_DEFAULT_REGION
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_SESSION_TOKEN

If none are set, boto3 falls back to its default credential provider chain:

  1. Environment variables
  2. ~/.aws/credentials
  3. IAM role (if running inside AWS)

Default Exclusions

The following directories are excluded by default:

.git
.hg
.svn
__pycache__
.pytest_cache
.mypy_cache
.ruff_cache
.venv
venv
node_modules
.DS_Store

Additional exclusions can be added using --exclude-dir.

Running with Docker

Build

make build

Start Environment

make up

Initialize (LocalStack + bucket creation)

make init S3_BUCKET=my-bucket

Running the Uploader

Standard Execution

make run S3_BUCKET=my-bucket S3_PREFIX=uploads/ SOURCE_DIR=./data

Dry Run

make run-dry S3_BUCKET=my-bucket SOURCE_DIR=./data

Direct Docker Execution

docker exec -it aws-uploader python upload_and_move.py \
  --bucket my-bucket \
  --prefix uploads/ \
  --source-dir ./data

LocalStack Usage

Default endpoint:

http://localstack:4566

Create bucket:

make create-bucket S3_BUCKET=my-bucket

List objects:

make list-objects S3_BUCKET=my-bucket

Clean bucket:

make clean-bucket S3_BUCKET=my-bucket

Dry Run Mode Explained

When --dry-run is enabled:

  • Files are scanned
  • S3 keys are generated
  • Upload operations are printed
  • No files are uploaded
  • No network transfer occurs

This is useful for:

  • Verifying key structure
  • Validating exclusions
  • Preventing accidental uploads
  • Testing CI/CD pipelines safely

Error Handling

The script:

  • Uses boto3 retry configuration (standard mode)
  • Reports individual upload failures
  • Returns exit code 1 if any upload fails
  • Returns exit code 0 on full success
  • Returns exit code 2 if the source directory is invalid

Makefile Targets

Target Description
build Build Docker images
up Start environment
down Stop environment
restart Restart environment
logs Follow logs
shell Open app container shell
localstack Open LocalStack shell
create-bucket Create S3 bucket
list-buckets List buckets
list-objects List bucket objects
run Execute uploader
run-dry Execute in dry-run mode
clean-bucket Remove all bucket objects
clean Stop and remove volumes

Requirements

  • Docker
  • Docker Compose
  • Python 3.x (inside container runtime)

Python dependencies:

boto3
awscli
awscli-local

Intended Use Cases

  • Local file synchronization to S3
  • CI artifact uploads
  • LocalStack integration testing
  • Educational and technical article demonstrations
  • Infrastructure automation experiments

License

MIT License.

Project Name

akoflow-driver-s3-uploader