GPU Pricing Scraper

This project periodically fetches GPU pricing and offer data from vast.ai and stores it in a Google Cloud Storage bucket. It uses Google Cloud Run for serverless execution and Cloud Scheduler for automation.

Features

• Fetches latest GPU offers from vast.ai using the vastai CLI.
• Stores results as timestamped .tsv files in a GCS bucket.
• Deployable as a container to Google Cloud Run.
• Automated periodic execution via Cloud Scheduler.

Quick Start

Prerequisites

• Google Cloud account and project
• gcloud CLI
• Billing enabled
• Permissions to create Cloud Run, Artifact Registry, Cloud Scheduler, and GCS resources

Deployment

1. Build and push the Docker image:

   gcloud builds submit --tag us-central1-docker.pkg.dev/<PROJECT_ID>/vast-ai-fetcher-repo/vast-ai-fetcher

2. Deploy to Cloud Run:

   gcloud run deploy vast-ai-fetcher \
     --image us-central1-docker.pkg.dev/<PROJECT_ID>/vast-ai-fetcher-repo/vast-ai-fetcher \
     --platform managed \
     --region us-central1 \
     --set-env-vars=GCS_BUCKET_NAME=<YOUR_BUCKET_NAME> \
     --allow-unauthenticated

3. Set up Cloud Scheduler:

   uri=$(gcloud run services describe vast-ai-fetcher --platform managed --region us-central1 --format="value(status.url)")
   service_account=<YOUR_SERVICE_ACCOUNT_EMAIL>
   gcloud scheduler jobs create http vast-ai-fetch-job \
     --schedule="*/30 * * * *" \
     --http-method=POST \
     --uri=$uri/fetch \
     --oidc-service-account-email=$service_account \
     --location=us-central1

Manual Trigger

You can manually trigger a fetch by running:

curl -X POST https://<CLOUD_RUN_URL>/fetch

File Structure

• main.py — Flask app and fetch logic
• Dockerfile — Container build instructions
• deploy.sh — Deployment and scheduler setup script

License

See LICENSE.

---

Disclaimer

This project is not affiliated with vast.ai. Use at your own risk.