Projects on AWS S3

Crystal has special support for opening projects hosted on AWS S3. This can be useful if you have large *.crystalproj projects that you want to archive in cloud storage while still being able to browse them with Crystal.

Projects opened from S3 are always opened in read-only mode.

Setup

1. (Recommended) Convert to Pack16 format before uploading

Before uploading, consider converting your project's revision storage format to Pack16. The Pack16 format bundles groups of 16 consecutive revisions together into uncompressed .zip files, increasing the average file size in the revisions directory from about 100 KB to about 1.5 MB. This is significantly more efficient when storing projects on S3, especially if using a storage class with a large minimum billable object size such as S3 Glacier (which has a minimum billable object size of 128 KB).

To convert, open your project locally in Crystal and change the revision storage format to Pack16 in the project settings before uploading.

2. Upload your project to S3

Upload your *.crystalproj directory to an S3 bucket, preserving its directory structure. You can use the AWS CLI, the AWS Console, or any S3-compatible upload tool.

For example, if your project is at ~/Website Backups/MySite.crystalproj/, you might upload it to:

s3://my-bucket/Website Backups/MySite.crystalproj/

3. Create or choose an IAM user

You need AWS credentials with permission to read objects from your S3 bucket.

Option A: Use your own IAM user account. If you already use the AWS CLI, your existing credentials likely have sufficient permissions.

Option B: Create a dedicated IAM user with minimal permissions. This is recommended if you want to limit access to only what Crystal needs. Create an IAM user and attach a policy like the following, which grants the minimum permissions needed to read a specific project:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:HeadObject"
            ],
            "Resource": "arn:aws:s3:::my-bucket/Website Backups/MySite.crystalproj/*"
        }
    ]
}

Replace my-bucket and the path with your actual bucket name and project path.

4. Create an Access Key

If your IAM user does not already have an Access Key, create one in the AWS Console under IAM > Users > (your user) > Security credentials > Access keys. Save the Access Key ID and Secret Access Key. You will need both.

5. (Recommended) Save your AWS credentials to a profile

An easy way to save AWS credentials is to configure the AWS CLI tool to use them. Crystal can read the credentials configured for that tool.

If you are using your regular IAM user account, run:

$ aws configure

This saves your Access Key ID and Secret Access Key to your default AWS profile.

If you are using a dedicated IAM user, save the credentials to a named profile:

$ aws configure --profile=crystal_s3_readonly

Running Crystal with an S3 project

Using the graphical interface

The easiest way to open an S3 project is through Crystal's graphical interface:

1. Launch Crystal (or close any currently open project).
2. Choose File > Open Project from S3... from the menu bar.
3. Enter the S3 URL of your project (e.g. s3://my-bucket/Website Backups/MySite.crystalproj/).
4. Choose how to provide your AWS credentials:
   • Use saved AWS profile — select a profile previously configured with aws configure.
   • Enter credentials manually — enter your Access Key ID and Secret Access Key directly.
   • Leave both credential fields empty to use boto3's default credential chain (environment variables, instance profile, etc.).
5. Click Open.

You can also embed credentials directly in the URL (see Embedding credentials in the URL below), in which case the credential section of the dialog is disabled automatically.

Using the command line interface

Crystal can also open an S3 project through its command line interface.

If you are using credentials from your default profile, open your project with:

$ crystal --readonly "s3://my-bucket/Website Backups/MySite.crystalproj/"

If you are using credentials from a named profile, set the AWS_PROFILE environment variable to the name of the profile to use when running Crystal:

$ AWS_PROFILE=crystal_s3_readonly crystal --readonly "s3://my-bucket/Website Backups/MySite.crystalproj/"

If you want to pass credentials manually, you can pass them using environment variables:

$ export AWS_ACCESS_KEY_ID=AKIA...
$ export AWS_SECRET_ACCESS_KEY=...
$ crystal --readonly "s3://my-bucket/Website Backups/MySite.crystalproj/"

You can also include credentials directly in the S3 URL. This is convenient for one-off use, but be careful not to expose credentials in shell history or scripts:

$ crystal --readonly "s3://$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY@my-bucket/Website Backups/MySite.crystalproj/?region=us-east-2"

The ?region= parameter is recommended when credentials are embedded in the URL. If omitted, the region defaults to us-east-1.

Notes

• Projects on S3 are always opened as read-only. Attempting to open an S3 project as writable will result in an error.
• Crystal incrementally streams data from the project's database file on demand. When browsing to individual revisions, only those revisions will be downloaded on demand.