Skip to content

Storage doc updates and fixes #134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 6, 2025
Merged

Storage doc updates and fixes #134

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
dc337ec
Removed inactive mountpoints
busolind Jun 5, 2025
0a21fc6
Fixes and updates to object storage page
busolind Jun 5, 2025
d2c2932
Update docs/storage/object.md
busolind Jun 5, 2025
2af84e4
Formatting fixes
busolind Jun 5, 2025
75e7a0f
prompt to title
busolind Jun 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 40 additions & 13 deletions docs/storage/object.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ CSCS offers a public cloud object storage service, based on the Ceph Object Gate

The first step is to configure the profile:

```bash
> aws configure --profile naret-testuser
```console
$ aws configure --profile naret-testuser
AWS Access Key ID [None]: [REDACTED]
AWS Secret Access Key [None]: [REDACTED]
Default region name [None]: cscs-zonegroup
Expand All @@ -42,8 +42,8 @@ s3 =

#### Creating a pre-signed URL

```bash
> aws --profile=naret-testuser s3 presign s3://test-bucket/file.txt --expires-in 300
```console
$ aws --profile=naret-testuser s3 presign s3://test-bucket/file.txt --expires-in 300

https://rgw.cscs.ch/test-bucket/file.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=IA6AOCNMKPDXQ0YNA3DP%2F20241209%2Fcscs-zonegroup%2Fs3%2Faws4_request&X-Amz-Date=20241209T080748Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=f2e2adb457f6fd43401124e4ea2650fba528e614ab661f9c05e2fa2e77691b5d
```
Expand All @@ -57,7 +57,7 @@ A more thorough explanation can be found in the [RGW documentation](https://docs
First, a bucket policy needs to be written:

```json
Copy link
Collaborator

@msimberg msimberg Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
```json
```console

Copy link
Contributor Author

@busolind busolind Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case I believe it looks better with the proper syntax highlight on the json. I'll change it so the filename is in the title

> cat test-public-bucket-anon-from-internet.json
$ cat test-public-bucket-anon-from-internet.json
{
"Version": "2012-10-17",
"Statement": [
Expand All @@ -76,16 +76,16 @@ First, a bucket policy needs to be written:

Then, it can be applied to the bucket:

```
> aws --profile=naret-testuser s3api put-bucket-policy \
```console
$ aws --profile=naret-testuser s3api put-bucket-policy \
--bucket test-public-bucket --policy \
file://test-public-bucket-anon-from-internet.json
```

At this point, the objects in test-public-bucket are accessible via direct links:

```
> s3cmd --configure
```console
$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Expand Down Expand Up @@ -133,15 +133,15 @@ And then confirm.

__IMPORTANT__: The configuration is not complete yet.

```
> s3cmd ls s3://test-bucket
```console
$ s3cmd ls s3://test-bucket
ERROR: S3 error: 403 (SignatureDoesNotMatch)
```

To fix this, it is necessary to edit the `.s3cfg` file, normally located in the user's home directory, and change the `signature_v2` setting to true.

```console
~ $ cat .s3cfg | grep signature_v2
$ cat .s3cfg | grep signature_v2
signature_v2 = True

$ s3cmd ls s3://test-bucket
Expand All @@ -152,6 +152,33 @@ $ s3cmd ls s3://test-bucket

#### Configuration

In order to be able to connect to the S3 endpoint using Cyberduck, a profile supporting path-style requests must be downloaded from here.
In order to be able to connect to the S3 endpoint using Cyberduck, a profile supporting path-style requests must be downloaded from [here](https://profiles.cyberduck.io/S3%20(Deprecated%20path%20style%20requests).cyberduckprofile) or copied from below.

```xml title="S3 (Deprecated path style requests).cyberduckprofile"
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Protocol</key>
<string>s3</string>
<key>Vendor</key>
<string>s3-path-style</string>
<key>Scheme</key>
<string>https</string>
<key>Description</key>
<string>S3 (Deprecated path style requests)</string>
<key>Hostname Configurable</key>
<true/>
<key>Port Configurable</key>
<true/>
<key>Username Configurable</key>
<true/>
<key>Properties</key>
<array>
<string>s3.bucket.virtualhost.disable=true</string>
</array>
</dict>
</plist>
```

![cyberduck](../images/storage/cyberduck.png)
6 changes: 0 additions & 6 deletions docs/storage/transfer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,9 @@ Currently Globus provide the following mount points at CSCS:

| Mount Point | Description |
|---|---|
| `/scratch/snx3000` | old Daint scratch area |
| `/store` | old Daint store area |
| `/project` | old Daint project area |
| `/users` | old Daint home directory |
| `/scratch/shared` | old Scratch-Shared area ( old meteoswiss clusters ) |
| `/iopsstor/scratch/cscs` | Mounted on Clariden |
| `/capstor/scratch/cscs` | New Alps Daint scratch area |
| `/capstor/store/cscs` | New Alps Daint store area |
| `/capstor/users/cscs` | Home directory for Bristen/Scopi/Errigal |
| `/vast/users/cscs` | New Alps vclusters home directory ( Alps Daint and others ) |

[](){#ref-data-xfer-internal}
Expand Down