Skip to content

Commit 7b752a3

Browse files
pedro-psbpedro-psb
committed
Update docs about STORAGES setting
1 parent 8234d4c commit 7b752a3

File tree

4 files changed

+188
-10
lines changed

4 files changed

+188
-10
lines changed

‎docs/admin/guides/_SUMMARY.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,9 @@
33
* [Using external service](auth/external.md)
44
* [Using Keycloak](auth/keycloak.md)
55
* [Using JSON Header](auth/json_header.md)
6+
* *.md
7+
* Configuration
8+
* [Introduction](configure-pulp/index.md)
9+
* [Configure Storage backends](configure-pulp/configure-storages.md)
610
* *.md
711

‎docs/admin/guides/configure-pulp/configure-storages.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Configure Storage Backend
2+
3+
Pulp uses [django-storages](https://django-storages.readthedocs.io) to support multiple storage backends.
4+
See the reference for the [`STORAGES` settings](site:pulpcore/docs/admin/reference/settings/#storages).
5+
6+
## Modifying the settings
7+
8+
The settings can be updated via the `settings.py` file or through environment variables
9+
and have support to dynaconf merge features (learn more on the [Settings Introduction](site:pulpcore/docs/admin/guides/configure-pulp/introduction/)).
10+
11+
To learn where and how to modify the files and envvars, refer to the appropriate installation method documentation:
12+
13+
* [Container (single-process) quickstart](site:pulp-oci-images/docs/admin/tutorials/quickstart/#single-container)
14+
* [Container (multi-process) quickstart](site:pulp-oci-images/docs/admin/tutorials/quickstart/#podman-or-docker-compose)
15+
* [Pulp Operator quickstart](site:pulp-operator/docs/admin/tutorials/quickstart-kubernetes/)
16+
17+
## Local Filesystem (default)
18+
19+
### Example
20+
21+
In this example, the storage file permission and `MEDIA_ROOT` are overridden:
22+
23+
```python
24+
STORAGES = {
25+
"default": {
26+
"BACKEND": "pulpcore.app.models.storage.FileSystem",
27+
"OPTIONS": {
28+
"file_permissions_mode": 0o600,
29+
"directory_permissions_mode": 0o600,
30+
},
31+
},
32+
}
33+
MEDIA_ROOT="/custom/media/root" # default: /var/lib/pulp/media
34+
```
35+
36+
Notes:
37+
38+
* The [`MEDIA_ROOT`](site:pulpcore/docs/admin/reference/settings/#media_root) setting specifies where Pulp
39+
will save the files.
40+
* Pulp customizes Django's default class, `django.core.files.storage.FileSystemStorage`.
41+
* There are some other related global definitions provided by django:
42+
* `MEDIA_URL`
43+
* `FILE_UPLOAD_PERMISSIONS`
44+
* `FILE_UPLOAD_DIRECTORY_PERMISSIONS`
45+
46+
Comprehensive options for Local Filesystem can be found in
47+
[Django docs](https://docs.djangoproject.com/en/4.2/ref/files/storage/#django.core.files.storage.FileSystemStorage).
48+
49+
### Amazon S3
50+
51+
#### Setup
52+
53+
Before you can configure Amazon S3 storage to use with Pulp, ensure that you complete the following steps
54+
(consult the official Amazon S3 docs for precise steps).
55+
56+
1. Set up an AWS account.
57+
2. Create an S3 bucket for Pulp to use.
58+
3. In AWS Identity and Access Management (IAM), create a user that Pulp can use to access your S3 bucket.
59+
4. Save the access key id and secret access key.
60+
61+
#### Example
62+
63+
In this example, the storage uses a bucket called `pulp3` that is hosted in region `eu-central-1`:
64+
65+
```python
66+
MEDIA_ROOT = ""
67+
STORAGES = {
68+
"default": {
69+
"BACKEND": "storages.backends.s3.S3Storage",
70+
"OPTIONS": {
71+
"access_key": 'AKIAIT2Z5TDYPX3ARJBA',
72+
"secret_key": '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',
73+
"bucket_name": 'pulp3',
74+
"signature_version": "s3v4",
75+
"addressing_style": "path",
76+
"region_name": "eu-central-1",
77+
},
78+
},
79+
}
80+
```
81+
82+
Notes:
83+
84+
* `MEDIA_ROOT` must be set to an empty string.
85+
* You can omit `access_key` and `secret_key` if:
86+
1. The system that hosts Pulp is running on AWS
87+
2. And the [`instance_profile`](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html) provides access to the S3 bucket
88+
89+
Comprehensive options for Amazon S3 can be found in
90+
[`django-storages` docs](https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#configuration-settings).
91+
92+
### Azure Blob storage
93+
94+
#### Setup
95+
96+
Before you can configure Azure Blob storage to use with Pulp, ensure that you complete the following steps
97+
(consult the official Azure documentation for precise steps).
98+
99+
1. Set up an Azure account and create a storage account.
100+
2. In your storage account, create a container under `Blob` service.
101+
3. Obtain the access credentials so that you can later configure Pulp to access your Azure Blob storage. You can find the access credentials
102+
at the storage account level, at Access keys (these are automatically generated).
103+
104+
#### Example
105+
106+
In this example, the storage uses a container called `pulp3` with the `pulp-account` username.
107+
108+
```python
109+
MEDIA_ROOT = ""
110+
STORAGES = {
111+
"default": {
112+
"BACKEND": "storages.backends.azure_storage.AzureStorage",
113+
"OPTIONS": {
114+
"account_name": 'pulp-account',
115+
"azure_container": 'pulp3',
116+
"account_key": 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==',
117+
"expiration_secs": 60,
118+
"overwrite_files": 'True',
119+
"location": 'pulp3'
120+
},
121+
},
122+
}
123+
```
124+
125+
Notes:
126+
127+
* `MEDIA_ROOT` must be set to an empty string.
128+
129+
Comprehensive options for Azure Blob can be found in
130+
[`django-storages` docs](https://django-storages.readthedocs.io/en/latest/backends/azure.html#configuration-settings).

docs/admin/guides/configure-pulp.md renamed to docs/admin/guides/configure-pulp/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Configure Pulp
1+
# Introduction
22

33
Pulp uses [dynaconf](https://www.dynaconf.com/) for its settings which allows you
44
to configure Pulp settings using various ways:

‎docs/admin/reference/settings.md‎

Lines changed: 53 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -73,16 +73,59 @@ instructions on how to configure the database, refer to `database installation <
7373

7474
### DEFAULT_FILE_STORAGE
7575

76-
By default, Pulp uses the local filesystem to store files. The default option which
77-
uses the local filesystem is `pulpcore.app.models.storage.FileSystem`.
76+
!!! warning "Deprecated in `3.70`"
77+
The `DEFAULT_FILE_STORAGE` setting was deprecated in
78+
[django `4.2`](https://docs.djangoproject.com/en/4.2/ref/settings/#default-file-storage)
79+
and will be removed from pulpcore on `3.85`.
80+
Between `3.70` and `3.85`, replace it with [`STORAGES`](#storages).
81+
82+
### STORAGES
83+
84+
!!! note "Added in `3.70`"
85+
Starting in `3.70`, this should be used in the place of [`DEFAULT_FILE_STORAGE`](#default-file-storage).
86+
87+
Pulp uses [django-storages](https://django-storages.readthedocs.io/en/latest/index.html) to support multiple storage backends.
88+
If no backend is configured, Pulp will by default use the local filesystem (`pulpcore.app.models.storage.FileSystem`).
89+
90+
To use another backend storage, you'll need to:
91+
92+
1. Setup your storage and gather required credentials and specific configs.
93+
2. Ensure the `django-storage[s3|google|azure]` python package is installed. This depends on the installation method.
94+
3. Configure the default `BACKEND` storage class.
95+
4. Configure available `OPTIONS` for that backend.
96+
97+
Learn more on the
98+
[Configure Storages](site:pulpcore/docs/admin/guides/configure-pulp/configure-storages/) guide.
99+
100+
As an example, an Amazon S3 might look like this:
101+
102+
```python
103+
STORAGES = {
104+
"default": {
105+
"BACKEND": "storages.backends.s3.S3Storage",
106+
"OPTIONS": {
107+
"access_key": 'AKIAIT2Z5TDYPX3ARJBA',
108+
"secret_key": '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',
109+
"bucket_name": 'pulp3',
110+
"signature_version": "s3v4",
111+
"addressing_style": "path",
112+
"region_name": "eu-central-1",
113+
},
114+
},
115+
}
116+
```
117+
118+
Overview of the integration with Pulp:
119+
120+
- **Supported**: We support (test) Amazon S3 and Azure.
121+
- **Untested**: Other backends provided by `django-storages` may work as well, but we provide no guarantee.
122+
- **Known caveat**: Using SFTP Storage is not recommended in Pulp's current state, and doing so can lead to file corruption. This is because Pulp currently uses coroutines that seem to be incompatible with Django's SFTPStorage implementation.
78123

79-
For more information about different Pulp storage options, see the
80-
`storage documentation `.
81124

82125
### REDIRECT_TO_OBJECT_STORAGE
83126

84127
When set to `True` access to artifacts is redirected to the corresponding Cloud storage
85-
configured in `DEFAULT_FILE_STORAGE` using pre-authenticated URLs. When set to `False`
128+
configured in `STORAGES['default']['BACKEND']` using pre-authenticated URLs. When set to `False`
86129
artifacts are always served by the content app instead.
87130

88131
Defaults to `True`; ignored for local file storage.
@@ -91,9 +134,10 @@ Defaults to `True`; ignored for local file storage.
91134

92135
The location where Pulp will store files. By default this is `/var/lib/pulp/media`.
93136

94-
This only affects storage location when `DEFAULT_FILE_STORAGE` is set to
95-
`pulpcore.app.models.storage.FileSystem`. See the `storage documentation ` for
96-
more info.
137+
This only affects storage location when `STORAGES['default']['BACKEND']` is set to
138+
`pulpcore.app.models.storage.FileSystem`.
139+
140+
See the [storage documentation](site:pulpcore/docs/admin/guides/configure-pulp/configure-storages/) for more info.
97141

98142
It should have permissions of:
99143

@@ -188,7 +232,7 @@ It should have permissions of:
188232
### CHUNKED_UPLOAD_DIR
189233

190234
A relative path inside the DEPLOY_ROOT directory used exclusively for uploaded chunks. The
191-
uploaded chunks are stored in the default storage specified by `DEFAULT_FILE_STORAGE`. This
235+
uploaded chunks are stored in the default storage specified by `STORAGES['default']['BACKEND']`. This
192236
option allows users to customize the actual place where chunked uploads should be stored within
193237
the declared storage. The default, `upload`, is sufficient for most use cases. A change to
194238
this setting only applies to uploads created after the change.

0 commit comments

Comments
 (0)