Skip to content

Commit e08a330

Browse files
MaxJa4MaxJa4
authored
Add new storage backend: Dropbox (#103) (#251)
* Add new storage backend: Dropbox (#103) * Remove duplicate check * Add concurrency level for parallel upload to dropbox. * Fixed some instabilites. Changed default concurrency to 6. * Added some env config vars to readme. WIP * Wrap errors for storage backend creation. * Fixed token issue, added OAuth2 including recipe and docs. * Readme typo fix * Test for dropbox integration * Update info and TOC * Missed a file * Docker-compose fix * Fix endpoint connection * Fix container names * Fix log fetching * Fix log fetching (again) * Print command output to logs * Addressing comments part 1 * Address comments part 2 * OpenAPI Mock spec path adjusted * Dropbox FileMetadata reflection refactored * NaturalNumber type added * Add OAuth2 mock server for CI testing * Fix env name of oauth2 endpoint * Remove hostname * Add forgotten change to commit... * Fix oauth2 endpoint "Worked on my machine" * Try again * Try suggested hostname again * Fix docker internal DNS resolving issues (as suggested by oauth2 mock docs) * Add docker network, remove hostname * Network not external * Last hostname try * Add more delay, add oauth2 endpoint log * Temp CI log output of command even when failing * Try different config and method * Add custom server-hostname. Rename test folder to accellerate debugging * Try that fix again * Adding quotes * Port fix attempt * Try localhost * Try extra hosts * Change network mode * Undo some changes * Use static IP * Remove specific IP binding * Change to default net driver * Fix static IP * Squash for revert * Revert "Squash for revert" This reverts commit e9b617b. * Actual fix for CI testing from #257
1 parent 47326c7 commit e08a330

File tree

10 files changed

+13292
-11
lines changed

10 files changed

+13292
-11
lines changed

‎README.md‎

Lines changed: 79 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,10 @@
44

55
# docker-volume-backup
66

7-
Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage or SSH compatible storage.
7+
Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage.
88

99
The [offen/docker-volume-backup](https://hub.docker.com/r/offen/docker-volume-backup) Docker image can be used as a lightweight (below 15MB) sidecar container to an existing Docker setup.
10-
It handles __recurring or one-off backups of Docker volumes__ to a __local directory__, __any S3, WebDAV, Azure Blob Storage or SSH compatible storage (or any combination) and rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.
10+
It handles __recurring or one-off backups of Docker volumes__ to a __local directory__, __any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage (or any combination) and rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.
1111

1212
<!-- MarkdownTOC -->
1313

@@ -36,6 +36,7 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
3636
- [Define different retention schedules](#define-different-retention-schedules)
3737
- [Use special characters in notification URLs](#use-special-characters-in-notification-urls)
3838
- [Handle file uploads using third party tools](#handle-file-uploads-using-third-party-tools)
39+
- [Setup Dropbox storage backend](#setup-dropbox-storage-backend)
3940
- [Recipes](#recipes)
4041
- [Backing up to AWS S3](#backing-up-to-aws-s3)
4142
- [Backing up to Filebase](#backing-up-to-filebase)
@@ -44,6 +45,7 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
4445
- [Backing up to WebDAV](#backing-up-to-webdav)
4546
- [Backing up to SSH](#backing-up-to-ssh)
4647
- [Backing up to Azure Blob Storage](#backing-up-to-azure-blob-storage)
48+
- [Backing up to Dropbox](#backing-up-to-dropbox)
4749
- [Backing up locally](#backing-up-locally)
4850
- [Backing up to AWS S3 as well as locally](#backing-up-to-aws-s3-as-well-as-locally)
4951
- [Running on a custom cron schedule](#running-on-a-custom-cron-schedule)
@@ -356,6 +358,26 @@ You can populate below template according to your requirements and use it as you
356358

357359
# AZURE_STORAGE_ENDPOINT="https://{{ .AccountName }}.blob.core.windows.net/"
358360

361+
# Absolute remote path in your Dropbox where the backups shall be stored.
362+
# Note: Use your app's subpath in Dropbox, if it doesn't have global access.
363+
# Consulte the README for further information.
364+
365+
# DROPBOX_REMOTE_PATH="/my/directory"
366+
367+
# Number of concurrent chunked uploads for Dropbox.
368+
# Values above 6 usually result in no enhancements.
369+
370+
# DROPBOX_CONCURRENCY_LEVEL="6"
371+
372+
# App key and app secret from your app created at https://www.dropbox.com/developers/apps/info
373+
374+
# DROPBOX_APP_KEY=""
375+
# DROPBOX_APP_SECRET=""
376+
377+
# Refresh token to request new short-lived tokens (OAuth2). Consult README to see how to get one.
378+
379+
# DROPBOX_REFRESH_TOKEN=""
380+
359381
# In addition to storing backups remotely, you can also keep local copies.
360382
# Pass a container-local path to store your backups if needed. You also need to
361383
# mount a local folder or Docker volume into that location (`/archive`
@@ -1020,6 +1042,37 @@ volumes:
10201042

10211043
Commands will be invoked with the filepath of the tar archive passed as `COMMAND_RUNTIME_BACKUP_FILEPATH`.
10221044

1045+
### Setup Dropbox storage backend
1046+
1047+
#### Auth-Setup:
1048+
1049+
1. Create a new Dropbox App in the [App Console](https://www.dropbox.com/developers/apps)
1050+
2. Open your new Dropbox App and set `DROPBOX_APP_KEY` and `DROPBOX_APP_SECRET` in your environment (e.g. docker-compose.yml) accordingly
1051+
3. Click on `Permissions` in your app and make sure, that the following permissions are cranted (or more):
1052+
- `files.metadata.write`
1053+
- `files.metadata.read`
1054+
- `files.content.write`
1055+
- `files.content.read`
1056+
4. Replace APPKEY in `https://www.dropbox.com/oauth2/authorize?client_id=APPKEY&token_access_type=offline&response_type=code` with the app key from step 2
1057+
5. Visit the URL and confirm the access of your app. This gives you an `auth code` -> save it somewhere!
1058+
6. Replace AUTHCODE, APPKEY, APPSECRET accordingly and perform the request:
1059+
```
1060+
curl https://api.dropbox.com/oauth2/token \
1061+
-d code=AUTHCODE \
1062+
-d grant_type=authorization_code \
1063+
-d client_id=APPKEY \
1064+
-d client_secret=APPSECRET
1065+
```
1066+
7. Execute the request. You will get a JSON formatted reply. Use the value of the `refresh_token` for the last environment variable `DROPBOX_REFRESH_TOKEN`
1067+
8. You should now have `DROPBOX_APP_KEY`, `DROPBOX_APP_SECRET` and `DROPBOX_REFRESH_TOKEN` set. These don't expire.
1068+
1069+
Note: Using the "Generated access token" in the app console is not supported, as it is only very short lived and therefore not suitable for an automatic backup solution. The refresh token handles this automatically - the setup procedure above is only needed once.
1070+
1071+
#### Other parameters
1072+
1073+
Important: If you chose `App folder` access during the creation of your Dropbox app in step 1 above, you can only write in the app's directory!
1074+
This means, that `DROPBOX_REMOTE_PATH` must start with e.g. `/Apps/YOUR_APP_NAME` or `/Apps/YOUR_APP_NAME/some_sub_dir`
1075+
10231076
## Recipes
10241077

10251078
This section lists configuration for some real-world use cases that you can mix and match according to your needs.
@@ -1187,6 +1240,30 @@ volumes:
11871240
data:
11881241
```
11891242
1243+
### Backing up to Dropbox
1244+
1245+
See [Dropbox Setup](#setup-dropbox-storage-backend) on how to get the appropriate environment values.
1246+
1247+
```yml
1248+
version: '3'
1249+
1250+
services:
1251+
# ... define other services using the `data` volume here
1252+
backup:
1253+
image: offen/docker-volume-backup:v2
1254+
environment:
1255+
DROPBOX_REFRESH_TOKEN: REFRESH_KEY # replace
1256+
DROPBOX_APP_KEY: APP_KEY # replace
1257+
DROPBOX_APP_SECRET: APP_SECRET # replace
1258+
DROPBOX_REMOTE_PATH: /Apps/my-test-app/some_subdir # replace
1259+
volumes:
1260+
- data:/backup/my-app-backup:ro
1261+
- /var/run/docker.sock:/var/run/docker.sock:ro
1262+
1263+
volumes:
1264+
data:
1265+
```
1266+
11901267
### Backing up locally
11911268
11921269
```yml

‎cmd/backup/config.go‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import (
1010
"io/ioutil"
1111
"os"
1212
"regexp"
13+
"strconv"
1314
"time"
1415
)
1516

@@ -70,6 +71,13 @@ type Config struct {
7071
AzureStorageContainerName string `split_words:"true"`
7172
AzureStoragePath string `split_words:"true"`
7273
AzureStorageEndpoint string `split_words:"true" default:"https://{{ .AccountName }}.blob.core.windows.net/"`
74+
DropboxEndpoint string `split_words:"true" default:"https://api.dropbox.com/"`
75+
DropboxOAuth2Endpoint string `envconfig:"DROPBOX_OAUTH2_ENDPOINT" default:"https://api.dropbox.com/"`
76+
DropboxRefreshToken string `split_words:"true"`
77+
DropboxAppKey string `split_words:"true"`
78+
DropboxAppSecret string `split_words:"true"`
79+
DropboxRemotePath string `split_words:"true"`
80+
DropboxConcurrencyLevel NaturalNumber `split_words:"true" default:"6"`
7381
}
7482

7583
func (c *Config) resolveSecret(envVar string, secretPath string) (string, error) {
@@ -135,3 +143,21 @@ func (r *RegexpDecoder) Decode(v string) error {
135143
*r = RegexpDecoder{Re: re}
136144
return nil
137145
}
146+
147+
type NaturalNumber int
148+
149+
func (n *NaturalNumber) Decode(v string) error {
150+
asInt, err := strconv.Atoi(v)
151+
if err != nil {
152+
return fmt.Errorf("config: error converting %s to int", v)
153+
}
154+
if asInt <= 0 {
155+
return fmt.Errorf("config: expected a natural number, got %d", asInt)
156+
}
157+
*n = NaturalNumber(asInt)
158+
return nil
159+
}
160+
161+
func (n *NaturalNumber) Int() int {
162+
return int(*n)
163+
}

‎cmd/backup/script.go‎

Lines changed: 28 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ import (
1919

2020
"github.com/offen/docker-volume-backup/internal/storage"
2121
"github.com/offen/docker-volume-backup/internal/storage/azure"
22+
"github.com/offen/docker-volume-backup/internal/storage/dropbox"
2223
"github.com/offen/docker-volume-backup/internal/storage/local"
2324
"github.com/offen/docker-volume-backup/internal/storage/s3"
2425
"github.com/offen/docker-volume-backup/internal/storage/ssh"
@@ -70,11 +71,12 @@ func newScript() (*script, error) {
7071
StartTime: time.Now(),
7172
LogOutput: logBuffer,
7273
Storages: map[string]StorageStats{
73-
"S3": {},
74-
"WebDAV": {},
75-
"SSH": {},
76-
"Local": {},
77-
"Azure": {},
74+
"S3": {},
75+
"WebDAV": {},
76+
"SSH": {},
77+
"Local": {},
78+
"Azure": {},
79+
"Dropbox": {},
7880
},
7981
},
8082
}
@@ -155,7 +157,7 @@ func newScript() (*script, error) {
155157
PartSize: s.c.AwsPartSize,
156158
}
157159
if s3Backend, err := s3.NewStorageBackend(s3Config, logFunc); err != nil {
158-
return nil, err
160+
return nil, fmt.Errorf("newScript: error creating s3 storage backend: %w", err)
159161
} else {
160162
s.storages = append(s.storages, s3Backend)
161163
}
@@ -170,7 +172,7 @@ func newScript() (*script, error) {
170172
RemotePath: s.c.WebdavPath,
171173
}
172174
if webdavBackend, err := webdav.NewStorageBackend(webDavConfig, logFunc); err != nil {
173-
return nil, err
175+
return nil, fmt.Errorf("newScript: error creating webdav storage backend: %w", err)
174176
} else {
175177
s.storages = append(s.storages, webdavBackend)
176178
}
@@ -187,7 +189,7 @@ func newScript() (*script, error) {
187189
RemotePath: s.c.SSHRemotePath,
188190
}
189191
if sshBackend, err := ssh.NewStorageBackend(sshConfig, logFunc); err != nil {
190-
return nil, err
192+
return nil, fmt.Errorf("newScript: error creating ssh storage backend: %w", err)
191193
} else {
192194
s.storages = append(s.storages, sshBackend)
193195
}
@@ -212,11 +214,28 @@ func newScript() (*script, error) {
212214
}
213215
azureBackend, err := azure.NewStorageBackend(azureConfig, logFunc)
214216
if err != nil {
215-
return nil, err
217+
return nil, fmt.Errorf("newScript: error creating azure storage backend: %w", err)
216218
}
217219
s.storages = append(s.storages, azureBackend)
218220
}
219221

222+
if s.c.DropboxRefreshToken != "" && s.c.DropboxAppKey != "" && s.c.DropboxAppSecret != "" {
223+
dropboxConfig := dropbox.Config{
224+
Endpoint: s.c.DropboxEndpoint,
225+
OAuth2Endpoint: s.c.DropboxOAuth2Endpoint,
226+
RefreshToken: s.c.DropboxRefreshToken,
227+
AppKey: s.c.DropboxAppKey,
228+
AppSecret: s.c.DropboxAppSecret,
229+
RemotePath: s.c.DropboxRemotePath,
230+
ConcurrencyLevel: s.c.DropboxConcurrencyLevel.Int(),
231+
}
232+
dropboxBackend, err := dropbox.NewStorageBackend(dropboxConfig, logFunc)
233+
if err != nil {
234+
return nil, fmt.Errorf("newScript: error creating dropbox storage backend: %w", err)
235+
}
236+
s.storages = append(s.storages, dropboxBackend)
237+
}
238+
220239
if s.c.EmailNotificationRecipient != "" {
221240
emailURL := fmt.Sprintf(
222241
"smtp://%s:%s@%s:%d/?from=%s&to=%s",

‎go.mod‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,13 @@ require (
2020
golang.org/x/sync v0.3.0
2121
)
2222

23+
require (
24+
github.com/golang/protobuf v1.5.2 // indirect
25+
golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 // indirect
26+
google.golang.org/appengine v1.6.7 // indirect
27+
google.golang.org/protobuf v1.28.1 // indirect
28+
)
29+
2330
require (
2431
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.6.0 // indirect
2532
github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 // indirect
@@ -28,6 +35,7 @@ require (
2835
github.com/docker/distribution v2.8.2+incompatible // indirect
2936
github.com/docker/go-connections v0.4.0 // indirect
3037
github.com/docker/go-units v0.4.0 // indirect
38+
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5
3139
github.com/dustin/go-humanize v1.0.1 // indirect
3240
github.com/fatih/color v1.13.0 // indirect
3341
github.com/gogo/protobuf v1.3.2 // indirect

‎go.sum‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,8 @@ github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKoh
257257
github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
258258
github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
259259
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
260+
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 h1:FT+t0UEDykcor4y3dMVKXIiWJETBpRgERYTGlmMd7HU=
261+
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5/go.mod h1:rSS3kM9XMzSQ6pw91Qgd6yB5jdt70N4OdtrAf74As5M=
260262
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
261263
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
262264
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
@@ -785,6 +787,7 @@ golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb/go.mod h1:jaDAt6Dkxork7Lm
785787
golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2/go.mod h1:jaDAt6Dkxork7LmZnYtzbRWj0W47D86a3TGe0YHBvmE=
786788
golang.org/x/oauth2 v0.0.0-20220822191816-0ebed06d0094/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg=
787789
golang.org/x/oauth2 v0.0.0-20220909003341-f21342109be1/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg=
790+
golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 h1:nt+Q6cXKz4MosCSpnbMtqiQ8Oz0pxTef2B4Vca2lvfk=
788791
golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg=
789792
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
790793
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -1046,6 +1049,7 @@ google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7
10461049
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
10471050
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
10481051
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
1052+
google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c=
10491053
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
10501054
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
10511055
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=

0 commit comments

Comments
 (0)