Skip to content

Commit a5da65e

Browse files
committed
Updated README to reflect S3StorageDriver
1 parent 9cc9a6e commit a5da65e

File tree

2 files changed

+26
-3
lines changed

2 files changed

+26
-3
lines changed

README.md

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,30 @@ radar-hdfs-restructure --compression gzip --nameservice <hdfs_node> --output-di
5050

5151
### Storage
5252

53-
When using local storage, to set the output user ID and group ID, specify the `-p local-uid=123` and `-p local-gid=12` properties.
53+
There are two storage drivers implemented: `org.radarbase.hdfs.storage.LocalStorageDriver` for an output directory on the local file system or `org.radarbase.hdfs.storage.S3StorageDriver` for storage on an object store.
54+
55+
`LocalStorageDriver` takes the following properties:
56+
```yaml
57+
storage:
58+
factory: org.radarbase.hdfs.storage.LocalStorageDriver
59+
properties:
60+
# User ID to write data as
61+
localUid: 123
62+
# Group ID to write data as
63+
localGid: 123
64+
```
65+
66+
With the `S3StorageDriver`, use the following configuration instead:
67+
```yaml
68+
storage:
69+
factory: org.radarbase.hdfs.storage.S3StorageDriver
70+
properties:
71+
# Object store URL
72+
s3EndpointUrl: s3://my-region.s3.aws.amazon.com
73+
# Bucket to use
74+
s3Bucket: myBucketName
75+
```
76+
Ensure that the environment variables contain the authorized AWS keys that allow the service to list, download and upload files to the respective bucket.
5477

5578
### Service
5679

src/main/java/org/radarbase/hdfs/storage/LocalStorageDriver.kt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,8 @@ class LocalStorageDriver : StorageDriver {
3030
private var gid = -1
3131

3232
override fun init(properties: Map<String, String>) {
33-
uid = properties["local-uid"]?.toIntOrNull() ?: -1
34-
gid = properties["local-gid"]?.toIntOrNull() ?: -1
33+
uid = properties["localUid"]?.toIntOrNull() ?: -1
34+
gid = properties["localGid"]?.toIntOrNull() ?: -1
3535
}
3636

3737
@Throws(IOException::class)

0 commit comments

Comments
 (0)