Skip to content

Commit 53599db

Browse files
Add format option and update README
1 parent 793025a commit 53599db

File tree

3 files changed

+21
-7
lines changed

3 files changed

+21
-7
lines changed

README.md

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,20 +15,30 @@ Build jar from source with
1515
```shell
1616
./gradlew build
1717
```
18-
and find the output JAR file as `build/libs/restructurehdfs-0.3.1-all.jar`. Then run with:
18+
and find the output JAR file as `build/libs/restructurehdfs-0.3.3-all.jar`. Then run with:
1919

2020
```shell
21-
java -jar restructurehdfs-0.3.1-all.jar <webhdfs_url> <hdfs_topic_path> <output_folder>
21+
java -jar restructurehdfs-0.3.3-all.jar --hdfs-uri <webhdfs_url> --hdfs-root-directory <hdfs_topic_path> --output-directory <output_folder>
22+
```
23+
or you can use the short form as well like -
24+
```shell
25+
java -jar restructurehdfs-0.3.3-all.jar -u <webhdfs_url> -i <hdfs_topic_path> -o <output_folder>
2226
```
2327

24-
By default, this will output the data in CSV format. If JSON format is preferred, use the following instead:
28+
To display the usage and all available options you can use the help option as follows -
29+
```shell
30+
java -jar restructurehdfs-0.3.3-all.jar --help
2531
```
26-
java -Dorg.radarcns.format=json -jar restructurehdfs-0.3.1-all.jar <webhdfs_url> <hdfs_topic_path> <output_folder>
32+
Note that the options preceded by the `*` in the above output are required to run the app.
33+
34+
By default, this will output the data in CSV format. If JSON format is preferred, use the following instead:
35+
```shell
36+
java -jar restructurehdfs-0.3.3-all.jar --format json --hdfs-uri <webhdfs_url> --hdfs-root-directory <hdfs_topic_path> --output-directory <output_folder>
2737
```
2838

2939
Another option is to output the data in compressed form. All files will get the `gz` suffix, and can be decompressed with a GZIP decoder. Note that for a very small number of records, this may actually increase the file size.
3040
```
31-
java -Dorg.radarcns.compress=gzip -jar restructurehdfs-0.3.1-all.jar <webhdfs_url> <hdfs_topic_path> <output_folder>
41+
java -jar restructurehdfs-0.3.3-all.jar --compression gzip --hdfs-uri <webhdfs_url> --hdfs-root-directory <hdfs_topic_path> --output-directory <output_folder>
3242
```
3343

34-
Finally, by default, files records are not deduplicated after writing. To enable this behaviour, specify the option `-Dorg.radarcns.deduplicate=true`. This set to false by default because of an issue with Biovotion data. Please see - [issue #16](https://github.com/RADAR-base/Restructure-HDFS-topic/issues/16) before enabling it.
44+
Finally, by default, files records are not deduplicated after writing. To enable this behaviour, specify the option `--deduplicate true`. This set to false by default because of an issue with Biovotion data. Please see - [issue #16](https://github.com/RADAR-base/Restructure-HDFS-topic/issues/16) before enabling it.

src/main/java/org/radarcns/RestructureAvroRecords.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@ public RestructureAvroRecords(String inputPath, String outputPath) {
115115
this.setOutputPath(outputPath);
116116

117117
String extension;
118-
if (System.getProperty("org.radarcns.format", "csv").equalsIgnoreCase("json")) {
118+
if (commandLineArgs.format.equalsIgnoreCase("json")) {
119119
logger.info("Writing output files in JSON format");
120120
converterFactory = JsonAvroConverter.getFactory();
121121
extension = "json";

src/main/java/org/radarcns/util/commandline/CommandLineArgs.java

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,10 @@
33
import com.beust.jcommander.Parameter;
44

55
public class CommandLineArgs {
6+
7+
@Parameter(names = { "-f", "--format" }, description = "Format to use when converting the files. JSON and CSV is available.")
8+
public String format = "csv";
9+
610
@Parameter(names = { "-c", "--compression" }, description = "Compression to use when converting the files. Gzip is available.")
711
public String compression = "none";
812

0 commit comments

Comments
 (0)