Skip to content

Commit e16ea03

Browse files
Crafter-Yitzg
Crafter-Y
and
itzg
authored
Implementing multiple destinations for mcopy downloads (#641)
Co-authored-by: Geoff Bourne <[email protected]>
1 parent add5395 commit e16ea03

File tree

4 files changed

+457
-90
lines changed

4 files changed

+457
-90
lines changed

‎.devcontainer/devcontainer.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
{
44
"name": "Java",
55
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
6-
"image": "mcr.microsoft.com/devcontainers/java:17",
6+
"image": "mcr.microsoft.com/devcontainers/java:21",
77

88
"features": {
99
"ghcr.io/devcontainers/features/java:1": {

‎README.md‎

Lines changed: 80 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -2,78 +2,92 @@
22
[![test](https://github.com/itzg/mc-image-helper/actions/workflows/test.yml/badge.svg)](https://github.com/itzg/mc-image-helper/actions/workflows/test.yml)
33
![LOC](https://img.shields.io/endpoint?url=https%3A%2F%2Fshields-codetab-code-loc-bridge.vercel.app%2Fapi%2Fcodeloc%3Fgithub%3Ditzg%2Fmc-image-helper%26language%3Djava)
44

5-
This tool does the complicated bits for the [itzg/minecraft-server](https://github.com/itzg/docker-minecraft-server) and [itzg/bungeecord](https://github.com/itzg/docker-bungeecord/) Docker images.
5+
This tool does the complicated bits for the [itzg/minecraft-server](https://github.com/itzg/docker-minecraft-server) and [itzg/docker-mc-proxy](https://github.com/itzg/docker-mc-proxy) Docker images.
66

77
## Usage
88

99
> **NOTE** The following documentation may not always be up-to-date. Please be sure to use `-h` or `--help` after any subcommand to view the current usage.
1010
1111
```
12-
Usage: mc-image-helper [-hs] [--debug] [COMMAND]
13-
--debug Enable debug output. Can also set environment variable
14-
DEBUG_HELPER
15-
-h, --help Show this usage and exit
16-
-s, --silent Don't output logs even if there's an error
12+
Usage: mc-image-helper [-hsV] [--debug | --logging=<loggingLevel>] [COMMAND]
13+
--debug Enable debug output. Can also set environment variables
14+
DEBUG_HELPER or DEBUG
15+
-h, --help Show this usage and exit
16+
--logging=<loggingLevel>
17+
Set logging to specific level.
18+
Valid values:
19+
-s, --silent Don't output logs even if there's an error
20+
-V, --version
1721
Commands:
18-
asciify Converts UTF-8 on stdin to ASCII by escaping
19-
Unicode characters
20-
assert Provides assertion operators for verifying
21-
container setup
22-
compare-versions Used for shell scripting, exits with success(0)
23-
when comparison is satisfied or 1 when not
24-
curseforge-files Download and manage individual mod/plugin files
25-
from CurseForge
26-
find Specialized replacement for GNU's find
27-
get Download a file
22+
asciify Converts UTF-8 on stdin to ASCII by escaping
23+
Unicode characters
24+
assert Provides assertion operators for verifying
25+
container setup
26+
compare-versions Used for shell scripting, exits with success
27+
(0) when comparison is satisfied or 1 when
28+
not
29+
curseforge-files Download and manage individual mod/plugin
30+
files from CurseForge
31+
find Specialized replacement for GNU's find
32+
get Download a file
2833
github
29-
hash Outputs an MD5 hash of the standard input
30-
install-curseforge Downloads, installs, and upgrades CurseForge
31-
modpacks
32-
install-fabric-loader Provides a few ways to obtain a Fabric loader with
33-
simple cleanup of previous loader instances
34-
install-forge Downloads and installs a requested version of Forge
35-
install-modrinth-modpack Supports installation of Modrinth modpacks along
36-
with the associated mod loader
37-
install-neoforge Downloads and installs a requested version of
38-
NeoForge
39-
install-paper Installs selected PaperMC
40-
install-purpur Downloads latest or selected version of Purpur
41-
install-quilt Installs Quilt mod loader
42-
interpolate Interpolates existing files in one or more
43-
directories
44-
java-release Outputs the Java release number, such as 8, 11, 17
34+
hash Outputs an MD5 hash of the standard input
35+
install-curseforge Downloads, installs, and upgrades CurseForge
36+
modpacks
37+
install-fabric-loader Provides a few ways to obtain a Fabric loader
38+
with simple cleanup of previous loader
39+
instances
40+
install-forge Downloads and installs a requested version of
41+
Forge
42+
install-modrinth-modpack Supports installation of Modrinth modpacks
43+
along with the associated mod loader
44+
install-neoforge Downloads and installs a requested version of
45+
NeoForge
46+
install-paper Installs selected PaperMC
47+
install-purpur Downloads latest or selected version of Purpur
48+
install-quilt Installs Quilt mod loader
49+
interpolate Interpolates existing files in one or more
50+
directories
51+
java-release Outputs the Java release number, such as 8,
52+
11, 17
4553
manage-users
46-
maven-download Downloads a maven artifact from a Maven repository
47-
modrinth Automates downloading of modrinth resources
48-
mcopy Multi-source file copy operation with with managed
49-
cleanup. Supports auto-detected sourcing from
50-
file list, directories, and URLs
51-
network-interfaces Provides simple operations to list network
52-
interface names and check existence
53-
patch Patches one or more existing files using JSON path
54-
based operations
55-
Supports the file formats:
56-
- JSON
57-
- JSON5
58-
- Yaml
59-
- TOML, but processed output is not pretty
60-
resolve-minecraft-version Resolves and validate latest, snapshot, and
61-
specific versions
62-
set-properties Maps environment variables to a properties file
63-
show-all-subcommand-usage Renders all of the subcommand usage as markdown
64-
sections for README
65-
sync Synchronizes the contents of one directory to
66-
another.
67-
sync-and-interpolate Synchronizes the contents of one directory to
68-
another with conditional variable interpolation.
54+
maven-download Downloads a maven artifact from a Maven
55+
repository
56+
modrinth Automates downloading of modrinth resources
57+
mcopy Multi-source file copy operation with with
58+
managed cleanup. Supports auto-detected
59+
sourcing from file list, directories, and
60+
URLs
61+
network-interfaces Provides simple operations to list network
62+
interface names and check existence
63+
patch Patches one or more existing files using JSON
64+
path based operations
65+
Supports the file formats:
66+
- JSON
67+
- JSON5
68+
- Yaml
69+
- TOML, but processed output is not pretty
70+
resolve-minecraft-version Resolves and validate latest, snapshot, and
71+
specific versions
72+
set-properties Maps environment variables to a properties
73+
file
74+
show-all-subcommand-usage Renders all of the subcommand usage as
75+
markdown sections for README
76+
sync Synchronizes the contents of one directory to
77+
another.
78+
sync-and-interpolate Synchronizes the contents of one directory to
79+
another with conditional variable
80+
interpolation.
6981
test-logging-levels
70-
toml-path Extracts a path from a TOML file using json-path
71-
syntax
72-
yaml-path Extracts a path from a YAML file using json-path
73-
syntax
74-
vanillatweaks Downloads Vanilla Tweaks resource packs, data
75-
packs, or crafting tweaks given a share code or
76-
pack file
82+
toml-path Extracts a path from a TOML file using
83+
json-path syntax
84+
vanillatweaks Downloads Vanilla Tweaks resource packs, data
85+
packs, or crafting tweaks given a share
86+
code or pack file
87+
version-from-modrinth-projects Finds a compatible Minecraft version across
88+
given Modrinth projects
89+
yaml-path Extracts a path from a YAML file using
90+
json-path syntax
7791
```
7892

7993
For [patch](#patch) command [see below](#patch-schemas) for a description of [PatchSet](#patchset) and [PatchDefinition](#patchdefinition) JSON schemas.
@@ -851,7 +865,7 @@ Downloads a maven artifact from a Maven repository
851865
### mcopy
852866

853867
```
854-
Usage: mc-image-helper mcopy [-hz] [--file-is-listing]
868+
Usage: mc-image-helper mcopy [-hz] [--file-is-listing]
855869
[--ignore-missing-sources] [--quiet-when-skipped]
856870
[--skip-existing] [--glob=GLOB]
857871
[--scope=<manifestId>] --to=<dest> SRC[,
@@ -860,6 +874,8 @@ Multi-source file copy operation with with managed cleanup. Supports
860874
auto-detected sourcing from file list, directories, and URLs
861875
SRC[,|<nl>SRC...]... Any mix of source file, directory, or URLs.
862876
Can be optionally comma or newline separated.
877+
Per-file destinations can be assigned by
878+
destination<source
863879
--file-is-listing Source files or URLs are processed as a line
864880
delimited list of sources.
865881
For remote listing files, the contents must all be
@@ -879,6 +895,7 @@ auto-detected sourcing from file list, directories, and URLs
879895
--to, --output-directory=<dest>
880896
881897
-z, --skip-up-to-date
898+
882899
```
883900

884901
### modrinth

‎src/main/java/me/itzg/helpers/sync/MulitCopyCommand.java‎

Lines changed: 59 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
import me.itzg.helpers.errors.GenericException;
1818
import me.itzg.helpers.errors.InvalidParameterException;
1919
import me.itzg.helpers.files.Manifests;
20+
import me.itzg.helpers.files.ReactiveFileUtils;
2021
import me.itzg.helpers.http.FailedRequestException;
2122
import me.itzg.helpers.http.Fetch;
2223
import me.itzg.helpers.http.SharedFetch;
@@ -74,18 +75,18 @@ public class MulitCopyCommand implements Callable<Integer> {
7475
paramLabel = "SRC",
7576
description = "Any mix of source file, directory, or URLs."
7677
+ "%nCan be optionally comma or newline separated."
78+
+ "%nPer-file destinations can be assigned by destination<source"
7779
)
7880
List<String> sources;
7981

82+
private final static String destinationDelimiter = "<";
83+
8084
@Override
8185
public Integer call() throws Exception {
82-
83-
Files.createDirectories(dest);
84-
8586
Flux.fromIterable(sources)
8687
.map(String::trim)
8788
.filter(s -> !s.isEmpty())
88-
.flatMap(source -> processSource(source, fileIsListingOption))
89+
.flatMap(source -> processSource(source, fileIsListingOption, dest))
8990
.collectList()
9091
.flatMap(this::cleanupAndSaveManifest)
9192
.block();
@@ -113,24 +114,55 @@ private Mono<?> cleanupAndSaveManifest(List<Path> paths) {
113114
});
114115
}
115116

116-
private Publisher<Path> processSource(String source, boolean fileIsListing) {
117-
if (Uris.isUri(source)) {
118-
return fileIsListing ? processRemoteListingFile(source) : processRemoteSource(source);
119-
} else {
120-
final Path path = Paths.get(source);
121-
if (!Files.exists(path)) {
122-
throw new GenericException(String.format("Source file '%s' does not exist", source));
123-
}
117+
private Publisher<Path> processSource(String source, boolean fileIsListing, Path parentDestination) {
118+
final Path destination;
119+
final String resolvedSource;
124120

125-
if (Files.isDirectory(path)) {
126-
return processDirectory(path);
121+
final int delimiterPos = source.indexOf(destinationDelimiter);
122+
if (delimiterPos > 0) {
123+
destination = parentDestination.resolve(Paths.get(source.substring(0, delimiterPos)));
124+
resolvedSource = source.substring(delimiterPos + 1);
125+
}
126+
else {
127+
destination = parentDestination;
128+
resolvedSource = source;
129+
}
130+
131+
if (fileIsListing) {
132+
if (Uris.isUri(resolvedSource)) {
133+
return processRemoteListingFile(resolvedSource, destination);
127134
} else {
128-
return fileIsListing ? processListingFile(path) : processFile(path);
135+
final Path path = Paths.get(resolvedSource);
136+
if (Files.isDirectory(path)) {
137+
throw new GenericException(String.format("Specified listing file '%s' is a directory", resolvedSource));
138+
}
139+
if (!Files.exists(path)) {
140+
throw new GenericException(String.format("Source file '%s' does not exist", resolvedSource));
141+
}
142+
return processListingFile(path, destination);
129143
}
130144
}
145+
146+
return ReactiveFileUtils.createDirectories(destination)
147+
.flatMapMany(ignored -> {
148+
if (Uris.isUri(resolvedSource)) {
149+
return processRemoteSource(resolvedSource, destination);
150+
} else {
151+
final Path path = Paths.get(resolvedSource);
152+
if (!Files.exists(path)) {
153+
return Mono.error(new GenericException(String.format("Source file '%s' does not exist", resolvedSource)));
154+
}
155+
156+
if (Files.isDirectory(path)) {
157+
return processDirectory(path, destination);
158+
} else {
159+
return processFile(path, destination);
160+
}
161+
}
162+
});
131163
}
132164

133-
private Flux<Path> processListingFile(Path listingFile) {
165+
private Flux<Path> processListingFile(Path listingFile, Path destination) {
134166
return Mono.just(listingFile)
135167
.publishOn(Schedulers.boundedElastic())
136168
.flatMapMany(path -> {
@@ -141,22 +173,23 @@ private Flux<Path> processListingFile(Path listingFile) {
141173
.filter(this::isListingLine)
142174
.flatMap(src -> processSource(src,
143175
// avoid recursive file-listing processing
144-
false));
176+
false,
177+
destination));
145178
} catch (IOException e) {
146179
return Mono.error(new GenericException("Failed to read file listing from " + path));
147180
}
148181
});
149182
}
150183

151-
private Mono<Path> processFile(Path source) {
184+
private Mono<Path> processFile(Path source, Path destination) {
152185

153186
return Mono.just(source)
154187
.publishOn(Schedulers.boundedElastic())
155-
.map(path -> processFileImmediate(source, dest));
188+
.map(path -> processFileImmediate(source, destination));
156189
}
157190

158191
/**
159-
* Non-mono version of {@link #processFile(Path)}
192+
* Non-mono version of {@link #processFile(Path, Path)}
160193
*
161194
* @param scopedDest allows for sub-directory destinations
162195
*/
@@ -203,7 +236,7 @@ private Path processFileImmediate(Path source, Path scopedDest) {
203236
return destFile;
204237
}
205238

206-
private Flux<Path> processDirectory(Path srcDir) {
239+
private Flux<Path> processDirectory(Path srcDir, Path destination) {
207240
return Mono.just(srcDir)
208241
.publishOn(Schedulers.boundedElastic())
209242
.flatMapMany(path -> {
@@ -220,7 +253,7 @@ private Flux<Path> processDirectory(Path srcDir) {
220253
try (DirectoryStream<Path> files = Files.newDirectoryStream(srcDir, fileGlob)) {
221254
for (final Path file : files) {
222255
//noinspection BlockingMethodInNonBlockingContext because IntelliJ is confused
223-
results.add(processFileImmediate(file, dest));
256+
results.add(processFileImmediate(file, destination));
224257
}
225258
}
226259
return Flux.fromIterable(results);
@@ -230,10 +263,10 @@ private Flux<Path> processDirectory(Path srcDir) {
230263
});
231264
}
232265

233-
private Mono<Path> processRemoteSource(String source) {
266+
private Mono<Path> processRemoteSource(String source, Path destination) {
234267
return Fetch.fetch(URI.create(source))
235268
.userAgentCommand("mcopy")
236-
.toDirectory(dest)
269+
.toDirectory(destination)
237270
.skipUpToDate(skipUpToDate)
238271
.skipExisting(skipExisting)
239272
.handleDownloaded((downloaded, uri, size) ->
@@ -263,7 +296,7 @@ private Mono<Path> processRemoteSource(String source) {
263296
.checkpoint("Retrieving " + source, true);
264297
}
265298

266-
private Flux<Path> processRemoteListingFile(String source) {
299+
private Flux<Path> processRemoteListingFile(String source, Path destination) {
267300
@SuppressWarnings("resource") // closed on terminate
268301
SharedFetch sharedFetch = Fetch.sharedFetch("mcopy", SharedFetch.Options.builder().build());
269302
return Mono.just(source)
@@ -273,7 +306,7 @@ private Flux<Path> processRemoteListingFile(String source) {
273306
.flatMapMany(content -> Flux.just(content.split("\\r?\\n")))
274307
.filter(this::isListingLine)
275308
)
276-
.flatMap(this::processRemoteSource)
309+
.flatMap(url -> processSource(url, false, destination))
277310
.doOnTerminate(sharedFetch::close)
278311
.checkpoint("Processing remote listing at " + source, true);
279312
}

0 commit comments

Comments
 (0)