Skip to content

[investigation] possible load optimization: use Seekable/Parallel reading #36

@sszuev

Description

@sszuev

In case the serialization format (lang) is not provided we have to iterate all supported formats to find the best suitable.
This peace of code can be optimized using file rewinding or parallel running.
To compare:

Standard sequence reading

    private static void loadStd(Path p, int n, long lines) throws IOException {
        int step = n / 100;
        for (int i = 0; i < n; i++) {
            if (i % step == 0) {
                System.out.print("#");
            }
            try (BufferedReader r = Files.newBufferedReader(p)) {
                if (lines != r.lines().count()) {
                    throw new IllegalStateException();
                }
            }
        }
        System.out.println();
    }

vs SeekableByteChannel reading

    private static void loadNio(Path p, int n, long lines) throws IOException {
        int step = n / 100;
        try (SeekableByteChannel chanel = Files.newByteChannel(p, StandardOpenOption.READ)) {
            for (int i = 0; i < n; i++) {
                if (i % step == 0) {
                    System.out.print("#");
                }
                BufferedReader r = new BufferedReader(Channels.newReader(chanel, StandardCharsets.UTF_8));
                if (lines != r.lines().count()) {
                    throw new IllegalStateException();
                }
                chanel.position(0);
            }
        }
        System.out.println();
    }

vs concurrent reading

    private static void loadParallel(Path p, int n, long lines) throws ExecutionException, InterruptedException {
        System.out.println("Go");
        ExecutorService service = Executors.newWorkStealingPool();
        CompletableFuture<?>[] futures = IntStream.range(0, n).mapToObj(x -> CompletableFuture.runAsync(() -> {
            try (BufferedReader r = Files.newBufferedReader(p)) {
                if (lines != r.lines().count()) {
                    throw new IllegalStateException();
                }
                System.out.println(Thread.currentThread() + " is done.");
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
        }, service)).toArray(CompletableFuture[]::new);
        CompletableFuture.allOf(futures).get();
        System.out.println("Done");
    }

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions