Read, write to "stream of files" to support gulp.js

[KostaCitrix]

Hi!

tl;dr: Request to make r.js work on a stream of files instead of using the node file API directly.

gulp.js is a js task runner that makes it easy to chain a multi-step build process together, e.g.

gulp.task("default", function () {
    return gulp.src("app/**/*.js")
        .pipe(jsLint())
        .pipe(stripComments())
        .pipe(rjsOptimize())
        .pipe(filter("main.js", "lib.js", "extras.js"))
        .pipe(minify())
        .pipe(addCopyrightHeaders)
        .pipe(gulp.dest('dist/'));
});

Technically, gulp.src() returns an object stream of Vinyl files that can be piped to plugins. A 'plugin' here is e.g. jsLint(), stripComments() etc.

There's about a dozen or so r.js plugins for gulp - but none of them worked for me. The reason is that r.js is can neither receive a "stream of files" nor write one. It directly operates on "native" node files. So to work with gulp you either need to scratch the whole "streams" idea (which is the whole point of gulp.js) or do something awkward like:

(This doesn't reall work this way, as a stream is not a promise, but you get the idea)

gulp.task("default", function () {
    return gulp.src("app/**/*.js")
        .pipe(jsLint())
        .pipe(stripComments())
        .pipe(gulp.dest('build/rjs-input')).then(function () {
            return rjsOptimize({
                baseDir: "build/rjs-input",
                dir: "build/rjs-output"
            });
        }).then(function () {
            gulp.src("build/rjs-output/**/*.js").
                .pipe(filter("main.js", "lib.js", "extras.js"))
                .pipe(minify())
                .pipe(addCopyrightHeaders)
                .pipe(gulp.dest('dist/'));
        });
});

Which makes it a bit awkward, you see?

Would you consider implementing / merging a patch to r.js that allows r.js to work on streams?

Thanks in advance for your reply!

Cheers,
Kosta

[jrburke]

I am a bit unsure how this would work in practice, maybe you can help me better understand.

For me, the stream of files seems like a linear thing, where the optimizer wants to trace dependencies in a tree like structure, so the order the optimizer wants to trace will likely be different than the stream order.

Or is it enough that r.js just accepts the files via a stream, stores them all in memory, then does the tracing operation, and r.js would know that the input stream ends at some point by some signal in the stream?

That seems to hurt memory consumption, as r.js would be keeping the whole set of files in memory before starting the tracing and ordering for the output. So I expect I would get bugs about it taking up too much memory at some point.

I am very open to ideas in this area, but at the moment do not see how the stream order meshes well with the tree traversal, at least for input. For output, I can see dumping to a stream (possible to dump now to stdout via out=stdout), but the input seems trickier.

[KostaCitrix]

I think it's ok to build up a map of the files in the stream. I mean, you can always look at a directory listing as a "stream of file objects", right? So I don't think it should be that big of a difference.

About the memory consumption: You get a stream of Vinyl files that should have a content property that is a node Buffer with the content of that file. So it already is in memory in this case. And if you have gigabytes of JavaScript sources to process with gulp, I guess you'll notice the too-high memory requirement when gulp reads these, and blame gulp. In other words, it shouldn't be your problem :)

The issue with dumping to stdout etc. is that that only works for a single file, right? It would already be great if you could write a stream of objects* instead of writing to a directory. That would solve 50% of the issue :)

*: For this use cases, those "Vinyl files" are probably best, but just plain objects with a path and content should be fine too. Then it should be trivial to write a thin mapper to Vinyl files.

[hilja]

👍

Btw, is there a way to make the requirejs node package to work with Gulp? Streaming or not. I could of course execute the command line command with gulp-shell.

[KostaCitrix]

@hilja What I currently do: I just run requirejs.optimize({baseUrl: ..., etc}, successCb, failureCb) from the npm package as a gulp task. For that, the gulp task needs to return a promise that needs to be resolved/rejected in the success/failure fallbacks. It's not "gulp-y", but it works for now.

I couldn't get any of the existing gulp.js wrappers to work, except the one that basically just did the above, except "gulp-requirejs-simple", which basically exactly does this: https://github.com/joshje/gulp-requirejs-simple/blob/master/index.js

(sorry to respond in-thread, but I think this relates to this issue)

[mikkotikkanen]

++++++ for this

For the first step, I think it would even be feasible if r.js would just return file streams/buffers (or even use vinyl-fs) instead of writing them to the disk, so you could just have requirejs.optimize() as a first thing in the chain (instead of src()), and just pipe the rest of the gulp plugins after that. It's not exactly the way it should work but at least it's closer.. :)

ie.

requirejs.optimize({appDir: 'js/'}).pipe(uglify()).pipe(gulp.dest('js-dist/'));

'cos the issue now is that you need to write your r.js optimized files into the disk somewhere before you can read them again to memory and continue with the optimization process, which is what makes Gulp so nice and fast as everything just runs in the memory.

[jrburke]

I have created a new project for this kind of use, amodro-trace, and there are specific hooks that can be used to adapt it to use non-file inputs.

Does that tool solve the problem? It is a lower level tool than r.js, but gives more flexibility in how it can be used. I am unlikely to build in specific knowledge of a specific build tool's io mechanism, like vinyl-fs, into r.js, but hopefully these alternative tool options give more flexibility anyway.

[KostaCitrix]

Wow - at first glace this looks really cool! I hope I can look more deeply into it soon.

[wmaca]

@KostaCitrix, as @jrburke told, r.js needs to read all files to determine the dependency tree. If you are just looking for a convinient way of using r.js as a gulp plugin, you can do it like me:

https://gist.github.com/waltermacambira/be0d24901b779b4f1fa7

It will make r.js work as any gulp plugin. It takes a stream, copies them to temporary folder, optimizes with r.js and returns the output as a stream. Hope, I've helped.