Suggestion: Improve Alcaeus Web DX by documenting eliminating/reducing Node.js dependencies?

[lambdakris]

While trying to use Alcaeus with Angular (which uses webpack for tooling under the hood) following the Alcaeus getting started instructions, I ran into a couple of issues when trying to run the application through the webpack based tooling.

Repro steps

Here is what I tried:

1. Download, unzip, and open our example Angular App: alcaeus-app.zip (based on the default Angular CLI template)

2. Install Alcaeus

npm install --save alcaeus

3. Import and use Alcaeus from inside a component

import { Component, OnInit } from '@angular/core';
import { Hydra } from 'alcaeus/web';

@Component({
  selector: 'app-root',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.scss']
})
export class AppComponent implements OnInit{
  async ngOnInit(): Promise<void> {
    var hydraRsrc = await Hydra.loadResource('https://sources.wikibus.org/');
    var root = hydraRsrc.representation!.root!;
    console.log(root.toJSON());
  }
}

5. Try to run the application

npm start

6. Notice and troubleshoot problems (see below)

Problems found

Problem 1 - resolving 'stream'

• Problem

./node_modules/jsonld-streaming-parser/lib/JsonLdParser.js:14:17-34 - Error: Module not found: Error: Can't resolve 'stream' in 'C:\Users\Kris.garcia\code\open\sema\alcaeus-examples\getting-started-example\node_modules\jsonld-streaming-parser\lib'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "stream": require.resolve("stream-browserify") }'
        - install 'stream-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "stream": false }

• Solution

* Same solution as Problem 2 & 3

1. Install the node-polyfill-webpack-plugin package

npm install --save-dev node-polyfill-webpack-plugin

2. Configure the node-polyfill-webpack-plugin in the custom-webpack.config.ts file

import { Configuration } from 'webpack';
import NodePolyfillPlugin from 'node-polyfill-webpack-plugin';

export default {
  plugins: [
    new NodePolyfillPlugin()
   ]
} as Configuration;

Problem 2 - resolving 'url'

• Problem

./node_modules/parse-link-header/index.js:4:10-24 - Error: Module not found: Error: Can't resolve 'url' in 'C:\Users\Kris.garcia\code\open\sema\alcaeus-examples\getting-started-example\node_modules\parse-link-header'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "url": require.resolve("url/") }'
        - install 'url'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "url": false }

• Solution

* Same solution as for Problem 1 & 3

1. Install the node-polyfill-webpack-plugin package

npm install --save-dev node-polyfill-webpack-plugin

2. Configure the node-polyfill-webpack-plugin in the custom-webpack.config.ts file

import { Configuration } from 'webpack';
import NodePolyfillPlugin from 'node-polyfill-webpack-plugin';

export default {
  plugins: [
    new NodePolyfillPlugin()
   ]
} as Configuration;

Problem 3 - resolving 'util'

• Problem

./node_modules/rdf-dataset-ext/fromStream.js:5:4-19 - Error: Module not found: Error: Can't resolve 'util' in 'C:\Users\Kris.garcia\code\open\sema\alcaeus-examples\getting-started-example\node_modules\rdf-dataset-ext'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "util": require.resolve("util/") }'
        - install 'util'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "util": false }

• Solution

* Same solution as for Problem 1 & 2

1. Install the node-polyfill-webpack-plugin package

npm install --save-dev node-polyfill-webpack-plugin

2. Configure the node-polyfill-webpack-plugin in the custom-webpack.config.ts file

import { Configuration } from 'webpack';
import NodePolyfillPlugin from 'node-polyfill-webpack-plugin';

export default {
  plugins: [
    new NodePolyfillPlugin()
   ]
} as Configuration;

Problem 4 - finding declaration file for 'clownface'

• Problem

...
Error: ../../../../../Code/open/sema/alcaeus-examples/getting-started-example/node_modules/@tpluscode/rdfine/RdfResource.d.ts:4:56 - error TS7016: Could not find a declaration file for module 'clownface'. 'C:/Users/Kris.garcia/Code/open/sema/alcaeus-examples/getting-started-example/node_modules/clownface/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/clownface` if it exists or add a new declaration (.d.ts) file containing `declare module 'clownface';`

4 import { MultiPointer, GraphPointer, AnyPointer } from 'clownface';
                                                         ~~~~~~~~~~~


Error: ../../../../../Code/open/sema/alcaeus-examples/getting-started-example/node_modules/@tpluscode/rdfine/factory.d.ts:2:40 - error TS7016: Could not find a declaration file for module 'clownface'. 'C:/Users/Kris.garcia/Code/open/sema/alcaeus-examples/getting-started-example/node_modules/clownface/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/clownface` if it exists or add a new declaration (.d.ts) file containing `declare module 'clownface';`

2 import { AnyContext, AnyPointer } from 'clownface';
...

• Solution

1. Install the @types/clownface package

npm install --save-dev @types/clownface

Summary

While we understand that this is not in any way a bug with Alcaeus per se, we would like to see if there is anything that can be done to improve the getting started DX when working with typescript and webpack based tooling like that which Angular uses. We would like to understand whether this means adding details to the documentation or making changes to how Alcaeus itself is packaged or potentially to how certain dependencies of Alcaeus are packaged, etc.. Please let us know if we can provide anymore information or collaboration on this and thanks!

[tpluscode]

Hello @lambdakris. Thank you for sharing your issues and workarounds.

Starting with easy, I could totally add @types/clownface as a dependency to @tpluscode/rdfine.

As for the others, admittedly, I've been kind of stuck with webpack 4. It's really nice of you to share your experience. As you point out, all those issues are caused by native packages being used by alcaeus' dependencies. Some may be possible to fix at the source, but if and when is uncertain at best. For example, parse-link-header does have a PR in progress apparently thlorenz/parse-link-header#24

Hence, I think I would propose to create a page in the documentation which would describe steps needed to work with webpack 5 as you outlined above.

[lambdakris]

Thanks @tpluscode , I was kinda thinking along the same lines. I appreciate you adding @types/clownface to the @tpluscode/rdfine and I would be happy to add our findings to the docs. I read through the PR for parse-link-header and hopefully that will get merged soon-ish.

With regards to adding to the docs, could you provide or point me to some brief directions of how to work with the docs? Can we just make additions/revisions to the docs/latest folder or is there a different workflow that you follow?

[tpluscode]

Yes, I would simply add the necessary new pages to docs/latest and an entry in _sidebar.md to have it in the main menu somewhere. That should be it. And in case you haven't noticed, yarn docs will start it locally for you.

The other dir next to latest are snapshots of some past versions but unless there are significant breaking changes I don't suppose I would add more.

[GwendolenLynch]

Edit: Below is in addition to requiring node-polyfill-webpack-plugin as noted by @lambdakris

Just to add to this for anyone else coming here to get this building under Webpack 5.

The patch on rdefine fixes the need for @types/clownface, but there was some fun and games with rdfjs due to versioning … ESM cutovers … or something … oh my, is it my bed time yet? 😆 💤

Anyway, the remaining bit of the puzzle for me was modifying the project's webpack.config.js file to add the following:

module.exports = {
     // ...
    resolve: {
        // modules = [path.resolve(__dirname, 'node_modules'), 'node_modules'],
        alias: {
            '@rdf-esm/to-ntriples': path.resolve(__dirname, 'node_modules/@rdfjs/to-ntriples/index.js'),
        },
    },
    fallback: {
        'rdf-dataset-ext/addAll': path.resolve(__dirname, 'node_modules/rdf-dataset-ext/addAll.js'),
        'rdf-dataset-ext/deleteMatch': path.resolve(__dirname, 'node_modules/rdf-dataset-ext/deleteMatch.js'),
        'rdf-dataset-ext/fromStream': path.resolve(__dirname, 'node_modules/rdf-dataset-ext/fromStream.js'),
    },
};

YMMV.

[tpluscode]

Hi @GwendolenLynch. I'm surprised with the hoops you had to jump. We have been using alcaeus in a vue3 app that builds with webpack 5. Other the node native stream and buffer, the config is pretty much the defaults.

I'm especially surprised you had to alias and @rdf-esm package. I forked and published the @rdf-esm package specifically as dual ESM/CommonJS package for them to work better in web environment. They are the last I would expect should be giving you trouble. 🤔

And I do not know the fallback setting. What does it do and why was that necessary?

[GwendolenLynch]

OK, under the context of "this is how I understood it" …

• The commented-out resolve.modules for some reason is required in my set-up, but it shouldn't be so it is left commented for someone that understands this better than I do to understand (part of) my thinking
• The resolve.alias and resolve.fallback settings were to workaround — at least in my project — deprecated/archived v0.x / v1.x copies of rdf-esm/(.+) and their merged updated v2.x versions; rdfsj/$1

tl;dr Webpack's module resolution referenced the rdfjs v2 libs, while the files fromalcaeus use import that are referencing rdf-esm/* < v2.x

… at least that is how I understood things 🤷‍♀️