[4.2][Idea] Automatically localize CDN assets

[tabacitu]

So there were 2 reasons why we decided to use Laravel Mix and NPM in Backpack 4.0:

1. For people to be able to use Backpack on intranets.
2. It's a generally a good idea to host your own CSS & JS files, instead of relying on a public CDNs. With something like Cloudflare, caching your assets is super-easy, so you get most of the benefits of a CDN, without risking your page stops working because one (of the many) public CDNs you use have a problem.

We maintainers took the burden of using NPM behind the scenes, so that end-developer, Backpack developers, don't have to. We pull all the dependencies, compile them and spit out the public/packages folder, which... is convenient for most developers, but it's a whopping 80 MB. Which currently resides both in public/packages and in vendor/backpack/crud/...etc. This makes it:

• kind of difficult to maintain for us (NPM is hell);
• kind of difficult for developers to host (for example, the /vendor/ folder is too large for it to be used with Laravel Vapor);

So it's an inconvenience. A problem. There's been talk of symlinking the public/packages folder https://github.com/Laravel-Backpack/CRUD/issues/2246#issuecomment-610219098 and that's a solution to bring it down from 160 MB to 80 MB, but... it's still a lot. Because all folders have files that we're not actually using. That's just one of the ways NPM sucks.

---

But. What if there was another way to achieve the same thing? One that didn't involve using NPM? One that was as easy as loading files from CDNs, like we used to do in Backpack v3, but as secure as loading them from a local directory? And it would solve both the maintainer problem and the developer problem? Here's what I've thought of:

Premise:

• assets in Backpack are already loaded using Laravel's asset() helper everywhere;

Solution:

• what if we create a different asset helper, and use that throughout Backpack? Say... cdn_asset();
• the cdn_asset() helper would localize the file (download it from CDN, and store it locally); then every time it's called, it will check the local folder to see if the file is there already; if the file has been localized, it will use the path to the local file, instead of the one to the public CDN;

🤯 💥 🎉 Mind. Blown. Right? This seems such a simple solution, that I don't know why we haven't thought of this before. Why nobody has thought of this before - I looked around, and it doesn't seem like anybody has built anything like this. Which makes me wonder... is this brilliant... or stupid? Am I missing something huge? Something that makes this a no-go? Does anybody see a problem with this?

---

Possible ways to do this:
A) with a command, upon composer update and composer install, it would go through all views (and package views somehow) and localize all assets that are loaded using the cdn_asset() helper;
B) on pageload, the first time the page loads it would load it from CDN, but behind the scenes it would start a process to download the file using wget or something; that should have zero impact to the user; the next time the same page loads, it would see the file is now localized, because it exists in the local folder, and it will use the local path instead of the public CDN path;

I like option B a lot better, if it's technically possible. I'm open to alternative implementations, of course.

---

Either way this would mean:

• replacing all asset loading from NPM to CDN, using cdn_asset(), in all operations, fields, filters, buttons, widgets, etc;
• creating this helper, that actually does the above 😄
• reserving a folder inside public for this kind of assets; we should probably create a symlink (not a folder), something like public/cdn_assets or public/localized_assets that points to storage/app/localized_assets; that's where it would be ok for the system to store the actual files;
• cdn_asset() should probably have a fallback - if the string given is not an actual external URL, it should call asset() on the string; so that you can also use it on local assets and it's not a problem to just find-and-replace all asset() usage with cdn_asset();
• in the spirit of [4.2.x] Fix CSS / JS assets loaded multiple times CRUD#2652 it would probably be cleaner to load CDN and JS assets using two blade directives - @loadCss() and @loadJs() - that would make the views even cleaner, and behind the scenes they would just output the script/link element, with the url handled by cdn_asset();
• new features like @loadOnce should probably use this cdn_asset() too;
• this would basically make it easy for people to use a bunch of CSS/JS files, instead of bundling them; which I still think is better, since HTTP/2 downloads them concurrently; but another considerable improvement would come from inlining small CSS/JS; so maybe we can add another blade directive that would include that CSS/JS file inline, within a script/style block, instead of referring to it; that way, you could use @inlineCss('path/to/css/file') and it would but it within a <style></style> block;
• another performance improvement to loading CSS files could be made by deferring loading CSS that is non-critical using rel="preload" and loadCss; that way, critical CSS is loaded first, then everything else; this could be done using a parameter or something - say @loadCss('path', false) which would basically instruct the package to do rel="preload" on that CSS file;

Using a helper like this would also make it possible to have a config file, so that developer can choose:

• the path inside the public folder where to generate the symlink;
• the path inside the storage folder where to actually store the localized files;
• where the assets should be loaded from (CDN or localized) - maybe you actually prefer loading them from public CDNs, to offload some bandwidth cost;
• query string for cachebusting (defaults to null; put anything else and it'll be slapped at the end of all assets, for example bootstrap.min.js?cbs=8afhu1; easily change that string in the config file to force cachebusting for all assets, and your users will be getting fresh assets;

--

Possible problems:

• (I) When and how do localized files get deleted? If you make a minor change to the path, for example bump the version, how will the system know that they should delete the old file because it's never going to be used again?
  • Maybe we delete the entire folder once in a while? That could work...
  • Maybe we make a command, something like php artisan cdnassets:clear that gets called upon composer update / composer install
• (II) What happens when the developer plugs in a URL to a malicious script? Well then they download the script to their server. Which sucks, but... I don't think this is something we need to worry about, the alternative would have been that they upload that script manually - so it's the exact same thing;

Does anybody see other problems with this? Am I missing something or would this be a really really cool way of never having to deal with NPM ever again? Feedback is very much appreciated - I'm excited about this right now, but it wouldn't be the first brilliant idea that turns out to be stupid 🙄

PS. I use the word "localize" here a lot - and it might be confusing, but it has NOTHING to do with translation - how it's sometimes used. It's about making a file LOCAL instead of EXTERNAL.

[tabacitu]

This should be a separate package. People should be able to use this even if they don't use Backpack - and I think many will. I'm thinking something like backpack/asset-manager or backpack/asset-loader, something like that. Don't have a good name for it really - open to suggestions.

To sum it up, here are all its features:

Easily localize files from public CDNs

• cdn_asset() - alternative to Laravel's default asset() helper, that you can use on public CDN paths too; it will pull the file from public CDN and serve it locally next time, so you have the ease of use of working with public CDN paths, and the benefit of serving your assets yourself; you'll never have to create a public/styles or public/scripts folder to store third-party styles; only your app's custom styles - everything else is handled automatically;

Blade syntax for loading CSS and JS assets (external or internal)

• @loadStyle() - blade directive that spits out a <link rel="stylesheet"> element for that file; localizes files that are served from external paths; aliases: @loadCSS() and @loadCss();
• @loadScript() - blade directive that spits out a <link rel="stylesheet"> element for that file; localizes files that are served from external paths; aliases: @loadJs(), @loadJavaScript() and @loadJavascript();

Easily make sure the same assets are never loaded twice per page - PR Laravel-Backpack/CRUD#2652

• @loadStyleOnce() - makes sure that CSS asset is loaded only once per page;
• @loadScriptOnce() - makes sure that JS asset is loaded only once per page;
• @loadOnce & @endLoadOnce runs that block only once per page;

Improve performance by loading small CSS/JS assets inline

• @inlineStyle() - take that CSS file, localize it if necessary and include it within a <style></style> block using PHP; saves one HTTP Request per file;
• @inlineScript() - take that JS file, localize it if necessary and serve it within a <script></script> block using PHP; saves one HTTP Request per file;

Improve performance by deferring loading non-critical CSS files

• @loadDeferredStyle() - load that CSS file after critical CSS has been loaded, using rel="preload" and polyfill; aliases: @loadNonCriticalStyle(), @deferStyle(), @deferredStyle(), @lazyStyle(), @loadLazyStyle, @lazyLoadStyle, @postponedStyle, @postponeStyle() - not sure about a good name for this one;

[martijnb92]

Sounds like a greate idea. I do however see one problem, the helper needs to be quite "smart". For example, how to deal with CSS files wich use additional files, like some fonts.
The helper needs to download the file from the CDN, check it for additional resources, download those files too and edit the downloaded file with new paths for the resources.

[tabacitu]

Ugh that's right! That's one thing I didn't think of, thank you @martijnb92 !

[eronisko]

Guess I'm kinda late into the story, but here's my view. I think what's being described here is very similar to proxy caching. I don't think it's a bad idea, but I think it comes with an added complexity that might not be obvious at first sight. E.g.

• How will the first request really work? If we redirect to a CDN then we can't really serve clients with no access to outside network. If we wait for the first download, we're introducing considerable latency. What if the server itself has no access to the internet?
• What happens if there are multiple "first" requests for an asset fired at all once? How do we know whether the asset is already being downloaded?
• How do we expire the cache? Do we keep assets for a period of time or do we simply keep everything ever requested?

I'd also be a little hesitant about introducing more non-standard Blade directives. I think this would make contributions to the project a bit harder.

My suggestions

Backpack should only come with assets necessary for its running -- their minimized versions at least.

/vendor/backpack/crud is currently at 80 MB. That's a lot, but many of these 'packages' are not meant to be served as-is. Lot of them come with minimzed versions. E.g. line-awesome alone takes up 15 MB in CRUD, but its dist is just 3.5 MB in size. Backack could do this pre-selection and export only what's required into its own packages/dist directory. Laravel Mix could help make this happen. (BTW large vendor/ is not a blocker for Vapor either)

Serve assets via PHP in dev. Use a symlink/copy in prod.

When in dev, why not simply serve assets through a custom 'assets' controller? That way we don't have to export anything, not even publish defaults into public/. No post-update scripts, or hooks, no duplication in source control. composer update simply updates everything (and compatible versions are kept together!).

Then in prod, I'd suggest doing something like storage:link, e.g. backpack:link for linking from public/backpack/assets to vendor/backpack/.../packages. If linking is not an option, then a simple copy (like it is today) would also be workable.

---

Just for the record, I still think (#2246) that going with the NPM/Laravel Mix flow like laravel/ui is the way to go. I understand why you may want to keep developers "shielded" from npm, but Laravel currently doesn't offer anything better IMHO (not that what it offers is bad -- I actually think the Mix integration is really good.). Beyond minimization, webpack (Mix) offers things like tree shaking or lazy loading which, when used correctly (and with a bit of black magic, admittedly) can make asset serving very efficient.

[tabacitu]

@eronisko - I've let this percolate a long time, just replied with a thumbs up because I agreed with what you said but didn't have a better solution. I left the problem for future-me. And here's future-me, who's had a small epiphany 😄

I think what's being described here is very similar to proxy caching. I don't think it's a bad idea, but I think it comes with an added complexity that might not be obvious at first sight.

Totally agree. So WHAT IF, instead of reinventing the wheel and coding the same logic (in a worse way probably), we use existing infrastructure and services? What if we:

• move all JS and CSS dependencies to Backstrap, which is the HTML theme we use; we can use Laravel Mix there, to compile, minimize, publish dependencies into /dist/ and trim their fat;
• then, since Backstrap is on NPM, we can use https://unpkg.com/ to load ANY asset we have in /dist/ from a CDN; and not any CDN, the reliable CloudFlare, which powers 15% of all requests from Fortune 1000 companies; this already works, you could use https://unpkg.com/@digitallyhappy/[email protected]/dist/vendors/jquery/js/jquery.min.js right now to load the exact jquery file that Backstrap provides in its dist folder; and we can do that to ALL our dependencies, like Bootstrap, datepickers, ckeditor, datatables; we move them to Backstrap instead of Backpack - which arguably should have been done a long time ago, to separate front-end from back-end;
• but not only that, unpkg.com also allows us to use semver ranges and tags in the URL; so we don't need to require a specific version of Backstrap, we can use something like https://unpkg.com/@digitallyhappy/backstrap@^0.3/dist/vendors/jquery/js/jquery.min.js (notice the ^ in the version number), which will allow us Backpack maintainers to update the dependencies inside Backstrap, release a patch version, and then Backpack developers (and Backpack end-users) will be getting the updated assets without doing absolutely anything; in this crazy world, where a JS dependency vulnerability is discovered every freakin' week, this would be a HUGE deal, then Backpack users can rest assured that they're using the latest JS and CSS assets that are compatible with their install; as soon as the cache expires, of course;
• additionally, we could allow developers the option to use local assets instead of cdn, but in a manual, not automatic way, by changing a config value:

[ 
    // inside config/backpack/base.php we'd have
    'asset_path' => 'https://unpkg.com/@digitallyhappy/backstrap@^0.3/dist/',

     // but that can easily be changed into 
    'asset_path' => 'public/assets/backstrap/',
]

• we'd just need to include instructions for people who want to install Backstrap using NPM and publish the /dist/ folder using their Laravel Mix; this would be a lot easier to do, because all assets are already inside the Backstrap repo, so it would only be one package to install and publish; one directory for all Backstrap assets;
• the config variable above could be used in PHP in a number of ways:

// using Laravel's default asset helper
asset(config('backpack.base.asset_path').'vendors/jquery/js/jquery.min.js');

// using a custom asset helper for Backpack, that just calls the above behind the scene
backpack_asset('vendors/jquery/js/jquery.min.js');

// or shorter
bp_asset('vendors/jquery/js/jquery.min.js');

// or maybe the correct word would actually be Backstrap, not Backpack
backstrap_asset('vendors/jquery/js/jquery.min.js');

---

This would:

• decrease the size of the Backpack repo (down to <10MB probably);
  • fixes #2118, you can do Backpack+Vapor with no workarounds;
  • fixes #2958, no more composer memory issues when installing Backpack;
• decrease the size of any project that uses Backpack (by about 160MB probably, since the JS and CSS files will be nowhere, not in vendor and not in public - most projects should be perfectly fine loading all assets from CDN);
• remove the need to publish the assets for point versions (fixes Laravel-Backpack/CRUD#2246); Backpack developers get up-to-date assets as soon as we launch a patch version of Backstrap; we do npm update, npm audit fix and all that, they do nothing and get the updated versions once we make sure everything's working fine;
• force us to keep all front-end JS/CSS assets in one place, one repo (Backstrap);
• provide the developer an easy way to load Backpack assets from CDN (or local);
• provide the developer an easy way to load Backpack assets locally, if needed (on intranets);

However, this is NOT a 100% solution for my first proposal. Because it wouldn't fall back automatically from CDN assets to local assets. The developer has to choose between serving assets locally or from CDN, with the default being from CDN.

What do you think @eronisko ? @martijnb92 ? @pxpm ?

[tabacitu]

Alternatively, if https://unpkg.com uptime guarantee is an issue, we could probably self-host unpkg or use something like https://www.skypack.dev/ for the uptime guarantee... But I would really prefer to pay for a service, rather than get into hosting...

[eronisko]

I think compiling the assets into a Webpack bundle is a step in the right direction. My suggestion would be serving assets locally by default and having the CDN delivery as an opt-in option. Purely because it seems to be a common pattern in the Laravel world.

In short, for me as a developer, the Telescope and frontend instructions make sense. I would like Backstrap installation instructions to be similar (which they are at the moment).

One other thought:

unpkg.com also allows us to use semver ranges and tags in the URL; so we don't need to require a specific version of Backstrap

I can see the appeal, but I'd be careful about encouraging this. The JavaScript world is broad and wild and semantic versioning is not always a guarantee of compatibility :) My preference is the exact opposite: specific version numbers for better reproducibility.

[martijnb92]

@tabacitu - Your reply got me thinking, I really like the idea but it could probably be done even simpler.
In your proposed solution you would still need to actively update the Backstrap package with all the assets whenever one has been updated. For example jQuery, why not just grab it straight from the source?

---

I made this example helper function which returns an asset by a "key", incase someone wants to use their own asset they can easily overwrite it via the base config file.

 // Overwrite Backpack assets with your own version
    'overwrite_assets' =>[
        'jquery' => '/jquery.min.js'
    ],

<?php
function backpack_asset($asset){

    //Create the overwrite asset array from the config file.
    $overwrite_asset = config("backpack.base.overwrite_asset", []);

    // Partial example list with some assets, might need a better location. (datatables assets are already combined into one instead of multiple)
    $asset_list = [
        "momentjs" => "//cdnjs.cloudflare.com/ajax/libs/moment.js/2.27.0/moment.min.js",
        "jquery" => "//code.jquery.com/jquery-3.5.1.min.js",
        "datatables.net" => "//cdn.datatables.net/v/bs4/dt-1.10.21/fh-3.1.7/r-2.2.5/datatables.min.js",
        "datatables.net-css"=> "//cdn.datatables.net/v/bs4/dt-1.10.21/fh-3.1.7/r-2.2.5/datatables.min.css",
    ];

    //Merge the overwrite assets, in case someone depends on custom assets they probably have and advanced use case.
    //This allows them to define a custom path.
    $asset_list = array_replace($asset_list,$overwrite_asset);

    //Check if the requested asset exists otherwise return the supplied variable (complete urls e.g.)
    return (array_key_exists($asset,$asset_list)) ? $asset_list[$asset] : $asset;

}
?>

---

This:
<script type="text/javascript" src="{{ asset('packages/moment/min/moment.min.js') }}"></script>
Would become:
<script type="text/javascript" src="{{ asset(backpack_asset('momentjs')) }}"></script>

And this:

  <script type="text/javascript" src="{{ asset('packages/datatables.net/js/jquery.dataTables.min.js') }}"></script>
  <script type="text/javascript" src="{{ asset('packages/datatables.net-bs4/js/dataTables.bootstrap4.min.js') }}"></script>
  <script type="text/javascript" src="{{ asset('packages/datatables.net-responsive/js/dataTables.responsive.min.js') }}"></script>
  <script type="text/javascript" src="{{ asset('packages/datatables.net-responsive-bs4/js/responsive.bootstrap4.min.js') }}"></script>
  <script type="text/javascript" src="{{ asset('packages/datatables.net-fixedheader/js/dataTables.fixedHeader.min.js') }}"></script>
  <script type="text/javascript" src="{{ asset('packages/datatables.net-fixedheader-bs4/js/fixedHeader.bootstrap4.min.js') }}"></script>

Would become:
<script type="text/javascript" src="{{ asset(backpack_asset('datatables.net')) }}"></script>

More examples:
With the example overwrite present
backpack_asset('jquery') would return /jquery.min.js
backpack_asset('momentjs') would return //cdnjs.cloudflare.com/ajax/libs/moment.js/2.27.0/moment.min.js
backpack_asset('example.com/script.js') would return example.com/script.js

This will be usefull for both JS and CSS files, we might need come up with a efficient naming scheme.

---

This would:

• Decrease the package size
• Easy way to load from CDN
• Easy way to overwrite with local files when needed
• Assets will be updated via the composer update command since the $asset_list is in Backpacks control
• Custom overwrites will always overwrite
• People which use the custom overwrite probably have an advanced use case, this gives them additional control over paths and versions.
• Some packages can be served from a local source, some can be served from a CDN

This would not:

• Automatic fallback to either local or CDN

---

• Probably not all packages will be avaiable on a CDN, so they still need to be published on unpkg.com via the Backstrap package as @tabacitu described.
• Might take some time to create the $asset_list (quite a few assets), but once that's done it's easy.
• Instruction need to be created for people who want all assets local, should be easy.

[tabacitu]

@martijnb92 can I just say that I love this... So simple yet so comprehensive. Then the CDN list is in our control, and the LOCAL list is in the developer control. Which is perfect. Thanks!

[pxpm]

Is this https://github.com/Laravel-Backpack/basset ? 👀