Configurable rules for exclusions in bundling of python_modules

[dom96]

Implements a new python_modules_excludes wrangler config field. The user can specify a list of globs in that field to exclude them from being vendored. By default every file in python_modules is included but for certain packages they contain so much stuff that sometimes it's good to be able to filter it out, especially since our bundle sizes are limited.

---

• Tests
  • Tests included
  • Tests not necessary because:
• Public documentation
  • Cloudflare docs PR(s): Adds docs for python_modules_excludes. cloudflare-docs#25725
  • Documentation not necessary because:
• Wrangler V3 Backport
  • Wrangler PR:
  • Not necessary because: new feature

---

Test Plan

pnpm run test --filter wrangler -- -t "should print vendor modules correctly in table" src/__tests__/deploy.test.ts

cd packages/wrangler/
pnpm run build
pnpm test:e2e --run --testNamePattern="can exclude vendored module during"

[changeset-bot]

🦋 Changeset detected

Latest commit: 93c0915

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@10927

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@10927

miniflare

npm i https://pkg.pr.new/miniflare@10927

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@10927

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@10927

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@10927

@cloudflare/vitest-pool-workers

npm i https://pkg.pr.new/@cloudflare/vitest-pool-workers@10927

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@10927

wrangler

npm i https://pkg.pr.new/wrangler@10927

commit: 93c0915

[hoodmane]

Looks very reasonable to me. Thanks @dom96!

[vicb]

I am chiming in there because rules / find_additional_modules is something that is very hard to grasp for the users (and for me TBH).

I'm afraid that adding one more config that is Python specific would make things even more confusing.

Because we already have rules, couldn't we have a concept of exlcude rules, i.e. adding {exclude: true} and it would apply for any language?

[dom96]

That's a fair point and I agree that all these options are challenging to understand.

I'd love to reuse find_additional_modules+rules here, but the problem is that the python_modules directory is outside the base_dir. Typically base_dir is set to ./src because that's where the source code lives. Whereas python_modules is at ./python_modules, so rules doesn't apply to it. Indeed, we have our own custom logic to include these files in findAdditionalModules.

Our pywrangler tool vendors the packages defined in a Worker's pyproject.toml file and it always creates the python_modules directory alongside pyproject.toml (and wrangler.toml/.jsonc). I suppose we could read the wrangler config file to grab its main and/or base_dir, then put the python_modules in the base_dir and treat these modules just like any other additional modules. We'd still need to define a default list of rules to add these modules, and then I think the problem would be that the user might accidentally override them without intending to when providing their own modules.

These modules are in some ways special, as they should always be implicitly included when present and wrangler should avoid printing out the full list of them (we have code that condenses these in the "Attaching additional modules" output).

So I think from that perspective allowing users to exclude certain modules explicitly makes sense. But maybe you have another idea of what might work better?

[dom96]

Another alternative is to do what was implemented here #10929, though that behaviour is a lot more subtle

[jamesopstad]

Discussed this with @petebacondarwin and we would prefer that Python specific configuration has its own section in the config rather than being mixed together with other top-level properties. How about the following?

{
  "python": {
    "include": [],
    "exclude": []
  }
}

You can just have exclude for now if include is not needed. Thanks.

[petebacondarwin]

Discussed this with @petebacondarwin and we would prefer that Python specific configuration has its own section in the config rather than being mixed together with other top-level properties. How about the following?

Or python_modules

[dom96]

Sounds good, I'll add a new python_modules section for this