[feature/pcie] feat: add swiotlb=noforce to default kernel command line

[Manciukic]

Changes

Add iommu=off to the default command line to disable SWIOTLB.

Reason

The kernel enables SWIOTLB when the physical address range exceeds 32b, which ends up wasting 64MB of the VM memory, which translates to higher RSS.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

• I have read and understand CONTRIBUTING.md.
• I have run tools/devtool checkbuild --all to verify that the PR passes
  build checks on all supported architectures.
• I have run tools/devtool checkstyle to verify that the PR passes the
  automated style checks.
• I have described what is done in these changes, why they are needed, and
  how they are solving the problem in a clear and encompassing way.
• I have updated any relevant documentation (both in code and in the docs)
  in the PR.
• I have mentioned all user-facing changes in CHANGELOG.md.
• If a specific issue led to this PR, this PR closes the issue.
• When making API changes, I have followed the
  Runbook for Firecracker API changes.
• I have tested all new and changed functionalities in unit tests and/or
  integration tests.
• I have linked an issue to every new TODO.

---

• This functionality cannot be added in rust-vmm.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.36%. Comparing base (411ecd1) to head (9b892fb).
⚠️ Report is 1 commits behind head on feature/pcie.

Additional details and impacted files

@@              Coverage Diff              @@
##           feature/pcie    #5336   +/-   ##
=============================================
  Coverage         80.36%   80.36%           
=============================================
  Files               265      265           
  Lines             30828    30828           
=============================================
  Hits              24775    24775           
  Misses             6053     6053           

Flag	Coverage Δ
5.10-c5n.metal	80.12% <ø> (-0.01%)	⬇️
5.10-m5n.metal	80.12% <ø> (ø)
5.10-m6a.metal	79.32% <ø> (-0.01%)	⬇️
5.10-m6g.metal	76.64% <ø> (ø)
5.10-m6i.metal	80.11% <ø> (-0.01%)	⬇️
5.10-m7a.metal-48xl	79.31% <ø> (-0.01%)	⬇️
5.10-m7g.metal	76.64% <ø> (ø)
5.10-m7i.metal-24xl	80.08% <ø> (-0.01%)	⬇️
5.10-m7i.metal-48xl	80.09% <ø> (ø)
5.10-m8g.metal-24xl	76.64% <ø> (ø)
5.10-m8g.metal-48xl	76.64% <ø> (-0.01%)	⬇️
6.1-c5n.metal	80.16% <ø> (ø)
6.1-m5n.metal	80.16% <ø> (+<0.01%)	⬆️
6.1-m6a.metal	79.37% <ø> (ø)
6.1-m6g.metal	76.64% <ø> (ø)
6.1-m6i.metal	80.16% <ø> (-0.01%)	⬇️
6.1-m7a.metal-48xl	79.36% <ø> (+<0.01%)	⬆️
6.1-m7g.metal	76.64% <ø> (ø)
6.1-m7i.metal-24xl	80.17% <ø> (-0.01%)	⬇️
6.1-m7i.metal-48xl	80.17% <ø> (+<0.01%)	⬆️
6.1-m8g.metal-24xl	76.64% <ø> (ø)
6.1-m8g.metal-48xl	76.64% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[roypat]

Since the problem is specifically swiotlb, maybe swiotlb=noforce would be more appropriate? :o

[Manciukic]

Since the problem is specifically swiotlb, maybe swiotlb=noforce would be more appropriate? :o

All of the mentions I saw in the kernel are about iommu=off. For example:
https://elixir.bootlin.com/linux/v5.10.240/source/arch/x86/include/asm/iommu_table.h#L7-L32

 * *: if SWIOTLB detected 'iommu=soft'/'swiotlb=force' it would skip
 * over the rest of IOMMUs and unconditionally initialize the SWIOTLB.
 * Also it would surreptitiously initialize set the swiotlb=1 if there were
 * more than 4GB and if the user did not pass in 'iommu=off'. The swiotlb
 * flag would be turned off by all IOMMUs except the Calgary one.

Also, the definition of swiotlb=noforce didn't look too promising ("for debugging") ahah

        swiotlb=        [ARM,IA-64,PPC,MIPS,X86]
                        Format: { <int> | force | noforce }
                        <int> -- Number of I/O TLB slabs
                        force -- force using of bounce buffers even if they
                                 wouldn't be automatically used by the kernel
                        noforce -- Never use bounce buffers (for debugging)

[Manciukic]

I just noticed from the A/B test run that ARM didn't improve. I may have to look better at it (and maybe use swiotlb=noforce?)

[Manciukic]

iommu= is an x86 only option, so we need to use swiotlb=noforce. I shouldn't have doubted Patrick.