Artifact add optimization on macOS and Windows

[Honny1]

This PR adds a new local artifact add API endpoint and enforces absolute path requirements for local file operations.

Fixes: https://issues.redhat.com/browse/RUN-3385
Fixes: #26321

Benchmark

• Command: podman artifact add quay.io/myartifact/benchmark:latest ./artifacts/random-file
• Artifacts Dir content:

$ du -a -h | sort -hr
5G	.
5.0G ./random-file

Benchmark Results:

Mac OS

• Number of runs: 10
• Machine:
  • Chip: Apple M3 Pro
  • RAM: 36 GB
  • SSD: 1 TB Apple SSD
• Podman machine configuration:
  • CPUs: 8
  • Memory: 8.1 GB
• report

Providers

applehv

• Before: 32.82 s
• After: 5.77 s

libkrun (krunkit 0.2.1)

• Before: 25.87 s
• After: 7.17 s

Windows

• Windows 10
• Number of runs: 10
• Machine:
  • CPU: Intel Core i7-8665U
  • RAM: 32 GB
  • SSD: 500 GB Samsung 970 Evo Plus
• Podman machine configuration:
  • CPUs: 8
  • Memory: 2 GB (WSL) | 8.2 GB (Hyper-v)
• report

Providers

WSL

• Memory: 2 GB
• Before: 35.25 s
• After: 78.31 s

Hyper-V

• Memory: 8.2 GB
• Before: 362.70 s
• After: 51.66 s

Checklist

Ensure you have completed the following checklist for your pull request to be reviewed:

• Certify you wrote the patch or otherwise have the right to pass it on as an open-source patch by signing all
  commits. (git commit -s). (If needed, use git commit -s --amend). The author email must match
  the sign-off email address. See CONTRIBUTING.md
  for more information.
• Referenced issues using Fixes: #00000 in commit message (if applicable)
• Tests have been added/updated (or no tests are needed)
• Documentation has been updated (or no documentation changes are needed)
• All commits pass make validatepr (format/lint checks)
• Release note entered in the section below (or None if no user-facing changes)

Does this PR introduce a user-facing change?

Added new local artifact add API endpoint (`libpod/local/artifacts/add`) for loading artifacts from local files
Image and artifact load endpoints now require absolute paths for local file operations to improve security and prevent path ambiguity

[TomSweeneyRedHat]

Quick breeze, it LGTM, and happy green test buttons to boot!

[baude]

really quick pass LGTM ... hopefully someone can walk through a little slower than today allowed.

[Honny1]

Based on benchmark results, I disabled this improvement for WSL. I rebenchmarked this improvement on Windows 11 and it didn't get better, but overall CPU usage is lower. Let me know if you think disabling this feature for WSL makes sense.

Benchmark report

PTAL: @Luap99 @baude @TomSweeneyRedHat @mheon

[Honny1]

Unrelated CI failure; fixed in #27562

[TomSweeneyRedHat]

LGTM
but I'd like a head nod or two from folks more familiar with macOS and Windows than I.

[Honny1]

PTAL @l0rd

[l0rd]

@Honny1 I have tested with hyper-v but I cannot see any difference in terms of perfs. I will look at it closer tomorrow.

[Honny1]

I reran that on Windows 11 and Hyper-V, and the results are the same; here is the report. I observed performance changes depending on the resources allocated to the Podman machine. I think I can set up an experiment to test different combinations, specifically comparing 2GB vs. 8GB of memory. Also, I am running these benchmarks on older hardware.

WDYT? @l0rd

Benchmark Report (hyperv)

[l0rd]

The problem was on my side. I didn't replace the podman service in the machine. Now I have fixed that and I can see the improvemnts in perfs (5s vs 65s!) and I have also quickly verified that a file in a drive that is not mounted in the machine is correctly copied (and is slower as expected).

LGTM on the Hyper-V side. I can have a look on WSL too if you want.

[Honny1]

That would be great to verify if WSL can benefit from newer hardware. Don't forget to build from source and drop the latest commit from this PR. (It disables this feature for WSL.)

[l0rd]

I confirm that on WSL, there is no performance difference before/after if I use the last commit of this PR (11s vs 5s with HyperV). But I get the same performances using commit 2f7094c too:

> .\bin\windows\podman version
Client:       Podman Engine
....
Git Commit:   2f7094c0dea104703c9adb9aedc505e503d34c42-dirty
....
Server:       Podman Engine
...
Git Commit:   2f7094c0dea104703c9adb9aedc505e503d34c42
...

> Measure-Command { ./bin/windows/podman artifact add quay.io/myartifact/benchmark:latest ./artifacts/random-file }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 11
Milliseconds      : 933
Ticks             : 119333670
TotalDays         : 0.000138117673611111
TotalHours        : 0.00331482416666667
TotalMinutes      : 0.19888945
TotalSeconds      : 11.933367
TotalMilliseconds : 11933.367

> wsl --version
WSL version: 2.6.1.0
Kernel version: 6.6.87.2-1
WSLg version: 1.0.66
MSRDC version: 1.2.6353
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.26200.7171

[Honny1]

This indicates that on faster computers, this operation can perform similarly. I would still keep this feature disabled for WSL.

[l0rd]

I have reviewed the code and, except a minor comment/question, LGTM.

Out of curiosity (and It's not something introduced by this PR anyway), but why do we require the artifact path to be absolute?

[l0rd]

Can't we use getProvider here?

[Honny1]

No, because this creates a small issue with providers and clients. When a user is using multiple providers, the client needs to determine which provider is used for the specific machine using that connection. I got caught by this. If we used getProvider, the user would need to specify CONTAINERS_MACHINE_PROVIDER before each command that utilizes the local API to determine which provider should be detected.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Honny1, l0rd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Honny1,l0rd]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Honny1]

Out of curiosity (and It's not something introduced by this PR anyway), but why do we require the artifact path to be absolute?

The local API facilitates path translation from client to server for convenience. Mounted host directories mimic the host's absolute paths, so a path on the host matches the path inside the VM (with a small exception for Windows, where C:\\ translates to /mnt/c/). Consequently, users utilizing the Podman client do not need to strictly use absolute paths. This is standard behavior across all local APIs.