rofl-scheduler: Investigate memory usage during OCI image pulls

[ptrus]

Testnet rofl-scheduler sometimes reached the 512MB limit while concurrently pulling 5+ OCI image layers. (current workaround: #2383)

We should also limit concurrency for this and not pull layers for all apps at once (e.g., on startup).

Some relevant logs:

{"instance_id":"00000000000003bd","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI image layers","ref":"rofl.sh/dd86bfee-7f3e-4f38-8ee0-14c33507c70c:1761900571@sha256:58fea041ff43efff4664181acb08a75da42b1f76813d1f528faf34d45a3652a3","temporary_name":"instance-00000000000003bd","ts":"2025-11-06T09:15:01.680937637Z"}
{"instance_id":"00000000000003ab","layer_name":"firmware.fd","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/d7d55e8f-f2a2-4917-8867-42d23727d54c:1761813307@sha256:2515ba4849b829a7dc73cdced60ff1e88b6bfed328c4f09a1497faf41da910d4","ts":"2025-11-06T09:15:01.692669744Z"}
{"instance_id":"0000000000000383","layer_name":"firmware.fd","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/f3e9b5d4-89e4-41d8-9650-7e29908ef7c7:1761138620@sha256:31d8330edfe29dd535f0a467156797f113baaa5d826a99457baf0ec2e754da52","ts":"2025-11-06T09:15:01.704687133Z"}
{"instance_id":"00000000000003bd","layer_name":"firmware.fd","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/dd86bfee-7f3e-4f38-8ee0-14c33507c70c:1761900571@sha256:58fea041ff43efff4664181acb08a75da42b1f76813d1f528faf34d45a3652a3","ts":"2025-11-06T09:15:01.716654472Z"}
{"instance_id":"000000000000035d","layer_name":"kernel.bin","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/3bd165af-2981-4601-96b6-890397d37556:1760461048@sha256:5edadc67bc9c59de3bd7aa15ce483a73dda3c5826e7d7e259f627c3f46f98000","ts":"2025-11-06T09:15:02.472870258Z"}
{"instance_id":"00000000000003a9","layer_name":"kernel.bin","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/54699ba7-3c32-47de-841a-6781544fb12d:1761647823@sha256:63caa8fc806605a776927939244bb5303ad0ac3ddad85f0b7b4942b85aba25b6","ts":"2025-11-06T09:15:02.566545569Z"}
{"instance_id":"00000000000003c8","layer_name":"kernel.bin","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/44109cf3-3da9-45a9-b9b4-4718be757e56:1762017912@sha256:74966595dd18bf958c1030b4cc83038f88b065c388b9bc933c9eec6af808ccd7","ts":"2025-11-06T09:15:03.236934474Z"}
{"instance_id":"00000000000003ab","layer_name":"kernel.bin","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/d7d55e8f-f2a2-4917-8867-42d23727d54c:1761813307@sha256:2515ba4849b829a7dc73cdced60ff1e88b6bfed328c4f09a1497faf41da910d4","ts":"2025-11-06T09:15:03.352059506Z"}
{"instance_id":"0000000000000383","layer_name":"kernel.bin","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/f3e9b5d4-89e4-41d8-9650-7e29908ef7c7:1761138620@sha256:31d8330edfe29dd535f0a467156797f113baaa5d826a99457baf0ec2e754da52","ts":"2025-11-06T09:15:03.384580871Z"}
{"instance_id":"00000000000003bd","layer_name":"kernel.bin","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/dd86bfee-7f3e-4f38-8ee0-14c33507c70c:1761900571@sha256:58fea041ff43efff4664181acb08a75da42b1f76813d1f528faf34d45a3652a3","ts":"2025-11-06T09:15:03.398276786Z"}
{"instance_id":"000000000000035d","layer_name":"stage2.img","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/3bd165af-2981-4601-96b6-890397d37556:1760461048@sha256:5edadc67bc9c59de3bd7aa15ce483a73dda3c5826e7d7e259f627c3f46f98000","ts":"2025-11-06T09:15:05.619190134Z"}
{"instance_id":"00000000000003a9","layer_name":"stage2.img","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/54699ba7-3c32-47de-841a-6781544fb12d:1761647823@sha256:63caa8fc806605a776927939244bb5303ad0ac3ddad85f0b7b4942b85aba25b6","ts":"2025-11-06T09:15:06.340557002Z"}
{"instance_id":"00000000000003c8","layer_name":"stage2.img","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/44109cf3-3da9-45a9-b9b4-4718be757e56:1762017912@sha256:74966595dd18bf958c1030b4cc83038f88b065c388b9bc933c9eec6af808ccd7","ts":"2025-11-06T09:15:07.117864804Z"}
{"instance_id":"00000000000003ab","layer_name":"stage2.img","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/d7d55e8f-f2a2-4917-8867-42d23727d54c:1761813307@sha256:2515ba4849b829a7dc73cdced60ff1e88b6bfed328c4f09a1497faf41da910d4","ts":"2025-11-06T09:15:07.203221145Z"}
{"instance_id":"00000000000003bd","layer_name":"stage2.img","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/dd86bfee-7f3e-4f38-8ee0-14c33507c70c:1761900571@sha256:58fea041ff43efff4664181acb08a75da42b1f76813d1f528faf34d45a3652a3","ts":"2025-11-06T09:15:07.230679577Z"}
{"instance_id":"0000000000000383","layer_name":"stage2.img","level":"info","module":"runtime/scheduler/manager","msg":"pulling OCI layer","ref":"rofl.sh/f3e9b5d4-89e4-41d8-9650-7e29908ef7c7:1761138620@sha256:31d8330edfe29dd535f0a467156797f113baaa5d826a99457baf0ec2e754da52","ts":"2025-11-06T09:15:07.24309199Z"}
{"err":"Cannot allocate memory (os error 12)","level":"warn","module":"runtime/protocol","msg":"Failed to write message","ts":"2025-11-06T09:15:25.256614859Z"}
{"err":"failed to fill whole buffer","level":"error","module":"runtime/protocol","msg":"Failed to handle message","ts":"2025-11-06T09:15:25.283382097Z"}
{"level":"info","module":"runtime/protocol","msg":"Protocol reader thread is terminating","ts":"2025-11-06T09:15:25.293153009Z"}