Separate images for runtime

[deitch]

There are 2 distinct scenarios.

1. Someone downloads a model from HF or OCI in onnx format, runs onnxruntime to compile to .bundle, save to local cache, and then run it
2. Someone downloads a model from HF or OCI in .bundle format, runs onnxruntime to run it, no compilation necessary.

There are two image-related issues with the second scenario.

First, the image is much larger than it needs to be. It still needs the inference server, and maybe neuralizer, but it does not need glow or gcc or llvm, etc. This makes it harder to use and is a worse user experience.

Second, the image name is confusing. It no longer is an _onnx_runner; it is a _bundle_runner. This is equivalent to using gcc tools to run an executable. Sure, developers can (and sometimes do) use gdb to run something, but for the vast majority of users, they do not need or want the toolchain; they just want the compiled binary.

We probably need to separate the image into two:

• bundlerunner - runs a compiled .bundle
• onnxruntime - compiles an onnx model into a .bundle and then optionally runs it

We can discuss if onnxruntime should be able to run the compiled bundle, or if the correct process is to say "run this" and then "run that". Since it all is wrapped in nekko, I would favour separate.

cc @jerenkrantz @rvs

[jerenkrantz]

Hmm - we should probably discuss this one in more detail. I'm not quite sure I follow along.

I think part of @rvs's general dislike of onnxruntime is that it combines the compilation and execution steps into Python code. So, I think we can't separate this out like this ... if we move to a llama.cpp/GGML world, it's probably more feasible to have it be distinct ... maybe.

[rvs]

A couple of comments to set the record strait @deitch:

1. when you say "runs onnxruntime to compile to .bundle" -- the problem is that there's no separate compilation step currently exposed. Which is a big deal and we need to fix it (since it is very much just calling a particular GlowAPI). But I like where you are going with this -- we need to make this step explicit in our CLI (even if it costs us some behind the scenes python API wrangling). Can you please capture a separate issue for this if you agree? For now -- the step you're describing -- the user doesn't even know the bundle is being compiled.
2. when you say "Someone downloads a model from HF or OCI in .bundle format, runs onnxruntime to run it" again I don't think I know how to do that easily today (I know it is doable -- but we need to have a separate Python entrypoint for that) -- again like where you are going with this -- we need to make this step explicit in our CLI. Can you please capture a separate issue for this if you agree?
3. when you say "First, the image is much larger than it needs to be." I really would like to understand what kind of optimizations you'd want to propose here.

Now, finally your overall proposal "We probably need to separate the image into two:" I'd absolutely agree with! That's how it should like at the level of the nekko CLI -- how we do it under the covers is less of a concern (I am sure we can come up with a reasonably python scripting for this).

[deitch]

Yes, to what you wrote (all of them) @rvs . I am saying - purely from a product user perspective - people should be able to do any of the below:

• download an onnx and run it from start to finish
• download an onnx and compile it and then stop, maybe push it
• run a compiled .bundle
• download a compiled .bundle
• run a compiled .bundle

All of the above assume .bundle exists (I don't care if it is a tarred .bundle file or a bundle/ directory). As long as there is an expensive compilation step, I should be able to do compilation and execution as separate steps or together; and I should be able to manage the source artifacts and executable artifacts independently or together.

[jerenkrantz]

ETGlow does have a compile-only flag - see:

https://github.com/nekkoai/atlantis/blob/3441e0739c391dfc04544285cad19f5cb924fa53/glow/lib/Backends/ETSOC/ETSOCCommandLine.cpp#L153

BTW, it might also be helpful to wrap our heads around ET’s script which is spiritually similar to what we are trying to achieve with cli:

https://github.com/nekkoai/atlantis/tree/3441e0739c391dfc04544285cad19f5cb924fa53/techDemo

techDemo/scripts/run-techdemo.sh is the main driver.

[deitch]

ETGlow does have a compile-only flag - see

In that case, it would just compile to OR directory and stop?

[jerenkrantz]

I believe that if you pass that compile-only flag and try to run inference, it would error out. So, the Python driver (eg, llm-kvc.py) would need to stop. The as-is version doesn’t do this, but I can experiment and report back…and maybe post a PR to that effect.

[deitch]

might also be helpful to wrap our heads around ET’s script

this line says that it is "bundledir", which implies that it creates a directory for the bundles, but maybe that means a directory in which to deposit a single bundle tar file? I don't think so, but still figuring it out.

The whole structure of techdemo is a bit strange. I am not sure they quite got how to build images and use them, but I guess whatever works. sw-platform.py does not seem to exist anywhere. I imagine it is not too hard to recreate the container used to run it. They also have a strange directory structure under models/ and datasets/.