Skip to content

Package context size #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ilopezluna merged 15 commits into from
Oct 20, 2025
Merged

Package context size #243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
015cdbe
feat(builder): add FromModel function to create builders from existin…
ilopezluna Oct 15, 2025
19afeb6
feat(package): add support for packaging from existing models
ilopezluna Oct 15, 2025
6fede01
feat(package): implement lightweight model writing for config-only mo…
ilopezluna Oct 15, 2025
7342fa4
make docs
ilopezluna Oct 15, 2025
f2f2827
Merge branch 'main' into package-context-size
ilopezluna Oct 20, 2025
1c3b1f4
Apply suggestion from @Copilot
ilopezluna Oct 20, 2025
f061eaf
feat(package): refactor package model initialization and error handling
ilopezluna Oct 20, 2025
5878f7f
Merge remote-tracking branch 'origin/package-context-size' into packa…
ilopezluna Oct 20, 2025
c1ed374
fix(store): update layer identification from DiffID to Digest
ilopezluna Oct 20, 2025
f0a038a
feat(builder): add originalLayers tracking and HasOnlyConfigChanges m…
ilopezluna Oct 20, 2025
82ca07d
feat(package): enhance packaging options to include existing models
ilopezluna Oct 20, 2025
13e3b2a
test(store): add tests for WriteLightweight method and index integrity
ilopezluna Oct 20, 2025
901f566
Apply suggestion from @Copilot
ilopezluna Oct 20, 2025
6c4e648
feat(builder): enhance FromModel to return error on layer retrieval f…
ilopezluna Oct 20, 2025
0fc2478
Merge remote-tracking branch 'origin/package-context-size' into packa…
ilopezluna Oct 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
206 changes: 152 additions & 54 deletions cmd/cli/commands/package.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"path/filepath"

"github.com/docker/model-runner/pkg/distribution/builder"
"github.com/docker/model-runner/pkg/distribution/distribution"
"github.com/docker/model-runner/pkg/distribution/packaging"
"github.com/docker/model-runner/pkg/distribution/registry"
"github.com/docker/model-runner/pkg/distribution/tarball"
Expand All @@ -26,11 +27,12 @@ func newPackagedCmd() *cobra.Command {
var opts packageOptions

c := &cobra.Command{
Use: "package (--gguf <path> | --safetensors-dir <path>) [--license <path>...] [--context-size <tokens>] [--push] MODEL",
Short: "Package a GGUF file or Safetensors directory into a Docker model OCI artifact.",
Long: "Package a GGUF file or Safetensors directory into a Docker model OCI artifact, with optional licenses. The package is sent to the model-runner, unless --push is specified.\n" +
Use: "package (--gguf <path> | --safetensors-dir <path> | --from <model>) [--license <path>...] [--context-size <tokens>] [--push] MODEL",
Short: "Package a GGUF file, Safetensors directory, or existing model into a Docker model OCI artifact.",
Long: "Package a GGUF file, Safetensors directory, or existing model into a Docker model OCI artifact, with optional licenses. The package is sent to the model-runner, unless --push is specified.\n" +
"When packaging a sharded GGUF model, --gguf should point to the first shard. All shard files should be siblings and should include the index in the file name (e.g. model-00001-of-00015.gguf).\n" +
"When packaging a Safetensors model, --safetensors-dir should point to a directory containing .safetensors files and config files (*.json, merges.txt). All files will be auto-discovered and config files will be packaged into a tar archive.",
"When packaging a Safetensors model, --safetensors-dir should point to a directory containing .safetensors files and config files (*.json, merges.txt). All files will be auto-discovered and config files will be packaged into a tar archive.\n" +
"When packaging from an existing model using --from, you can modify properties like context size to create a variant of the original model.",
Args: func(cmd *cobra.Command, args []string) error {
if len(args) != 1 {
return fmt.Errorf(
Expand All @@ -41,16 +43,27 @@ func newPackagedCmd() *cobra.Command {
)
}

// Validate that either --gguf or --safetensors-dir is provided (mutually exclusive)
if opts.ggufPath == "" && opts.safetensorsDir == "" {
// Validate that exactly one of --gguf, --safetensors-dir, or --from is provided (mutually exclusive)
sourcesProvided := 0
if opts.ggufPath != "" {
sourcesProvided++
}
if opts.safetensorsDir != "" {
sourcesProvided++
}
if opts.fromModel != "" {
sourcesProvided++
}

if sourcesProvided == 0 {
return fmt.Errorf(
"Either --gguf or --safetensors-dir path is required.\n\n" +
"One of --gguf, --safetensors-dir, or --from is required.\n\n" +
"See 'docker model package --help' for more information",
)
}
if opts.ggufPath != "" && opts.safetensorsDir != "" {
if sourcesProvided > 1 {
return fmt.Errorf(
"Cannot specify both --gguf and --safetensors-dir. Please use only one format.\n\n" +
"Cannot specify more than one of --gguf, --safetensors-dir, or --from. Please use only one source.\n\n" +
"See 'docker model package --help' for more information",
)
}
Expand Down Expand Up @@ -133,6 +146,7 @@ func newPackagedCmd() *cobra.Command {

c.Flags().StringVar(&opts.ggufPath, "gguf", "", "absolute path to gguf file")
c.Flags().StringVar(&opts.safetensorsDir, "safetensors-dir", "", "absolute path to directory containing safetensors files and config")
c.Flags().StringVar(&opts.fromModel, "from", "", "reference to an existing model to repackage")
c.Flags().StringVar(&opts.chatTemplatePath, "chat-template", "", "absolute path to chat template file (must be Jinja format)")
c.Flags().StringArrayVarP(&opts.licensePaths, "license", "l", nil, "absolute path to a license file")
c.Flags().StringArrayVar(&opts.dirTarPaths, "dir-tar", nil, "relative path to directory to package as tar (can be specified multiple times)")
Expand All @@ -146,65 +160,133 @@ type packageOptions struct {
contextSize uint64
ggufPath string
safetensorsDir string
fromModel string
licensePaths []string
dirTarPaths []string
push bool
tag string
}

func packageModel(cmd *cobra.Command, opts packageOptions) error {
var (
target builder.Target
err error
)
if opts.push {
target, err = registry.NewClient(
registry.WithUserAgent("docker-model-cli/" + desktop.Version),
).NewTarget(opts.tag)
} else {
target, err = newModelRunnerTarget(desktopClient, opts.tag)
}
if err != nil {
return err
}
// builderInitResult contains the result of initializing a builder from various sources
type builderInitResult struct {
builder *builder.Builder
distClient *distribution.Client // Only set when building from existing model
cleanupFunc func() // Optional cleanup function for temporary files
}

// initializeBuilder creates a package builder from GGUF, Safetensors, or existing model
func initializeBuilder(cmd *cobra.Command, opts packageOptions) (*builderInitResult, error) {
result := &builderInitResult{}

if opts.fromModel != "" {
// Get the model store path
userHomeDir, err := os.UserHomeDir()
if err != nil {
return nil, fmt.Errorf("get user home directory: %w", err)
}
modelStorePath := filepath.Join(userHomeDir, ".docker", "models")
if envPath := os.Getenv("MODELS_PATH"); envPath != "" {
modelStorePath = envPath
}

// Create package builder based on model format
var pkg *builder.Builder
if opts.ggufPath != "" {
// Create a distribution client to access the model store
distClient, err := distribution.NewClient(distribution.WithStoreRootPath(modelStorePath))
if err != nil {
return nil, fmt.Errorf("create distribution client: %w", err)
}
result.distClient = distClient

// Package from existing model
cmd.PrintErrf("Reading model from store: %q\n", opts.fromModel)

// Get the model from the local store
mdl, err := distClient.GetModel(opts.fromModel)
if err != nil {
return nil, fmt.Errorf("get model from store: %w", err)
}

// Type assert to ModelArtifact - the Model from store implements both interfaces
modelArtifact, ok := mdl.(types.ModelArtifact)
if !ok {
return nil, fmt.Errorf("model does not implement ModelArtifact interface")
}

cmd.PrintErrf("Creating builder from existing model\n")
result.builder, err = builder.FromModel(modelArtifact)
if err != nil {
return nil, fmt.Errorf("create builder from model: %w", err)
}
} else if opts.ggufPath != "" {
cmd.PrintErrf("Adding GGUF file from %q\n", opts.ggufPath)
pkg, err = builder.FromGGUF(opts.ggufPath)
pkg, err := builder.FromGGUF(opts.ggufPath)
if err != nil {
return fmt.Errorf("add gguf file: %w", err)
return nil, fmt.Errorf("add gguf file: %w", err)
}
result.builder = pkg
} else {
// Safetensors model from directory
cmd.PrintErrf("Scanning directory %q for safetensors model...\n", opts.safetensorsDir)
safetensorsPaths, tempConfigArchive, err := packaging.PackageFromDirectory(opts.safetensorsDir)
if err != nil {
return fmt.Errorf("scan safetensors directory: %w", err)
return nil, fmt.Errorf("scan safetensors directory: %w", err)
}

// Clean up temp config archive when done
// Set up cleanup for temp config archive
if tempConfigArchive != "" {
defer os.Remove(tempConfigArchive)
result.cleanupFunc = func() {
os.Remove(tempConfigArchive)
}
}

cmd.PrintErrf("Found %d safetensors file(s)\n", len(safetensorsPaths))
pkg, err = builder.FromSafetensors(safetensorsPaths)
pkg, err := builder.FromSafetensors(safetensorsPaths)
if err != nil {
return fmt.Errorf("create safetensors model: %w", err)
return nil, fmt.Errorf("create safetensors model: %w", err)
}

// Add config archive if it was created
if tempConfigArchive != "" {
cmd.PrintErrf("Adding config archive from directory\n")
pkg, err = pkg.WithConfigArchive(tempConfigArchive)
if err != nil {
return fmt.Errorf("add config archive: %w", err)
return nil, fmt.Errorf("add config archive: %w", err)
}
}
result.builder = pkg
}

return result, nil
}

func packageModel(cmd *cobra.Command, opts packageOptions) error {
var (
target builder.Target
err error
)
if opts.push {
target, err = registry.NewClient(
registry.WithUserAgent("docker-model-cli/" + desktop.Version),
).NewTarget(opts.tag)
} else {
target, err = newModelRunnerTarget(desktopClient, opts.tag)
}
if err != nil {
return err
}

// Initialize the package builder based on model format
initResult, err := initializeBuilder(cmd, opts)
if err != nil {
return err
}
// Clean up any temporary files when done
if initResult.cleanupFunc != nil {
defer initResult.cleanupFunc()
}

pkg := initResult.builder
distClient := initResult.distClient

// Set context size
if opts.contextSize > 0 {
cmd.PrintErrf("Setting context size %d\n", opts.contextSize)
Expand All @@ -227,32 +309,48 @@ func packageModel(cmd *cobra.Command, opts packageOptions) error {
}
}

// Process directory tar archives
if len(opts.dirTarPaths) > 0 {
// Determine base directory for resolving relative paths
var baseDir string
if opts.safetensorsDir != "" {
baseDir = opts.safetensorsDir
} else {
// For GGUF, use the directory containing the GGUF file
baseDir = filepath.Dir(opts.ggufPath)
}
// Check if we can use lightweight repackaging (config-only changes from existing model)
useLightweight := opts.fromModel != "" && pkg.HasOnlyConfigChanges()

processor := packaging.NewDirTarProcessor(opts.dirTarPaths, baseDir)
tarPaths, cleanup, err := processor.Process()
if err != nil {
return err
if useLightweight {
cmd.PrintErrln("Creating lightweight model variant...")

// Get the model artifact with new config
builtModel := pkg.Model()

// Write using lightweight method
if err := distClient.WriteLightweightModel(builtModel, []string{opts.tag}); err != nil {
return fmt.Errorf("failed to create lightweight model: %w", err)
}
defer cleanup()

for _, tarPath := range tarPaths {
pkg, err = pkg.WithDirTar(tarPath)
cmd.PrintErrln("Model variant created successfully")
} else {
// Process directory tar archives
if len(opts.dirTarPaths) > 0 {
// Determine base directory for resolving relative paths
var baseDir string
if opts.safetensorsDir != "" {
baseDir = opts.safetensorsDir
} else {
// For GGUF, use the directory containing the GGUF file
baseDir = filepath.Dir(opts.ggufPath)
}

processor := packaging.NewDirTarProcessor(opts.dirTarPaths, baseDir)
tarPaths, cleanup, err := processor.Process()
if err != nil {
return fmt.Errorf("add directory tar: %w", err)
return err
}
defer cleanup()

for _, tarPath := range tarPaths {
pkg, err = pkg.WithDirTar(tarPath)
if err != nil {
return fmt.Errorf("add directory tar: %w", err)
}
}
}
}

if opts.push {
cmd.PrintErrln("Pushing model to registry...")
} else {
Expand Down
16 changes: 13 additions & 3 deletions cmd/cli/docs/reference/docker_model_package.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
command: docker model package
short: |
Package a GGUF file or Safetensors directory into a Docker model OCI artifact.
Package a GGUF file, Safetensors directory, or existing model into a Docker model OCI artifact.
long: |-
Package a GGUF file or Safetensors directory into a Docker model OCI artifact, with optional licenses. The package is sent to the model-runner, unless --push is specified.
Package a GGUF file, Safetensors directory, or existing model into a Docker model OCI artifact, with optional licenses. The package is sent to the model-runner, unless --push is specified.
When packaging a sharded GGUF model, --gguf should point to the first shard. All shard files should be siblings and should include the index in the file name (e.g. model-00001-of-00015.gguf).
When packaging a Safetensors model, --safetensors-dir should point to a directory containing .safetensors files and config files (*.json, merges.txt). All files will be auto-discovered and config files will be packaged into a tar archive.
usage: docker model package (--gguf <path> | --safetensors-dir <path>) [--license <path>...] [--context-size <tokens>] [--push] MODEL
When packaging from an existing model using --from, you can modify properties like context size to create a variant of the original model.
usage: docker model package (--gguf <path> | --safetensors-dir <path> | --from <model>) [--license <path>...] [--context-size <tokens>] [--push] MODEL
pname: docker model
plink: docker_model.yaml
options:
Expand Down Expand Up @@ -39,6 +40,15 @@ options:
experimentalcli: false
kubernetes: false
swarm: false
- option: from
value_type: string
description: reference to an existing model to repackage
deprecated: false
hidden: false
experimental: false
experimentalcli: false
kubernetes: false
swarm: false
- option: gguf
value_type: string
description: absolute path to gguf file
Expand Down
46 changes: 23 additions & 23 deletions cmd/cli/docs/reference/model.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,29 +5,29 @@ Docker Model Runner

### Subcommands

| Name | Description |
|:------------------------------------------------|:-------------------------------------------------------------------------------|
| [`df`](model_df.md) | Show Docker Model Runner disk usage |
| [`inspect`](model_inspect.md) | Display detailed information on one model |
| [`install-runner`](model_install-runner.md) | Install Docker Model Runner (Docker Engine only) |
| [`list`](model_list.md) | List the models pulled to your local environment |
| [`logs`](model_logs.md) | Fetch the Docker Model Runner logs |
| [`package`](model_package.md) | Package a GGUF file or Safetensors directory into a Docker model OCI artifact. |
| [`ps`](model_ps.md) | List running models |
| [`pull`](model_pull.md) | Pull a model from Docker Hub or HuggingFace to your local environment |
| [`push`](model_push.md) | Push a model to Docker Hub |
| [`reinstall-runner`](model_reinstall-runner.md) | Reinstall Docker Model Runner (Docker Engine only) |
| [`requests`](model_requests.md) | Fetch requests+responses from Docker Model Runner |
| [`restart-runner`](model_restart-runner.md) | Restart Docker Model Runner (Docker Engine only) |
| [`rm`](model_rm.md) | Remove local models downloaded from Docker Hub |
| [`run`](model_run.md) | Run a model and interact with it using a submitted prompt or chat mode |
| [`start-runner`](model_start-runner.md) | Start Docker Model Runner (Docker Engine only) |
| [`status`](model_status.md) | Check if the Docker Model Runner is running |
| [`stop-runner`](model_stop-runner.md) | Stop Docker Model Runner (Docker Engine only) |
| [`tag`](model_tag.md) | Tag a model |
| [`uninstall-runner`](model_uninstall-runner.md) | Uninstall Docker Model Runner (Docker Engine only) |
| [`unload`](model_unload.md) | Unload running models |
| [`version`](model_version.md) | Show the Docker Model Runner version |
| Name | Description |
|:------------------------------------------------|:------------------------------------------------------------------------------------------------|
| [`df`](model_df.md) | Show Docker Model Runner disk usage |
| [`inspect`](model_inspect.md) | Display detailed information on one model |
| [`install-runner`](model_install-runner.md) | Install Docker Model Runner (Docker Engine only) |
| [`list`](model_list.md) | List the models pulled to your local environment |
| [`logs`](model_logs.md) | Fetch the Docker Model Runner logs |
| [`package`](model_package.md) | Package a GGUF file, Safetensors directory, or existing model into a Docker model OCI artifact. |
| [`ps`](model_ps.md) | List running models |
| [`pull`](model_pull.md) | Pull a model from Docker Hub or HuggingFace to your local environment |
| [`push`](model_push.md) | Push a model to Docker Hub |
| [`reinstall-runner`](model_reinstall-runner.md) | Reinstall Docker Model Runner (Docker Engine only) |
| [`requests`](model_requests.md) | Fetch requests+responses from Docker Model Runner |
| [`restart-runner`](model_restart-runner.md) | Restart Docker Model Runner (Docker Engine only) |
| [`rm`](model_rm.md) | Remove local models downloaded from Docker Hub |
| [`run`](model_run.md) | Run a model and interact with it using a submitted prompt or chat mode |
| [`start-runner`](model_start-runner.md) | Start Docker Model Runner (Docker Engine only) |
| [`status`](model_status.md) | Check if the Docker Model Runner is running |
| [`stop-runner`](model_stop-runner.md) | Stop Docker Model Runner (Docker Engine only) |
| [`tag`](model_tag.md) | Tag a model |
| [`uninstall-runner`](model_uninstall-runner.md) | Uninstall Docker Model Runner (Docker Engine only) |
| [`unload`](model_unload.md) | Unload running models |
| [`version`](model_version.md) | Show the Docker Model Runner version |



Expand Down
Loading
Loading