Bumped llama cpp and updated deprecated functions #3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…desired using `-1` (abetlen#1364)
* allow for possibly non-pooled embeddings * add more to embeddings section in README.md --------- Co-authored-by: Andrei <[email protected]>
… definition in bindings
Add github-actions update
* Bump actions/setup-python@v4 to v5 * Update build-wheels-metal.yaml * Update build-wheels-metal.yaml * Update build-wheels-metal.yaml
* fix completion tokens tracking, prompt forming * fix 'function_call' and 'tool_calls' depending on 'functions' and 'tools', incompatibility with python 3.8 * Updated README * fix for openai server compatibility --------- Co-authored-by: Andrei <[email protected]>
…abetlen#1397) Bumps [conda-incubator/setup-miniconda](https://github.com/conda-incubator/setup-miniconda) from 2.2.0 to 3.0.4. - [Release notes](https://github.com/conda-incubator/setup-miniconda/releases) - [Changelog](https://github.com/conda-incubator/setup-miniconda/blob/main/CHANGELOG.md) - [Commits](conda-incubator/setup-miniconda@v2.2.0...v3.0.4) --- updated-dependencies: - dependency-name: conda-incubator/setup-miniconda dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/cache](https://github.com/actions/cache) from 3.3.2 to 4.0.2. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v3.3.2...v4.0.2) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [docker/login-action](https://github.com/docker/login-action) from 2 to 3. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@v2...v3) --- updated-dependencies: - dependency-name: docker/login-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 4 to 5. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v4...v5) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.16.5 to 2.17.0. - [Release notes](https://github.com/pypa/cibuildwheel/releases) - [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md) - [Commits](pypa/cibuildwheel@v2.16.5...v2.17.0) --- updated-dependencies: - dependency-name: pypa/cibuildwheel dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add support for cuda 12.4.1 * Update build-wheels-cuda.yaml * Update build-wheels-cuda.yaml * Update build-wheels-cuda.yaml * Update build-wheels-cuda.yaml * Update build-wheels-cuda.yaml * Update build-wheels-cuda.yaml * Update build-wheels-cuda.yaml * Update build-wheels-cuda.yaml * Update build-wheels-cuda.yaml Revert
…n release (abetlen#1392) * Update test.yaml Bump actions/checkout@v3 to v4 Bump action/setup-python@v4 to v5 * Update test-pypi.yaml Bum actions/setup-python@v4 to v5 * Update build-and-release.yaml Bump softprops/action-gh-release@v1 to v2 Bump actions/checkout@v3 to v4 Bump actions/setup-python@v3 to v5 * Update publish.yaml Bump actions/checkout@v3 to v4 Bump actions/sertup-python@v4 to v5 * Update publish-to-test.yaml Bump actions/checkout@v3 to v4 Bump actions/setup-python @v4 to v5 * Update test-pypi.yaml Add Python 3.12 * Update build-and-release.yaml * Update build-docker.yaml Bump docker/setup-qemu-action@v2 to v3 Bump docker/setup-buildx-action@v2 to v3 * Update build-and-release.yaml * Update build-and-release.yaml
Bumps [softprops/action-gh-release](https://github.com/softprops/action-gh-release) from 1 to 2. - [Release notes](https://github.com/softprops/action-gh-release/releases) - [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md) - [Commits](softprops/action-gh-release@v1...v2) --- updated-dependencies: - dependency-name: softprops/action-gh-release dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/configure-pages](https://github.com/actions/configure-pages) from 4 to 5. - [Release notes](https://github.com/actions/configure-pages/releases) - [Commits](actions/configure-pages@v4...v5) --- updated-dependencies: - dependency-name: actions/configure-pages dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v3...v4) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…t for Multimodal Models (Obsidian, LLaVA1.6, Moondream) (abetlen#1147) * Test dummy image tags in chat templates * Format and improve types for llava_cpp.py * Add from_pretrained support to llava chat format. * Refactor llava chat format to use a jinja2 * Revert chat format test * Add moondream support (wip) * Update moondream chat format * Update moondream chat format * Update moondream prompt * Add function calling support * Cache last image embed * Add Llava1.6 support * Add nanollava support * Add obisidian support * Remove unnecessary import * Re-order multimodal chat formats * Logits all no longer required for multi-modal models * Update README.md * Update docs * Update README * Fix typo * Update README * Fix typo
Has to skip deserializing RNG state but based on new branch.
Add libggml.so and libllama.so under `llama_cpp/lib` (path expected in `_load_shared_library` in `llama_cpp.py`). This means that will be able to locate once bundled. Also prevent from adding *full* OpenBLAS dir (unnecessary) and just the `libopenblas.so` onece built. This shrinks binary size from 200 MB to 40 MB.
Passing mcpu/march to cmake
- Verbose logging / output when compiling - Save buildlog (though currently don't export) so that can inspect build afterward - Build changes: - Disable LLAMAFILE (needed for Q4_0_4_4) - Set march/mcpu/mtune for C/C++ - Make pip verbose (so that get cmake compile output)
- scikit-build -> scikit-build-core - Remove -DGGML_BLAS=ON, OpenBLAS cmake tags from build - Needed (see ggml-org/llama.cpp#5780 (review)) to get this to work properly - Built, deployed, and tested with llama3.1 with q4_0_4_4 quantization
- Add logging statements (seed, timing, match length, etc.) - Change logic: - Use `find_longest_prefix_key` to find length of longest key in cache - If cache_prefix_len > eval_prefix_len, load from disk - Otherwise, skip loading. - Change logging to decode to utf-8 (helpful for JP prompts)
- Create `reload_from_cache_state` method - Still using LLamaState as container - Use low level `ctx.get_logits_ith` to get last calculated logits. - Add StateReloadError so that can be fallible. - Change Llama class to use this instead of `load_state` directly. - Default implementation still uses `load_state`.
Catch StateReloadError and add logging if runs into this when running.
- Check when saving that model doesn't need logits - Ad note in `reload_from_cache` state to revisit
- Make default to *not* save logits - Error if needed and save_logits False in build_cache - Handle reloading with/without scores if needed + available
- Add more tests - Make llama_state / small_model module scope (so don't need to reload for each test) - Setting env var in `.env` file
Optimize KV cache size
Take out references to OpenBLAS since no longer used
Add deploy target for mac server bundle
- Needs glibc <= 2.31 - But have to use GCC-11+ to build with specific march/mcpu/mtune.
- Set CC / CXX for cmmake build - Make sure python3.9-dev installed (needed for linking w/ pyinstaller into standalone) - Set CMAKE_BUILD_TYPE as env var instead of as cmake var (based on build logs)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
38 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.