feat: Update with upstream pr 460

Author: rhys101

.github/workflows/test.yml

@@ -23,25 +23,25 @@ jobs:
       - name: TypeScript validity
         run: npm run test:typescript
 
-  compile-docs:
-    name: Test docs compilation
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          lfs: true
-          fetch-depth: 0
-          fetch-tags: true
-      - uses: actions/setup-node@v4
-        with:
-          node-version: "20"
-      - name: Install modules
-        run: npm ci
-      - name: Build
-        run: npm run build
-      - name: Download latest llama.cpp release
-        env:
-          CI: true
-        run: node ./dist/cli/cli.js source download --release latest --skipBuild --noBundle --noUsageExample --updateBinariesReleaseMetadataAndSaveGitBundle
-      - name: Compile docs
-        run: npm run docs:build
+#  compile-docs:
+#    name: Test docs compilation
+#    runs-on: ubuntu-latest
+#    steps:
+#      - uses: actions/checkout@v4
+#        with:
+#          lfs: true
+#          fetch-depth: 0
+#          fetch-tags: true
+#      - uses: actions/setup-node@v4
+#        with:
+#          node-version: "20"
+#      - name: Install modules
+#        run: npm ci
+#      - name: Build
+#        run: npm run build
+#      - name: Download latest llama.cpp release
+#        env:
+#          CI: true
+#        run: node ./dist/cli/cli.js source download --release latest --skipBuild --noBundle --noUsageExample --updateBinariesReleaseMetadataAndSaveGitBundle
+#      - name: Compile docs
+#        run: npm run docs:build

.vitepress/config.ts

@@ -470,8 +470,6 @@ export default defineConfig({
             }
         },
         sidebar: {
-            "/api/": getApiReferenceSidebar(),
-
             "/guide/": [{
                 text: "Guide",
                 base: "/guide",
@@ -550,7 +548,9 @@ export default defineConfig({
                         ]
                     }
                 ]
-            }]
+            }],
+
+            "/api/": getApiReferenceSidebar()
         },
         socialLinks: [
             {icon: "npm", link: "https://www.npmjs.com/package/node-llama-cpp"},

docs/cli/pull.md

@@ -20,7 +20,7 @@ If a file already exists and its size matches the expected size, it will not be
 
 The supported URI schemes are:
 - **HTTP:** `https://`, `http://`
-- **Hugging Face:** `hf:<user>/<model>:<quant>` (`#<quant>` is optional, [but recommended](../guide/downloading-models.md#hf-scheme-specify-quant))
+- **Hugging Face:** `hf:<user>/<model>:<quant>` (`:<quant>` is optional, [but recommended](../guide/downloading-models.md#hf-scheme-specify-quant))
 - **Hugging Face:** `hf:<user>/<model>/<file-path>#<branch>` (`#<branch>` is optional)
 
 Learn more about using model URIs in the [Downloading Models guide](../guide/downloading-models.md#model-uris).

docs/guide/CUDA.md

@@ -114,6 +114,33 @@ set NODE_LLAMA_CPP_CMAKE_OPTION_CMAKE_GENERATOR_TOOLSET=%CUDA_PATH%
 
 Then run the build command again to check whether setting the `CMAKE_GENERATOR_TOOLSET` cmake option fixed the issue.
 
+### Fix the `forward compatibility was attempted on non supported HW` Error {#fix-cuda-forward-compatibility}
+This error usually happens when the CUDA version you have installed on your machine is older than the CUDA version used in the prebuilt binaries supplied by `node-llama-cpp`.
+
+To resolve this issue, you can either [update your CUDA installation](https://developer.nvidia.com/cuda-downloads) to the latest version (recommended) or [build `node-llama-cpp` on your machine](#building) against the CUDA version you have installed.
+
+### Fix the `Binary GPU type mismatch. Expected: cuda, got: false` Error {#fix-cuda-gpu-type-mismatch}
+This error usually happens when you have multiple conflicting CUDA versions installed on your machine.
+
+To fix it, uninstall older CUDA versions and restart your machine (important).
+
+:::: details Check which CUDA libraries are picked up by `node-llama-cpp`'s prebuilt binaries on your machine
+
+Run this command inside of your project:
+
+::: code-group
+```shell [Linux]
+ldd ./node_modules/@node-llama-cpp/linux-x64-cuda/bins/linux-x64-cuda/libggml-cuda.so
+```
+
+```cmd [Windows]
+"C:\Program Files\Git\usr\bin\ldd.exe" node_modules\@node-llama-cpp\win-x64-cuda\bins\win-x64-cuda\ggml-cuda.dll
+```
+:::
+
+::::
+
+
 ## Using `node-llama-cpp` With CUDA
 It's recommended to use [`getLlama`](../api/functions/getLlama) without specifying a GPU type,
 so it'll detect the available GPU types and use the best one automatically.

docs/guide/awesome.md

@@ -2,17 +2,32 @@
 description: Awesome projects that use node-llama-cpp
 ---
 # Awesome `node-llama-cpp`
-Awesome projects that use `node-llama-cpp`.
+:sunglasses: Awesome projects that use `node-llama-cpp`.
+
+<script setup lang="ts">
+import DataBadge from "../../.vitepress/components/DataBadge/DataBadge.vue";
+</script>
 
 ## Open Source
 * [CatAI](https://github.com/withcatai/catai) - a simplified AI assistant API for Node.js, with REST API support
+  <br /><DataBadge title="License" content="MIT"/>
+
+* [Manzoni](https://manzoni.app/) ([GitHub](https://github.com/gems-platforms/manzoni-app)) - a text editor running local LLMs
+  <br /><DataBadge title="License" content="AGPL-3.0"/>
+
 
 ## Proprietary
-> List your project here!
+* [BashBuddy](https://bashbuddy.run) ([GitHub](https://github.com/wosherco/bashbuddy)) - write bash commands with natural language
+  <br /><DataBadge title="Partially open source" content="Source available" href="https://github.com/wosherco/bashbuddy/blob/main/LICENSE.md"/>
+
+* [nutshell](https://withnutshell.com) - Private AI meeting notes processed completely on your device
+
 
 
 <br />
 
 ---
 
+> To add a project to this list, [open a PR](https://github.com/withcatai/node-llama-cpp/edit/master/docs/guide/awesome.md).
+>
 > To have a project listed here, it should clearly state that it uses `node-llama-cpp`.

docs/guide/cmakeOptions.data.ts

@@ -90,6 +90,12 @@ function parseCmakeOptions(cmakeListsTxt: string, optionFilter: ((key: string) =
             }
         } else if (option.defaultValue === "${BUILD_SHARED_LIBS_DEFAULT}")
             option.defaultValue = htmlEscapeWithCodeMarkdown("`OFF` on MinGW, `ON` otherwise");
+        else if (option.defaultValue === "${GGML_CUDA_GRAPHS_DEFAULT}")
+            option.defaultValue = htmlEscapeWithCodeMarkdown("`ON`");
+        else if (option.defaultValue === "${GGML_NATIVE_DEFAULT}")
+            option.defaultValue = htmlEscapeWithCodeMarkdown("`OFF` when building for a different architecture,\n`ON` otherwise");
+        else if (option.key === "LLAMA_CURL")
+            option.defaultValue = htmlEscapeWithCodeMarkdown("`OFF`");
         else
             option.defaultValue = htmlEscapeWithCodeMarkdown(
                 option.defaultValue != null

docs/guide/downloading-models.md

@@ -76,7 +76,7 @@ You can reference models using a URI instead of their full download URL when usi
 When downloading a model from a URI, the model files will be prefixed with a corresponding adaptation of the URI.
 
 To reference a model from Hugging Face, you can use one of these schemes:
-* `hf:<user>/<model>:<quant>` (`#<quant>` is optional, [but recommended](#hf-scheme-specify-quant))
+* `hf:<user>/<model>:<quant>` (`:<quant>` is optional, [but recommended](#hf-scheme-specify-quant))
 * `hf:<user>/<model>/<file-path>#<branch>` (`#<branch>` is optional)
 
 Here are example usages of the Hugging Face URI scheme:

docs/guide/index.md

@@ -316,4 +316,5 @@ Explore the [API reference](../api/functions/getLlama.md) to learn more about th
 and use the search bar (press <kbd class="doc-kbd">/</kbd>) to find documentation for a specific topic or API.
 
 Check out the [roadmap](https://github.com/orgs/withcatai/projects/1) to see what's coming next,<br/>
+visit the [awesome list](./awesome.md) to find great projects that use `node-llama-cpp`,<br/>
 and consider [sponsoring `node-llama-cpp`](https://github.com/sponsors/giladgd) to accelerate the development of new features.

llama/CMakeLists.txt

@@ -1,4 +1,4 @@
-cmake_minimum_required(VERSION 3.14)
+cmake_minimum_required(VERSION 3.19)
 
 if (NLC_CURRENT_PLATFORM STREQUAL "win-x64" OR NLC_CURRENT_PLATFORM STREQUAL "win-arm64")
     set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON)
@@ -70,7 +70,9 @@ include_directories(${NODE_ADDON_API_DIR} ${CMAKE_JS_INC})
 add_subdirectory("llama.cpp")
 include_directories("llama.cpp")
 include_directories("./llama.cpp/common")
-include_directories("./llama.cpp/src")
+
+# This is needed to use methods in "llama-grammar.h" and "unicode.h"
+target_include_directories(llama PUBLIC "./llama.cpp/src")
 
 unset(GPU_INFO_HEADERS)
 unset(GPU_INFO_SOURCES)

llama/addon/AddonContext.cpp

@@ -2,7 +2,6 @@
 #include <algorithm>
 #include <cmath>
 #include "common/common.h"
-#include "llama-grammar.h"
 #include "llama.h"
 
 #include "addonGlobals.h"