Skip to content

Commit 597d329

Browse files
SanftMonsterSanftMonster
authored
docs: update README.md
1 parent 17ec890 commit 597d329

File tree

1 file changed

+72
-49
lines changed

1 file changed

+72
-49
lines changed

‎README.md‎

Lines changed: 72 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -6,22 +6,50 @@
66
[![LLamaSharp Badge](https://img.shields.io/nuget/v/LLamaSharp.Backend.Cpu?label=LLamaSharp.Backend.Cpu)](https://www.nuget.org/packages/LLamaSharp.Backend.Cpu)
77
[![LLamaSharp Badge](https://img.shields.io/nuget/v/LLamaSharp.Backend.Cuda11?label=LLamaSharp.Backend.Cuda11)](https://www.nuget.org/packages/LLamaSharp.Backend.Cuda11)
88
[![LLamaSharp Badge](https://img.shields.io/nuget/v/LLamaSharp.Backend.Cuda12?label=LLamaSharp.Backend.Cuda12)](https://www.nuget.org/packages/LLamaSharp.Backend.Cuda12)
9-
[![LLamaSharp Badge](https://img.shields.io/nuget/v/LLamaSharp.Backend.MacMetal?label=LLamaSharp.Backend.MacMetal)](https://www.nuget.org/packages/LLamaSharp.Backend.MacMetal)
109
[![LLamaSharp Badge](https://img.shields.io/nuget/v/LLamaSharp.semantic-kernel?label=LLamaSharp.semantic-kernel)](https://www.nuget.org/packages/LLamaSharp.semantic-kernel)
10+
[![LLamaSharp Badge](https://img.shields.io/nuget/v/LLamaSharp.kernel-memory?label=LLamaSharp.kernel-memory)](https://www.nuget.org/packages/LLamaSharp.kernel-memory)
1111

1212

1313
**The C#/.NET binding of [llama.cpp](https://github.com/ggerganov/llama.cpp). It provides higher-level APIs to inference the LLaMA Models and deploy it on local device with C#/.NET. It works on
1414
both Windows, Linux and MAC without requirment for compiling llama.cpp yourself. Even without GPU or not enought GPU memory, you can still apply LLaMA models well with this repo. 🤗**
1515

1616
**Furthermore, it provides integrations with other projects such as [semantic-kernel](https://github.com/microsoft/semantic-kernel), [kernel-memory](https://github.com/microsoft/kernel-memory) and [BotSharp](https://github.com/SciSharp/BotSharp) to provide higher-level applications.**
1717

18+
<details>
19+
<summary>Table of Contents</summary>
20+
<ul>
21+
<li><a href="#Documentation">Documentation</a></li>
22+
<li><a href="#Examples">Examples</a></li>
23+
<li><a href="#Installation">Installation</a></li>
24+
<li>
25+
<a href="#(Quick Start)">Quick Start</a>
26+
<ul>
27+
<li><a href="#Model Inference and Chat Session">Model Inference and Chat Session</a></li>
28+
<li><a href="#Quantization">Quantization</a></li>
29+
<li><a href="#Web API">Web API</a></li>
30+
</ul>
31+
</li>
32+
<li><a href="#Features">Features</a></li>
33+
<li><a href="#Console Demo">Console Demo</a></li>
34+
<li><a href="#FAQ">FAQ</a></li>
35+
<li><a href="#Contributing">Contributing</a></li>
36+
<li><a href="#Contact us">Contact us</a></li>
37+
<li>
38+
<a href="#Apendix">Apendix</a>
39+
<ul>
40+
<li><a href="#Mapping from LLamaSharp to llama.cpp">Mapping from LLamaSharp to llama.cpp</a></li>
41+
</ul>
42+
</li>
43+
</ul>
44+
</details>
45+
1846

1947
## Documentation
2048

21-
- [Quick start](https://scisharp.github.io/LLamaSharp/0.5/GetStarted/)
22-
- [Tricks for FAQ](https://scisharp.github.io/LLamaSharp/0.5/Tricks/)
23-
- [Full documentation](https://scisharp.github.io/LLamaSharp/0.5/)
24-
- [API reference](https://scisharp.github.io/LLamaSharp/0.5/xmldocs/)
49+
- [Quick start](https://scisharp.github.io/LLamaSharp/latest/GetStarted/)
50+
- [Tricks for FAQ](https://scisharp.github.io/LLamaSharp/latest/Tricks/)
51+
- [Full documentation](https://scisharp.github.io/LLamaSharp/latest/)
52+
- [API reference](https://scisharp.github.io/LLamaSharp/latest/xmldocs/)
2553

2654
## Examples
2755
- [Official Console Examples](./LLama.Examples/NewVersion/)
@@ -65,34 +93,6 @@ LLamaSharp.kernel-memory
6593
In general, there may be some break changes between two minor releases, for example 0.5.1 and 0.6.0. On the contrary, we don't introduce API break changes in patch release. Therefore it's recommended to keep the highest patch version of a minor release. For example, keep 0.5.6 instead of 0.5.3.
6694

6795

68-
### Mapping from LLamaSharp to llama.cpp
69-
Here's the mapping of them and corresponding model samples provided by `LLamaSharp`. If you're not sure which model is available for a version, please try our sample model.
70-
71-
The llama.cpp commit id will help if you want to compile a DLL yourself.
72-
73-
| LLamaSharp.Backend | LLamaSharp | Verified Model Resources | llama.cpp commit id |
74-
| - | - | -- | - |
75-
| - | v0.2.0 | This version is not recommended to use. | - |
76-
| - | v0.2.1 | [WizardLM](https://huggingface.co/TheBloke/wizardLM-7B-GGML/tree/previous_llama), [Vicuna (filenames with "old")](https://huggingface.co/eachadea/ggml-vicuna-13b-1.1/tree/main) | - |
77-
| v0.2.2 | v0.2.2, v0.2.3 | [WizardLM](https://huggingface.co/TheBloke/wizardLM-7B-GGML/tree/previous_llama_ggmlv2), [Vicuna (filenames without "old")](https://huggingface.co/eachadea/ggml-vicuna-13b-1.1/tree/main) | 63d2046 |
78-
| v0.3.0, v0.3.1 | v0.3.0, v0.4.0 | [LLamaSharpSamples v0.3.0](https://huggingface.co/AsakusaRinne/LLamaSharpSamples/tree/v0.3.0), [WizardLM](https://huggingface.co/TheBloke/wizardLM-7B-GGML/tree/main) | 7e4ea5b |
79-
| v0.4.1-preview (cpu only) | v0.4.1-preview | [Open llama 3b](https://huggingface.co/SlyEcho/open_llama_3b_ggml), [Open Buddy](https://huggingface.co/OpenBuddy/openbuddy-llama-ggml)| aacdbd4 |
80-
| v0.4.2-preview (cpu,cuda11) |v0.4.2-preview | [Llama2 7b GGML](https://huggingface.co/TheBloke/llama-2-7B-Guanaco-QLoRA-GGML)| 3323112 |
81-
| v0.5.1 | v0.5.1 | [Llama2 7b GGUF](https://huggingface.co/TheBloke/llama-2-7B-Guanaco-QLoRA-GGUF)| 6b73ef1 |
82-
| v0.6.0 | v0.6.0 | | [cb33f43](https://github.com/ggerganov/llama.cpp/commit/cb33f43a2a9f5a5a5f8d290dd97c625d9ba97a2f) |
83-
| v0.7.0 | v0.7.0 | [Thespis-13B](https://huggingface.co/TheBloke/Thespis-13B-v0.5-GGUF/tree/main?not-for-all-audiences=true), [LLaMA2-7B](https://huggingface.co/TheBloke/llama-2-7B-Guanaco-QLoRA-GGUF) | [207b519](https://github.com/ggerganov/llama.cpp/commit/207b51900e15cc7f89763a3bb1c565fe11cbb45d) |
84-
85-
Many hands make light work. If you have found any other model resource that could work for a version, we'll appreciate it for opening an PR about it! 😊
86-
87-
88-
## FAQ
89-
90-
1. GPU out of memory: Please try setting `n_gpu_layers` to a smaller number.
91-
2. Unsupported model: `llama.cpp` is under quick development and often has break changes. Please check the release date of the model and find a suitable version of LLamaSharp to install, or generate `gguf` format weights from original weights yourself.
92-
3. Cannot load native lirary: 1) ensure you installed one of the backend packages. 2) Run `NativeLibraryConfig.WithLogs()` at the very beginning of your code to print more informations. 3) check if your system supports avx2, which is the default settings of official runtimes now. If not, please compile llama.cpp yourself and specify it with `NativeLibraryConfig.WithLibrary`.
93-
94-
95-
9696
## Quick Start
9797

9898
#### Model Inference and Chat Session
@@ -164,16 +164,6 @@ We provide [the integration of ASP.NET core](./LLama.WebAPI) and a [web app demo
164164

165165
Since we are in short of hands, if you're familiar with ASP.NET core, we'll appreciate it if you would like to help upgrading the Web API integration.
166166

167-
## Console Demo
168-
169-
![demo-console](Assets/console_demo.gif)
170-
171-
## How to Find a Model
172-
173-
Models in format `gguf` are valid for LLamaSharp (and `ggml` before v0.5.1). If you're new to LLM/LLaMA, it's a good choice to search `LLama` and `gguf` on [huggingface](https://huggingface.co/) to find a model.
174-
175-
Another choice is generate gguf format file yourself with a pytorch weight (or any other), pleae refer to [convert.py](https://github.com/ggerganov/llama.cpp/blob/master/convert.py) and [convert-llama-ggml-to-gguf.py](https://github.com/ggerganov/llama.cpp/blob/master/convert-llama-ggml-to-gguf.py) to get gguf file through a ggml transformation.
176-
177167
## Features
178168

179169
---
@@ -206,22 +196,55 @@ Another choice is generate gguf format file yourself with a pytorch weight (or a
206196

207197
🔳 MAUI Integration
208198

199+
## Console Demo
200+
201+
![demo-console](Assets/console_demo.gif)
202+
203+
## FAQ
204+
205+
1. GPU out of memory: Please try setting `n_gpu_layers` to a smaller number.
206+
2. Unsupported model: `llama.cpp` is under quick development and often has break changes. Please check the release date of the model and find a suitable version of LLamaSharp to install, or generate `gguf` format weights from original weights yourself.
207+
3. Cannot load native lirary: 1) ensure you installed one of the backend packages. 2) Run `NativeLibraryConfig.WithLogs()` at the very beginning of your code to print more informations. 3) check if your system supports avx2, which is the default settings of official runtimes now. If not, please compile llama.cpp yourself and specify it with `NativeLibraryConfig.WithLibrary`.
208+
4. How to find a model: Models in format `gguf` are valid for LLamaSharp (and `ggml` before v0.5.1). If you're new to LLM/LLaMA, it's a good choice to search `LLama` and `gguf` on [huggingface](https://huggingface.co/) to find a model. Another choice is generate gguf format file yourself with a pytorch weight (or any other), pleae refer to [convert.py](https://github.com/ggerganov/llama.cpp/blob/master/convert.py) and [convert-llama-ggml-to-gguf.py](https://github.com/ggerganov/llama.cpp/blob/master/convert-llama-ggml-to-gguf.py) to get gguf file through a ggml transformation.
209+
210+
209211
## Contributing
210212

211-
Any contribution is welcomed! Please read the [contributing guide](https://scisharp.github.io/LLamaSharp/0.4/ContributingGuide/). You can do one of the followings to help us make `LLamaSharp` better:
213+
Any contribution is welcomed! There's a TODO list in [LLamaSharp Dev Project](https://github.com/orgs/SciSharp/projects/5) and you could pick an interested one to start. Please read the [contributing guide](https://scisharp.github.io/LLamaSharp/latest/ContributingGuide/) for more informations.
214+
215+
You can also do one of the followings to help us make LLamaSharp better:
212216

213-
- Append a model link that is available for a version. (This is very important!)
214-
- Star and share `LLamaSharp` to let others know it.
215-
- Add a feature or fix a BUG.
217+
- Submit a feature request.
218+
- Star and share LLamaSharp to let others know it.
219+
- Write a blog or demo about LLamaSharp.
216220
- Help to develop Web API and UI integration.
217-
- Just start an issue about the problem you met!
221+
- Just open an issue about the problem you met!
218222

219223
## Contact us
220224

221-
Join our chat on [Discord](https://discord.gg/7wNVU65ZDY).
225+
Join our chat on [Discord](https://discord.gg/7wNVU65ZDY) (please contact Rinne to join the dev channel if you want to be a contributor).
222226

223227
Join [QQ group](http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=sN9VVMwbWjs5L0ATpizKKxOcZdEPMrp8&authKey=RLDw41bLTrEyEgZZi%2FzT4pYk%2BwmEFgFcrhs8ZbkiVY7a4JFckzJefaYNW6Lk4yPX&noverify=0&group_code=985366726)
224228

229+
## Apendix
230+
231+
### Mapping from LLamaSharp to llama.cpp
232+
Here's the mapping of them and corresponding model samples provided by `LLamaSharp`. If you're not sure which model is available for a version, please try our sample model.
233+
234+
The llama.cpp commit id will help if you want to compile a DLL yourself.
235+
236+
| LLamaSharp | Verified Model Resources | llama.cpp commit id |
237+
| - | -- | - |
238+
| v0.2.0 | This version is not recommended to use. | - |
239+
| v0.2.1 | [WizardLM](https://huggingface.co/TheBloke/wizardLM-7B-GGML/tree/previous_llama), [Vicuna (filenames with "old")](https://huggingface.co/eachadea/ggml-vicuna-13b-1.1/tree/main) | - |
240+
| v0.2.2, v0.2.3 | [WizardLM](https://huggingface.co/TheBloke/wizardLM-7B-GGML/tree/previous_llama_ggmlv2), [Vicuna (filenames without "old")](https://huggingface.co/eachadea/ggml-vicuna-13b-1.1/tree/main) | 63d2046 |
241+
| v0.3.0, v0.4.0 | [LLamaSharpSamples v0.3.0](https://huggingface.co/AsakusaRinne/LLamaSharpSamples/tree/v0.3.0), [WizardLM](https://huggingface.co/TheBloke/wizardLM-7B-GGML/tree/main) | 7e4ea5b |
242+
| v0.4.1-preview | [Open llama 3b](https://huggingface.co/SlyEcho/open_llama_3b_ggml), [Open Buddy](https://huggingface.co/OpenBuddy/openbuddy-llama-ggml)| aacdbd4 |
243+
|v0.4.2-preview | [Llama2 7b GGML](https://huggingface.co/TheBloke/llama-2-7B-Guanaco-QLoRA-GGML)| 3323112 |
244+
| v0.5.1 | [Llama2 7b GGUF](https://huggingface.co/TheBloke/llama-2-7B-Guanaco-QLoRA-GGUF)| 6b73ef1 |
245+
| v0.6.0 | | [cb33f43](https://github.com/ggerganov/llama.cpp/commit/cb33f43a2a9f5a5a5f8d290dd97c625d9ba97a2f) |
246+
| v0.7.0, v0.8.0 | [Thespis-13B](https://huggingface.co/TheBloke/Thespis-13B-v0.5-GGUF/tree/main?not-for-all-audiences=true), [LLaMA2-7B](https://huggingface.co/TheBloke/llama-2-7B-Guanaco-QLoRA-GGUF) | [207b519](https://github.com/ggerganov/llama.cpp/commit/207b51900e15cc7f89763a3bb1c565fe11cbb45d) |
247+
225248
## License
226249

227250
This project is licensed under the terms of the MIT license.

0 commit comments

Comments
 (0)