File tree Expand file tree Collapse file tree 1 file changed +6
-3
lines changed Expand file tree Collapse file tree 1 file changed +6
-3
lines changed Original file line number Diff line number Diff line change @@ -32,7 +32,10 @@ $ gem install llama_cpp -- --with-opt-dir=/opt/homebrew
32
32
## Usage
33
33
34
34
Prepare the quantized model by refering to [ the usage section on the llama.cpp README] ( https://github.com/ggerganov/llama.cpp#usage ) .
35
- For example, preparing the quatization model based on [ open_llama_7b] ( https://huggingface.co/openlm-research/open_llama_7b ) is as follows:
35
+ For example, you could prepare the quatization model based on
36
+ [ open_llama_7b] ( https://huggingface.co/openlm-research/open_llama_7b )
37
+ or more useful in the context of Ruby might be a smaller model such as
38
+ [ tiny_llama_1b] ( https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0 ) :
36
39
37
40
``` sh
38
41
$ cd ~ /
@@ -44,9 +47,9 @@ $ python3 -m pip install -r requirements.txt
44
47
$ cd models
45
48
$ git clone https://huggingface.co/openlm-research/open_llama_7b
46
49
$ cd ../
47
- $ python3 convert.py models/open_llama_7b
50
+ $ python3 convert-hf-to-gguf .py models/open_llama_7b
48
51
$ make
49
- $ ./quantize ./models/open_llama_7b/ggml-model-f16.gguf ./models/open_llama_7b/ggml-model-q4_0.bin q4_0
52
+ $ ./llama- quantize ./models/open_llama_7b/ggml-model-f16.gguf ./models/open_llama_7b/ggml-model-q4_0.bin q4_0
50
53
```
51
54
52
55
An example of Ruby code that generates sentences with the quantization model is as follows:
You can’t perform that action at this time.
0 commit comments