llama.cpp (OVHai Olala fork)
llama-server and llama-cli on the PATH come from the OVHai fork of llama.cpp
(/opt/llama-cpp-olala, commit in /opt/llama-cpp-olala/REVISION), CPU only.
They load olala GGUF models, which upstream llama.cpp cannot.
- Serve:
llama-server -m <file.gguf> -t <threads> -fa 1 -nr -ctk q8_0 -ctv q8_0 -c 32768 -np 4 --host 0.0.0.0 --port 8080 - In this project, prefer
scripts/olala.sh up|serve|stop|status(downloads the model if needed). - Recommended sampling:
temperature=0.8,top_p=0.9,top_k=40;reasoning_effort: disabled, low, medium, high.
1 2 3 4 5 6 7 8 9 |
|