[Docker Agent] Work with llama.cpp

Prerequisites

Install llama.cpp

https://llama.app/

curl -LsSf https://llama.app/install.sh | sh

Start llama.cpp and download a model

Run this:

llama serve -hf JetBrains/Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M:Q4_K_M

Check the llama.cpp server

# Server health
curl http://localhost:8080/health
# {"status":"ok"}

# Exact name of the exposed model
curl http://localhost:8080/v1/models

# First chat completion
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "JetBrains/Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M:Q4_K_M",
    "messages": [{"role": "user", "content": "Say hello in one short sentence."}],
    "max_tokens": 800
  }'

# {"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Hello!"}}], ...

Install Docker Agent

🚧 work in progress

Create a Docker Agent configuration

agent.yaml:

providers:
  llamacpp:
    api_type: openai_chatcompletions
    base_url: http://localhost:8080/v1
    #base_url: http://host.docker.internal:8080/v1

models:
  mellum2:
    provider: llamacpp
    model: JetBrains/Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M:Q4_K_M
    #max_tokens: 8192
    temperature: 0.7
    provider_opts:
      context_size: 262144

agents:
  root:
    model: mellum2
    description: A helpful AI assistant running on a local llama.cpp server
    instruction: |
      You name is Bob 🤓, you are a knowledgeable code assistant that helps users with various tasks.
      Be helpful, accurate, and concise in your responses.
      You have access to the local filesystem and shell: use these tools
    welcome_message: |
      🤖 Local Assistant propulsed by **llama.cpp** 🦙
      
    toolsets:
      - type: filesystem
      - type: shell 

Run Docker Agent

docker agent run agent.yaml

or docker-agent run agent.yaml depending your installation