Halkali merkez neighborhood fatih st ozgur apt no 46 , Kucukcekmece , Istanbul , 34303 , TR
llama.cpp is a lightweight C/C++ inference runtime for GGUF models across CPU, GPU and hybrid offload. The current `llama-server` supports OpenAI-compatible chat/responses/embeddings routes, reranking, parallel decoding and monitoring.
No. The project supports NVIDIA CUDA, AMD HIP, Metal, Vulkan, SYCL and other backends, including partial CPU+GPU offload. This is useful on heterogeneous hardware but may have higher latency than full-GPU serving.
The HTTP server accepts OpenAI-compatible requests, slot/continuous-batching logic manages contexts and GGUF layers run on CPU and/or GPU according to the selected backend.
Quantization and broad backend support let it run from small CPU systems to Apple Silicon and NVIDIA/AMD GPU hosts.
The runtime supports chat, embeddings and reranking, but production often benefits from separate instances for different models and latency profiles.
Replace the model reference with your GGUF model.
Turkey (Turkish)
Germany (German)
Worldwide (English)
llama-server -hf ggml-org/Qwen3.5-0.8B-GGUF --host 127.0.0.1 --port 8080curl -s http://127.0.0.1:8080/v1/models | headcurl -s http://127.0.0.1:8080/health | headps aux | grep '[l]lama-server'ss -lntp | grep ':8080'The built-in API key provides basic authentication; Internet-facing production needs TLS, rate limits, request-size limits, access logging and tenant quotas.
Yes. Current llama-server docs list OpenAI-compatible chat completions, responses and embeddings routes.
Yes. Current server docs include reranking endpoints and a dedicated reranking mode.
More aggressive quantization reduces memory but can reduce quality depending on model/task; benchmark it.
Share GGUF model, quantization, context, CPU/RAM, GPU/VRAM and users; we can design full-GPU or hybrid serving.