LocalAI

Go server wrapping various C++/Python inference backends (llama.cpp, whisper.cpp, diffusers) behind an OpenAI-compatible REST API. No database — models are loaded from a mounted volume and served largely stateless per-request.

Each replica is an independent, mostly-stateless inference worker, which scales fine for throughput by adding pods, but per-pod GPU/CPU/memory requirements for loaded models mean scaling is expensive and coarse-grained compared to a hosted API's elastic capacity.

Single pod with a persistent volume for the multi-gigabyte model cache; CPU-only images pull cleanly via mirror.gcr.io, but GPU variants need an NVIDIA base image outside the mirror. No built-in auth, so front it with a Nexlayer auth proxy pod if exposed beyond internal use.