My local chat app was fast every single time I tested it, and slow every single time I actually used it.
That's the tell, and I ignored it for weeks. I'd type a question during development, get a first token in under a second, ship the change. Then I'd come back after lunch, ask one thing, and sit there for eleven seconds watching a cursor blink. Same box. Same model. Same prompt.
It wasn't the model being slow. It was Ollama loading the model off disk again, because it had quietly evicted it while I was living my life. The knob is called keep_alive, and Ollama keep_alive turns out to have three separate ways of not doing what you think it does.
So I put a timer on every request for 24 hours. 1,180 requests, 214 model load events. Here's the autopsy.
TL;DR
Ollama unloads a model after 5 minutes idle by default. Any request arriving after that pays a full cold load from disk.
On my box that cold load cost 11.4s to first token vs 0.9s warm — 18.1% of my requests were co
Discussion
Your thoughts matter!
Your input is valuable—be the first to share it!