Local Multimodal LLM on iOS with `llama.cpp` (Swift + ObjC++)

I want a real local pipeline: image in, structured JSON out, no cloud dependency. Optimized to run Metal / ANE or whatever apple exposes ? My goal is to infer a json-struct of variables from image using FM. Sounds simple, but it ain't so as of May 2026. And I really want it. After doing a bit of research, llama.cpp provides optimization and all the necesary low level work. I just need to make swift bindings that are worth the trouble... This is a complete tutorial on how i did it. i will use something like quickbooks / wise.com receipt capture example to make it real and safe. Bon courage! What We’re Building A local inference stack with clear separation of concerns: llama.cpp as an iOS XCFramework (vendor/llama.cpp/build-apple/llama.xcframework) Objective-C++ bridge (Controllers/LlamaBridge.h, Controllers/LlamaBridge.mm) Swift-facing API in Controllers/LLMFunctionsController.swift Typed decode API: let result: ReceiptResult = try await LLMFunctionsController.s