myModel

myModel is a microservice that deploys local LLMs over HTTP, served through vLLM for inference and Quart for the Python API layer. It turns a GPU machine into a private model endpoint without depending on external providers.

The problem

Running a local LLM means wrangling vLLM launch flags, model paths, and context-length configurations by hand — and then exposing the result to the rest of the fleet over a stable HTTP interface. Without a service layer, each consumer re-implements its own launch script and ad-hoc API wrapper, and model swaps break callers that hard-coded the previous endpoint.

The approach

myModel wraps vLLM behind a Quart HTTP surface: one configuration file declares which models to serve and at what context lengths, the service starts them, and downstream consumers hit a stable OpenAI-compatible endpoint. Typed source with 10 tests and 9 documentation pages keeps the API contract checkable. CI gates each push. Pre-commit hooks are not configured yet — the typing and CI discipline are in place, but the hook layer is a known gap.

What it demonstrates

A stage-5 published service with CI and typed source demonstrates the operational discipline of running local inference as a real deployable service rather than a notebook experiment. The missing pre-commit configuration is honest scope: the contract surface is small and guarded by CI, and the hook layer will close when the service matures.