How to run a LLM local in your web app

Originally published on JavaScript in Plain English (Medium, Dec 2024).

Yes, it is possible. I am still learning the possibilities for developing with LLMs, but it’s obviously a piece of technology any builder can’t ignore now.

For context on what LLMs are, see What is a Large Language Model (LLM)? Explained in Plain English.

It is becoming the standard of most app stacks. We have the database, the client, the server, and now we have the LLMs. Not surprisingly, as with the others, the LLMs are just another technology that exists both as paid and proprietary code and as open-source. You can decide on using it as a service or hosting (or serving in the browser?) yourself.

This is Caleb blowing my mind with his X post

Have you heard about Hugging Face?

Truthfully, six months ago, I hadn’t. And it’s astonishing, as there is a massive community of open-source models you can interact with or even run on their own platform 🤯. It goes from very specific ML models to the more generalist LLMs.

And by navigating through this I realized that these small models are extremely applicable. At their current state, take the qwen-2.5 code interpreter, some of their versions (I’ll explain it in a bit) can compete shoulder-to-shoulder with SOTA (State of The Art) models.

What I mean by versions is the set of different models that have undertaken the same training procedure but with differences, such as the number of parameters it has — check this article on why the number of parameters matters.

So.. how do I run it in the browser?

Okay, okay. I will tell you, you came this far after all.

“Show me the code and I’ll tell you who you are” — said Shakespeare in a dimension he was a builder. So here it is the repository for the post from Caleb above: https://github.com/cfahlgren1/qwen-2.5-code-interpreter.

The most expressive snippet is this one:

It uses this package called mlc-ai/web-llm

WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU.

And it’s actually quite easy to load the models. And use them in the browser. But yes, it will likely crash your browser eventually. Even the tiny 1.5B qwen model caused my tab to crash after several interactions.

But not everything is lost, this community is booming and the recently published SmolLM2–1.7B model is said to be able to run in the browser.

I actually tried replacing it in the repo above, and it worked like a charm, not as precise as the qwen-1.5B but it didn’t crash my browser tab.

The point is

It’s exciting times to be a developer folks. We had an extremely powerful piece to add to our applications and it doesn’t come just in the form of an API from OpenAI. It has many shapes and forms, and different applications can potentially use many different models for different tasks. Some self-hosted, some from the powerful frontier models and some.. in the browser.

One thing that excites me a lot about local models is privacy; the data doesn’t go anywhere; it’s being processed in your computer.

If you are curious, take the repo above for a spin. It’s actually simple code but mighty code. And it might spark some ideas for your next projects, as it did to me.

By the way, I’ve just launched a Substack called Build First. This is a place where I’ll share my “bigger thoughts” on software development, artificial intelligence, and building early-stage startups. If this interests you, feel free to check it out!

I will keep posting on Medium for the bread and butter of software development.

That’s it for today.