Cognition as a service: LLMs as infrastructure

Originally published on Build First (Substack, Dec 2024).

Introduction

When building modern applications, it’s common to pick a database, a language framework, a cloud provider, a UI library Kit, and so on. These are all Lego pieces that one needs to fit together for the problem the application aims to solve. For example, suppose you are building a powerful search feature in your application. In that case, you might consider using ElasticSearch or another NoSQL database to bring performance for that while still using a regular SQL database for the other parts of the application.

Not so different is the transformation of AI. Most of the mainstream focus is on the frontier models and the use cases that their companies are going for. For applications that want to add AI to their stack, it becomes another Lego piece to choose from. While the top-shelf models (Sonnet, GPT 4o, GPT o1) are great for advanced reasoning, they are expensive and not necessarily the best fit for every use case.

Cognition as a Service (CaaS)

In the same vein, databases can be seen as ‘storage as a service’ or cloud computing as ‘infrastructure as a service’; I challenge you to think of LLMs as ‘Cognition as Service’.

While building software that is strongly tied to AI, I face the same type of decisions I would when choosing my database, framework, or pretty much any other part.

Does it work well? Is it easy to integrate? Is it fast? How well-maintained is this? Is it open source so I can self-host if I need to? Does it address privacy issues?

You can see where I am going with this. LLMs, when you zoom-out of their internals, are yet another part of our Software machinery, a strong one, that enable us to build the next generation of apps. But nonetheless it shares much of the same principles than other architectural parts of modern software.

How can LLMs be integrated into apps

Use cases are vast. But let’s look at some of the most popular ones and some sneak features you might not even notice AI present:

• Chats: The most popular pattern is probably the Chats, which works in a question-answer basis. This is likely the most frequent and direct user-ai interaction a application can provider. Here the user has direct access to the reasoning of the LLM.

• Summarizers: Select a text and ask the AI to summarize that information. Here, the pattern is a more direct input-output where the user won’t know the reasoning of the LLM. It will just get an output.

• Code-generators: This can come in many forms. Either via chat or in-the-editor where it can auto-complete what the user is typing. Likely there won’t be any insights in the LLM reasoning.

• Background tasks: AI in the form of background tasks often operates invisibly to the user, enhancing functionality without requiring direct interaction. These tasks include activities like data classification, sorting, and enriching user-provided data.

There are many other use cases. But you can see that, going from chats to code-generators, the user has different levels of insight into the AI’s reasoning. Where in the Chat, the LLM feels very much like an external part of the application, for summarizers and code-generators, if done well, it can feel like a single piece; the integration of AI into the software is much more subtle.

Furthermore, there are instances of AI integrated into software where the user won’t even notice, because the AI was used to pre-process some data the user is interacting with. This could include a pre-summarization of the content. Imagine a task manager that summarizes the content of tasks into concise headlines so the user can immediately understand what a task is about from the list view. This transformation, powered by AI, operates internally, delivering a better UX while masking the presence of the LLM entirely.

How to choose a LLM

This framework is a work in progress. But one thing is for sure: the decision comes from your privacy requirements, and then it trickles down to everything else.

If privacy is a critical concern, where sending client data to a third party such as OpenAI or Anthropic is a deal breaker, then you are certainly looking towards on-premise open-source models.

Before going deep on this, let’s look at some of the main categories one can choose from:

Proprietary models

Models that aren’t open source and cannot be self-hosted:

• Frontier models: these are the latest released models by the main players such as Claude Sonnet, GPT 4o, and o1. They are the most accurate, easy-to-use, and most expensive models to pick from.

• Frontier tier 2: these models are also very capable but they are not the strongest offering of the provider. These could be Claude Haiku 3.5 or GPT 4o mini, Gemini 1.5 Flash. They are significantly cheaper than the first tier and still pretty powerful.

Open-source models

These models aren’t as powerful as the frontier models (yet) but they are strong enough for a variety of tasks, and they can be customized and trained for specific tasks.

Here, I won’t differentiate the models in a category because it’s very (very!) vast. You can find from generalist LLMs to specialized text-to-code or text-to-image models.

The important part is how you can service these models, which will often come with 3 options:

• On-premise open-source models: pick an open-source model that fits your task and host within your infrastructure.

• Managed open-source models: pick an open-source model and use a service provider to host and manage it for you. Some examples of such services are:

  • https://openrouter.ai/
  • https://groq.com/

---

In summary, when choosing a LLM, I’d think about these first:

• What are my privacy concerns?

  • Low: pick from proprietary models

  • High: look into on-premise open-source models

• How complex is my task? (no to low privacy concern)

  • Low: use a cheap tier 2 frontier model or a managed open-source model

  • High: use a top-tier frontier model

• How complex is my task? (high privacy concern)

  • Low: on-premise open-source model that fits the task (explore options through a managed service first)

  • High: on-premise state-of-art open-source models, and consider fine-tuning the task at hand.

The tl;dr is that privacy not being a deal breaker, I’d always start from the best model and then optimize for cost vs accuracy. I’d keep going down the list of cheaper frontier models until I find one that does the job at a minimal cost. Privacy being critical, I’d start from managed open-source models to figure out what also works best at a minimal cost and eventually go for an on-premise solution.

Conclusion

I believe, as a software engineer, a founder, or a ‘technologist,’ it is important we develop our language around LLMs as a new infrastructure piece of the puzzle. Not seeing it as an isolated part from traditional application development, but rather an intrinsic part of it.

The result isn’t just in the infrastructure, cost, and privacy—it is the user experience. It means the software we build can be truly smart, sometimes explicitly, sometimes implicitly. But at the end of the day, we can enable users to complete their tasks better and faster.

LLMs are first-class citizens in the modern app stack. It’s a matter of choosing your weapons wisely but avoiding paralysis. As with everything in software, applications are made to change, and early optimizations are the root of all evil.

By thoughtfully integrating LLMs, we have the opportunity to redefine what software can do, making intelligence as seamless and essential as any other system component.