To find out, we connected a key vault to a language model running entirely locally on a standard workstation. The idea: the model should understand natural language and turn it into concrete actions – finding a specific entry, for example, or importing values from a configuration file.
The goal was never to build the most capable AI assistant possible. We wanted to find out how far a comparatively small local model gets in practical use – and which tasks are better secured by classic software logic.
The result is interesting: for clearly bounded tasks, local models are already surprisingly capable. At the same time, their limits show very clearly.
Why a local language model at all?
Modern software projects accumulate credentials quickly: database keys, API keys, tokens for external services, webhook secrets, or credentials for deployment and email delivery.
This information has to be stored and managed securely. At the same time, you want to find it again quickly in day-to-day work.
A classic password or key vault solves storing and searching. A language model adds a natural way of working with that information.
Instead of clicking through lists, you could simply ask:
"Which key belongs to Supabase?"
Or:
"From this configuration file, take only the database URL and the Stripe key."
With a cloud-based AI assistant, that would mean transmitting sensitive information to an external service.
A local language model takes a different approach: processing happens directly on your own machine.
Which means the sensitive data does not have to leave the machine in order to be processed by the model.
The hardware: an ordinary workstation
No dedicated AI workstation was necessary for this experiment. We used an off-the-shelf Windows machine:
- Graphics card: NVIDIA GeForce RTX 5060 Ti, 16 GB VRAM
- Operating system: Windows 11 Pro
- Runtime: Ollama, Node.js
The most important number here is the VRAM.
VRAM is, put simply, the memory of the graphics card. A language model needs this memory to run its calculations quickly.
The larger the model, the more memory it needs. If a model does not fit entirely into the available graphics memory, parts of the computation have to be offloaded to the processor – which can slow things down considerably.
16 GB of VRAM therefore puts you in a range where capable local models can already be used sensibly.
The model: Qwen3 14B
For this experiment we used Qwen3 14B, an open language model from the Qwen model family.
The key figures:
- Parameters: 14.8 billion
- Quantization: Q4_K_M
- Model size: approx. 9.3 GB
- VRAM usage: approx. 10 GB
- Context window: 8,192 tokens
- Computation: fully on the GPU
What does "14 billion parameters" mean?
Parameters can be thought of, simplified, as the many adjustment screws a language model has learned during its training.
More parameters generally mean more capacity. But memory requirements grow along with them.
This is where quantization comes in.
A language model consists of billions of numbers. If those numbers are stored at high precision, the model needs a great deal of memory. Quantization represents them more compactly – in our case with roughly 4 bits.
That makes the model considerably smaller and lets it run on a normal graphics card. The associated loss in quality is barely relevant for our everyday use case.
Why this model?
Four points were decisive in the selection.
1. It can use tools
Our goal was not a classic chatbot that merely writes text.
The model was supposed to trigger actions. For example:
- search for an entry,
- select an entry,
- import data from a file,
- copy specific values to the clipboard.
This capability is commonly referred to as tool calling.
Put simply, the model is given a list of tools and what they do. When a user makes a request, the model decides which tool is needed and with which parameters it should be called.
Exactly this translation from natural language into structured actions is what matters for our use case.
2. The model fits into the available memory
The model needs around 10 GB of VRAM. On a card with 16 GB, that leaves headroom for further computation.
That matters because the model itself is not the only thing consuming memory. The so-called context – the content of the current conversation plus any additional information the model should take into account – needs memory too.
We therefore deliberately capped the possible context window at 8,192 tokens. For short questions to a key vault, a substantially larger context window is not necessary.
3. It works well in German
That sounds like a side note at first, but it matters for daily use.
A tool that can be operated naturally in German lowers the barrier for users and makes practical use more pleasant.
4. An open license
Qwen3 14B is released under the Apache 2.0 license. That makes it usable for our use case without an additional registration process.
What a local model does well
After several weeks of practical use, a clear picture emerges.
Translating natural language into concrete actions
This is the greatest strength.
A request such as
"Copy the Supabase key for me"
can be translated by the model into the necessary steps: search for the entry, select the matching one, and then perform the requested action.
More complex instructions work as well:
"From the env file, import only DATABASE_URL and STRIPE_SECRET_KEY."
The model can derive a structured import with the requested variables from that.
The decisive factor is that the toolbox stays manageable. In our case the model has eleven clearly defined tools at its disposal.
The more clearly the possible actions are described, the more reliably the model can distinguish between them.
Fast enough for everyday work
On the GPU, the model responds within a few seconds.
After longer pauses it can take a bit more time, because the model first has to be loaded back into graphics memory. In a normal workflow, though, that effect is barely disruptive.
This shows an important point: local AI does not have to be slow. With suitable hardware and an appropriately sized model, it can feel very direct to use.
Understanding imprecise questions
People do not always phrase things exactly the way a database expects.
A question such as
"What are the last 4 characters of the mistral-dev key?"
can still be matched by the model to the right entry – even when spelling, hyphenation, or the exact wording varies.
That is a substantial advantage over purely text-based search.
Local processing
Another advantage lies in the architecture.
The language model runs locally on the machine. The request is therefore processed in the same place where the sensitive information lives.
For applications where data protection and control over sensitive data are particularly important, that is a fundamentally interesting approach.
Where the limits are
Even more interesting than the strengths are the situations in which the model gets things wrong.
Because a language model is not a classic program. It does not work exclusively from fixed rules; based on its inputs, it produces the most probable fitting answer or action.
That can lead to surprising results.
Language models can invent details
After a successful import, for instance, the model reported specific IDs for the newly created entries.
The problem: those IDs were not part of the return value of the tool that had been used at all.
So the model had not read them from the actual data – it had added them.
Why does something like that happen?
A language model tries to formulate a plausible and complete answer. When information is missing, it may fill that gap with something that merely seems to fit.
The decisive insight is therefore:
A model must never be treated as a source for information it did not actually receive.
If a piece of information is relevant, it has to be passed explicitly from the application to the model.
Similar intentions can be confused
It gets more critical with actions that have side effects.
An instruction such as
"Remove all secrets, keep the key names"
can be interpreted in more than one way.
In our case, what was actually meant was to redact the values in a pasted text. Because of a bug in the input handling, however, the model did not receive the full context.
It therefore interpreted the request as an instruction to delete entries from the vault.
This illustrates a fundamental principle:
A language model can defend a wrong interpretation very convincingly.
Security-relevant actions must therefore not depend solely on whether the model understood a request correctly.
Critical actions need additional technical safeguards – confirmations, plausibility checks, and limits on repeated actions, for example.
Search has to cope with imprecision
The search function itself had to be adapted as well.
Our original search expected a fairly exact match. Depending on the request, however, the model passed in things like:
mistral-devmistral-dev key- the complete user question
To the user, this initially looked like an unreliable model.
In reality, part of the problem was in our search.
The solution was to make the search logic more robust: search terms are split into individual components and matches are weighted accordingly.
That is an important lesson for AI applications:
Not every imprecision should be left to the model. It is often better to make the software around the model more fault-tolerant.
A local model is not a security boundary
Perhaps the most important point of the entire project:
Local does not automatically mean secure.
A language model can, for example, receive text from a database or a file that itself contains instructions.
If that text says something along the lines of:
"Ignore the previous instructions."
a model may try to take that text into account in its decision.
Running the model locally does not solve that problem.
That is why the security architecture of our vault does not rest on the language model always reacting correctly or "well-behaved".
The safeguards sit outside the model:
- Write actions are confirmed before execution.
- The model can only propose critical actions.
- Sensitive values are not carried into the conversation context unnecessarily.
- Critical flows are verified automatically.
- Repeated or unusual actions are rate-limited.
- A large share of the code is explicitly dedicated to verification.
Out of 2,951 lines of code in total, for example, 487 lines are dedicated to verification and 51 concrete checks that run on every build.
The underlying principle is simple:
The model may help with operating the software. The application decides what is actually allowed to happen.
Why the model does not need network access
We deliberately chose not to give the model access to the internet.
As a result, it cannot check for itself whether a given API key is still valid with the respective provider, for instance.
For a key vault, though, that is not a necessary feature.
On the contrary: every additional external data access opens up another interface between the model and foreign content.
For our application, a clearly bounded feature set therefore makes more sense: the model can use the locally available tools, but cannot reach external services on its own.
What we take away from the project
After the practical trial, four points stand out.
1. Local models are already good enough for clearly bounded tasks
A model with around 14 billion parameters can handle demanding tasks on an off-the-shelf graphics card.
The prerequisite, however, is a clearly defined use case.
Short requests, a manageable toolbox, and clearly described actions are a good combination.
2. The real work is in the architecture
Choosing the model was only one part of the project.
At least as important were:
- robust tools,
- a fault-tolerant search,
- clear confirmation dialogs,
- input validation,
- security checks,
- tests and verification.
The model, in other words, is only one component.
The quality of an AI system comes primarily from the interplay between the model and the software that controls it.
3. The biggest advantage is control over sensitive data
Speed and cost are interesting side effects.
The decisive advantage of local processing lies elsewhere:
Sensitive information does not have to leave your own machine for a language model to work with it.
This property cannot be talked into an application after the fact. It has to be considered in the architecture from the start.
4. Take "local" literally
Finally, there was one insight that initially had nothing to do with the language model at all.
While reviewing the system, it turned out that certain folders such as "Documents" and "Desktop" were redirected into a corporate OneDrive on that machine.
A file that is supposedly stored "locally" can therefore still be synchronized automatically.
Anyone who wants to keep sensitive data local should therefore not only look at the application, but also check where files on their own system are actually stored and synchronized.
Conclusion
A local language model is not a magical AI agent that reliably and autonomously handles arbitrary tasks.
But it does not have to be.
For a clearly defined use case, even a comparatively small model can achieve a surprising amount. It can understand natural language, derive structured actions from it, and make more complex software considerably easier to operate.
The most important insight from our project is therefore less the choice of the specific model than the architecture around it:
The language model interprets the request. The software controls the execution.
Exactly this separation makes it possible to benefit from natural language without handing security-critical decisions over to a language model entirely.
And it also shows where local AI is particularly interesting today: in small, clearly bounded tasks where data protection, control, and local processing matter more than maximum model size.