top | item 46844410

Coding Agent VMs on NixOS with Microvm.nix

109 points| secure | 1 month ago |michael.stapelberg.ch

53 comments

order

the_harpia_io|27 days ago

The sandbox-or-not debate is important but it's only half the picture. Even a perfectly sandboxed agent can still generate code with vulnerabilities that get deployed to production - SQL injection, path traversal, hardcoded secrets, overly permissive package imports.

The execution sandbox stops the agent from breaking out during development, but the real risk is what gets shipped downstream. Seeing more tools now that scan the generated code itself, not just contain the execution environment.

nh2|27 days ago

I find that a bit of a weird point.

The goal of such sandboxing is that you can allow the agent to freely write/execute/test code during development, so that it can propose a solution/commit without the human having to approve every dangerous step ("write a Python file, then execute it" is already a dangerous step). As the post says: "To safely run a coding agent without review".

You would then review the code, and use it if it's good. Turning many small reviews where you need to be around and babysit every step into a single review at the end.

What you seem to be asking for (shipping the generated code to production without review) is a completely different goal and probably a bad idea.

If there really were a tool that can "scan the generated code" so reliably that it is safe to ship without human review, then that could just be part of the tool that generates the code in the first place so that no code scanning would be necessary. Sandboxing wouldn't be necessary either then. So then sandboxing wouldn't be "half the picture"; it would be unnecessary entirely, and your statement simplifies to "if we could auto-generate perfect code, we wouldn't need any of this".

ryanrasti|27 days ago

Precisely! There's a fundamental tension: 1. Agents need to interact with the outside world to be useful 2. Interacting with the outside world is dangerous

Sandboxes provide a "default-deny policy" which is the right starting point. But, current tools lack the right primitives to make fine grained data-access and data policy a reality.

Object-capabilities provide the primitive for fine-grained access. IFC (information flow control) for dataflow.

mystifyingpoi|27 days ago

> not just contain the execution environment.

See, my typical execution environment is a Linux vm or laptop, with a wide variety of SSH and AWS keys configured and ready to be stolen (even if they are temporary, it's enough to infiltrate prod, or do some sneaky lateral movement attack). On the other hand, typical application execution environment is an IAM user/role with strictly scoped permissions.

rootnod3|27 days ago

That is quite an involved setup to get a costly autocomplete going.

Is that really where we are at? Just outsource convenience to a few big players that can afford the hardware? Just to save on typing and god forbid…thinking?

“Sorry boss, I can’t write code because cloudflare is down.”

Cyph0n|27 days ago

Keep in mind that this setup is a one-time cost. Also, a lot of the code is related to configuring it the way the author wants it (via Home Manager).

Generally speaking, once you have a working NixOS config, incremental changes become extremely trivial, safe, and easy to rollback.

groby_b|26 days ago

If you believe "costly autocomplete" is all you get, you absolutely shouldn't bother.

You're opting for "sorry boss, it's going to take me 10 times as long, but it's going to be loving craftsmanship, not industrial production" instead. You want different tools, for a different job.

0xcb0|27 days ago

I was looking for a way to isolate my agents in a more convenient way, and I really love your idea. I'm going to give this a try over the weekend and will report back.

But the one-time setup seems like a really fair investment for having a more secure development. Of course, what concerns the problem of getting malicious code to production, this will not help. But this will, with a little overhead, I think, really make development locally much more secure.

And you can automate it a lot. And it will be finally my chance to get more into NixOS :D

NJL3000|27 days ago

A pair of containers felt a bit cheaper than a VM:

https://github.com/5L-Labs/amp_in_a_box

I was going to add Gemini / OpenCode Kilo next.

There is some upfront cost to define what endpoints to map inside, but it definitely adds a veneer of preventing the crazy…

phrotoma|27 days ago

One problem with using containers as an isolation environment for a coding assistant is that it becomes challenging to have the agent work on a containerized project. You often need some janky "docker-in-docker" nonsense that hampers efforts.

giancarlostoro|27 days ago

This brings me back to my college days. We had Windows, and Deep Freeze. Students could do anything on the computer, we restart it and its all wiped and new. How long before Deep Freeze realizes they could sell their tool to Vibe Coders, they have Deep Freeze for Mac but not for Linux, funnily enough.

mxs_|27 days ago

I there a way to make this work with macOS hosts, preferably without having to install a Linux toolchain inside the VM for the language the agent will be writing code in?

ghxst|27 days ago

I'm working on a shared remote box for AI assisted development, will definitely look at this for some inspiration.

messh|27 days ago

I use shellbox.dev to create sandboxes through ssh, without ever leaving the terminal

heliumtera|27 days ago

Couldn't you replicate all of your setup with qemu microvm?

Without nix I mean

rictic|27 days ago

Yep. What nix adds is a declarative and reproducible way to build customized OS images to boot into.

clawsyndicate|1 month ago

we run ~10k agent pods on k3s and went with gvisor over microvms purely for density. the memory overhead of a dedicated kernel per tenant just doesn't scale when you're trying to pack thousands of instances onto a few nodes. strict network policies and pid limits cover most of the isolation gaps anyway.

secure|1 month ago

Yeah, when you run ≈10k agents instead of ≈10, you need a different solution :)

I’m curious what gVisor is getting you in your setup — of course gVisor is good for running untrusted code, but would you say that gVisor prevents issues that would otherwise make the agent break out of the kubernetes pod? Like, do you have examples you’ve observed where gVisor has saved the day?

alexzenla|27 days ago

This is a big reason for our strategy at Edera (https://edera.dev) of building hypervisor technology that eliminates the standard x86/ARM kernel overhead in favor of deep para-virtualization.

The performance of gVisor is often a big limiting factor in deployment.

souvik1997|27 days ago

Hey @clawsyndicate I'd love to learn more about your use case. We are working on a product that would potentially get you the best of both worlds (microVM security and containers/gVisor scalability). My email is in my profile.

dist-epoch|27 days ago

LXC containers inside a VM scales. bonus point that LXC containers feel like a VM.