Enterprise likes it, because it gives them more visibility and control over what's going on in the dev box. That gives more angles to control against e.g. exfiltration threats. It also makes dev environments much more homogeneous: you worry less about client machines. In principle you could have a fleet of stateless Chromebooks whose main function is ssh and http, all connecting to identical virtualized desktops that are easily provisioned and deprovisioned.
The advantage of docker based developer workflows is that it makes things repeatable. If you’ve ever tried to support a team installing their own dependencies you’ll understand the pain!
If everyone is working in docker in VS Code it’s not a huge jump to have them develop in remote VMs. Now your devs can get setup instantaneously and can use thin and light MacBook Airs instead of heavy MacBook Pros.
> Someone, anyone, tell me which hypervisor doesn't enforce a memory limit on VMs?
They don’t mean the memory usage is allowed to grow indefinitely.
They mean, if your laptop has 32 GB RAM, and if your OS and apps consume 20 GB, it’s using 20 GB RAM.
If a VM has 32 GB of vRAM allocated, and the OS is aggressively caching, that VM will consume 32 GB of RAM on the physical host, even if only 20 GB of vRAM is being used within that guest VM.
Of course, every good hypervisor will have a way to prevent this from happening, so the article does seem disingenuous on this point.
If you work from the same computer in the same place everyday there are no advantages to having a remote box... If you work 50:50 from office & home you start getting annoyed by either having a different environment at home & work or by having to drag your laptop around with you... If you're a digital nomad wanting to work while travelling through "cheaper" parts of the world it's less worrying to carry around a $500 laptop than a $4000 laptop...
> Someone, anyone, tell me which hypervisor doesn't enforce a memory limit on VMs?
Not to mention the popular virtualization platforms like kvm, vmware, and hyper-v all support ballooning[1][2] where you can define a base amount of memory and a max, and when the guest garbage collects its heap within the vm the memory is reclaimed by the hypervisor.
This also allows overcommitting to better utilize the resources that would otherwise site mostly idle, and trust me every cloud provider overcommits their hypervisors, I used to work for a major one but anyone can check the cpu steal time in top and see how thrashed their host is
Author doesn’t mention it but I wonder if tried or considered Nix/NixOS’s reproducible developer environments and ruled them out for any reason. I couldn’t tell from the article if there’s something unique to his requirements that disqualifies them.
Nix solves a different problem than Hocus. Nix lets you define a development environment, Hocus gives you a way to run it on a remote server. Right now we use Dockerfiles to let users define the packages they need in their dev env, but we would like to support Nix in the future too. Interestingly, you can use custom BuildKit syntax https://docs.docker.com/build/dockerfile/frontend/ to build Nix environments with Docker https://github.com/reproducible-containers/buildkit-nix, and that's probably what we will end up supporting.
I am curious how important 100GB dev environment support and efficiently binpacking dev environments and rat-holing on memory efficiency actually matters for the first X enterprise customers for Hocus.
Memory and CPU cores are cheap. Getting the UX, resource allocation/sizing/instance type, and general pattern for multi-tenant dev boxes seems more fruitful.
Would love to see any data supporting this prioritization, and what the answer is for Workstation / HEDT-type environments given the industry's current focus on AI.
TFA goes right into the core complexity and misses it.
"Dev environment" depends on what kind of work you are doing. What a web developer needs is worlds apart from what a driver developer needs in terms of capability, resourcing, isolation, and rebuilds. Trying to offer a single environment for anyone who builds anything using a computer, is analogous to a single environment for anyone who uses a screwdriver. An electrician is using the same tool, sure, but they're doing fundamentally different work from a kitchen installer or a plumber and have vastly different needs. The single environment with all the capabilities that any computer developer needs is called a "computer."
IMO this is the genius of VSCode remotes. They built a core capability of running the bulk of the IDE "somewhere else" with just the display layer on your local machine. Then they can allow the developer to choose if container(s), VM(s), and local or remote are right for their case, without having to learn new tooling. They can even let you choose if "owning a computer" is required, or if a web browser is sufficient.
I keep trying to use Jetbrains Gateway. It spins up the IDE on a remote machine and renders it locally.
I honestly can't find a use for it. My personal machines are ancient quad core setups, but I have a beefy server at my disposal.
But the benefit of faster build times isn't worth the hassle. Between the suboptimal performance of the remote interface, syncing files back to my local machine, and all of the little idiosyncrasies of setting up the remote environment, it's still faster to develop locally. I just go grab a glass of water when I start my compile.
This might make more sense if you're working on some C++ monstrosity with millions of lines of code that takes hours to build on a mainframe, but if you're in the region of minutes per build, there's just no benefit.
Or I guess if all you have is a thin client, this might be better than nothing. But then again, you can get an IDE and compiler for most languages on an android tablet. If you have a machine with a processor, it can compile your code.
I prefer having a powerful development machine and install and use development environments natively. The only times I am prepared to use a (local or remote) VM is when the development environment cannot be installed on the OS of my development machine, or when I absolutely do not want to install those tools natively (because they might impact my main OS).
I am often involved with multiple projects at the same time, but if those projects are set up well (and the toolset is decent), that does not cause any issues. In practice that means that developers always have to take into account that the project itself includes references to the specific versions of the tools that the project needs. And that those tools are -as-much-as-possible- installed in the context of those projects instead of in the context of the whole machine.
A concrete example in node world is to use "nvm" and to always install tools locally in the context of a project instead of globally. In dotnet world, use global.json to specify the version of the tools you want to work with, install dotnet tools locally in the context of a project instead of globally.
I have worked on projects where this is a problem. For example, projects that depend on hardcoded paths. Or projects that depend on hardcoded connection strings for a database. Or projects that do not use something like nvm to specify the version of node/npm they are compatible with.
Each of those things is an issue with the setup of the project and is something that you will bump against when you try to setup a CI/CD pipeline for the project. Although, I agree that it can be useful to use docker containers with a correct version of the tools to create builds in a build pipeline.
The reason that I prefer to develop natively is that it is faster, snappier and more enjoyable. I also believe that it is easier to diagnose issues and investigate performance. Aside from that, renting a virtual machine with the same performance as my development machine, would be a lot more expensive, and I would still have to deal with the lag. I am aware of recent changes that allow you to for example run a JetBrains IDE locally (similar to what VSCode already had for a long time?) that is connected with a remote headless version of the IDE. I do not have any experience with that kind of setup yet, but I assume that would remove most of the lag.
Consider exploring LXC for a more mature alternative to Docker. It's not confined to the OCI ecosystem and offers a higher degree of isolation for development environments.
I don't know if I'd use the word "mature" (Docker is quite mature after all), but as a long-time Docker user I did jump into lxc/lxd and can confidently say it's "system container" approach is better for dev environments.
I used to use WSL 2 or a VM, but on a Linux host LXD is a really nice workflow. I can create a fully isolated Linux instance with all my dev tooling on it (optionally script/automate the install of all my tools), and with nested containers enabled I can even run _Docker in that LXD container_ so that when I type `docker ps` on that instance, and `docker ps` on my host, they each have their own set of containers running. For example, I have an ElasticSearch and Redis instance running in the dev box but Syncthing on the host.
Then I can use snapshots to back up the dev environment, or blow it away completely once it gathers too much cruft all without affecting my host. It's really great.
Pair it with VS Code remote SSH and you have a very feature rich setup with little effort.
I know this is an article specific to hocus so I don't mean to take away from that but the title is much more open ended and plural so I'd like to ask this -
What has been your favorite local dev environment?
I personally docker-compose everything w/ .env overrides and I don't really feel like I've ever needed anything more than this.
Docker Compose with .env overrides is very close to an ideal local dev environment, the only downside is if you don't use Linux you can hit some I/O bottlenecks (it got better with VirtioFS in Docker Desktop but still not native-speed)..
I still wait for properly usable remote work environment that trumps local in performance and convenience.... I feel like the VS Code approach with server-client IDE architecture and the client running as a web application is going in the right direction but not quite there yet.., Especially if you can leverage lots of RAM and GPU power in your workflow and often work from different locations, don't want to carry your laptop around but still want to use the same power, environment & configuration on any machine you work from....
How do you handle tooling? Install it in the parent or in the container? I quite like the idea of VS Code dev containers since it does a lot of the bootstrapping work to make the container be more like a full system, and I wish this “remote environment” idea was more widely supported amongst editors.
> Many cloud providers, like AWS inside VM-based EC2 instances, won't let you run a VM since they don't support nested virtualization
Does anyone know if there's a technical limitation to nested virt on EC2 or if it's just so people use .metal? I know Azure recently (1-2 years?) started offering it.
This blog doesn't convince me why I should care about Hocus over solutions like VSCode devcontainer + Vagrant. That is more what it is competing with, not raw Docker/VM.
Oh yes, in this post I was not trying to. Hocus gives you a web interface that lets you spin up a dev env with a single click of a button. We also implemented a git-integrated CI system that prebuilds your dev env on new commits. It’s basically a self-hosted Gitpod or GitHub Codespaces.
yea, great for small projects but no good when you're trying to expand into enterprise capabilities -- I have to get one tool for dev, have that config diverge from CI, and then from staging
then i have to hire a large devops team to manage it all -- super inefficient
Good god, no. Or, at least, let us never make this a sort of standard way of developing that you're going to push on to your engineers.
Containers have their place, they can be great, but I've tried the whole "containerize everything" approach, and it's no less time consuming than not containers, and now you have another Thing You Have To Learn (TM).
This isn't to say that virtualization is a bad thing for a development environment. It can be a very good thing, actually, but you don't need containers. A development environment, in my opinion, should be well organized, easy for a new developer to pick up, but also scrappy and adaptable to new and interesting situations. Containers, at least in their current conception, are antithetical to this. They're best kept to deployments or running things in a scenario where you know for sure you're not going to tinker with anything. I don't think I've worked on a software project where, at some point or another, I didn't need to reach under the hood in some unconventional way to investigate an issue or simulate some circumstance. Yes, you can do those things with containers, but now you have to deal with that complexity, which can get in the way by merely being confusing and indirect.
Having given up on using containers for development, when I want to use virtualization, I just spin up minimalist instances of Debian under Qemu (using the UTM GUI). This way, I can run a Thing (TM) in isolation, easily open up a shell, or SSH into it from the host shell, or start a GUI from within it if I need to, or install however many services I want within it and open up ports to the host, or have shared folders, etc. etc. etc.
But what if I need multiple of a Thing (TM) running at the same time? In that case, I just clone the VM. No Dockerfiles, no docker-compose.yml, no "volumes", no orchestration, no images failing to build because reasons, no qcow2 files blowing up with Docker For Mac (though that situation has improved), no pruning of orphaned containers/images, no worrying about whether your container should only be running a single process/service, and so on.
The only drawback to this is that, if the host is Linux, then it's indeed kinda dumb to be running other Linux kernels in VMs. But hey, it's 2023 and the difference in performance may not be meaningful depending on what you're doing. I know that, on an M1 Mac, I can run graphical applications at monitor resolution with incredible speed (perhaps moreso with Parallels instead of Qemu), so I'd struggle to care about any performance drawback if my host was Linux on the same hardware.
I'm sure that, for some, containers in development will be totally worthwhile. But if they don't seem worthwhile to you, dear reader, then that's probably because they wouldn't be.
As a consultant who jumps around between code bases with vastly different requirements, containers make for a much better experience than managing that on my main machine, or spinning up VMs for each separate project. It’s an ideal situation for me.
[+] [-] stephenr|2 years ago|reply
I don't really understand the point of a "dev box" that's hosted in a DC and shared, at all - at least not the way they're painting it here.
Hardware capabilities for even consumer level laptops and desktops have progressed much faster than average network connections.
Having testing/preview/branch named environments in a DC? Sure. But this line:
> They should be able to run any software they want inside their own workspaces without impacting each other.
What does that even mean?
Is this about someone working on a feature branch that uses some new dependency that needs to be installed?
That's 100% the sort of thing your local development environment is for, until you're ready to push it to your hosted test/whatever environment.
> But when it's running in a VM, the VM gobbles up as much memory as it can and, by itself, shows no inclination to give it back.
Someone, anyone, tell me which hypervisor doesn't enforce a memory limit on VMs?
[+] [-] scarmig|2 years ago|reply
[+] [-] Terretta|2 years ago|reply
https://code.visualstudio.com/docs/remote/ssh-tutorial
See also:
https://code.visualstudio.com/docs/devcontainers/containers
People pay for this.
https://github.com/features/codespaces
[+] [-] laurencerowe|2 years ago|reply
If everyone is working in docker in VS Code it’s not a huge jump to have them develop in remote VMs. Now your devs can get setup instantaneously and can use thin and light MacBook Airs instead of heavy MacBook Pros.
[+] [-] halfcat|2 years ago|reply
They don’t mean the memory usage is allowed to grow indefinitely.
They mean, if your laptop has 32 GB RAM, and if your OS and apps consume 20 GB, it’s using 20 GB RAM.
If a VM has 32 GB of vRAM allocated, and the OS is aggressively caching, that VM will consume 32 GB of RAM on the physical host, even if only 20 GB of vRAM is being used within that guest VM.
Of course, every good hypervisor will have a way to prevent this from happening, so the article does seem disingenuous on this point.
[+] [-] black3r|2 years ago|reply
[+] [-] ransackdev|2 years ago|reply
Not to mention the popular virtualization platforms like kvm, vmware, and hyper-v all support ballooning[1][2] where you can define a base amount of memory and a max, and when the guest garbage collects its heap within the vm the memory is reclaimed by the hypervisor.
This also allows overcommitting to better utilize the resources that would otherwise site mostly idle, and trust me every cloud provider overcommits their hypervisors, I used to work for a major one but anyone can check the cpu steal time in top and see how thrashed their host is
1. https://www.linux-kvm.org/page/Projects/auto-ballooning
2. https://pve.proxmox.com/wiki/Dynamic_Memory_Management#Ballo...
[+] [-] SkyMarshal|2 years ago|reply
https://nixos.org/explore
https://nix.dev/
[+] [-] hugodutka|2 years ago|reply
[+] [-] nickstinemates|2 years ago|reply
Memory and CPU cores are cheap. Getting the UX, resource allocation/sizing/instance type, and general pattern for multi-tenant dev boxes seems more fruitful.
Would love to see any data supporting this prioritization, and what the answer is for Workstation / HEDT-type environments given the industry's current focus on AI.
[+] [-] ohthehugemanate|2 years ago|reply
"Dev environment" depends on what kind of work you are doing. What a web developer needs is worlds apart from what a driver developer needs in terms of capability, resourcing, isolation, and rebuilds. Trying to offer a single environment for anyone who builds anything using a computer, is analogous to a single environment for anyone who uses a screwdriver. An electrician is using the same tool, sure, but they're doing fundamentally different work from a kitchen installer or a plumber and have vastly different needs. The single environment with all the capabilities that any computer developer needs is called a "computer."
IMO this is the genius of VSCode remotes. They built a core capability of running the bulk of the IDE "somewhere else" with just the display layer on your local machine. Then they can allow the developer to choose if container(s), VM(s), and local or remote are right for their case, without having to learn new tooling. They can even let you choose if "owning a computer" is required, or if a web browser is sufficient.
[+] [-] siliconc0w|2 years ago|reply
[+] [-] overnight5349|2 years ago|reply
I honestly can't find a use for it. My personal machines are ancient quad core setups, but I have a beefy server at my disposal.
But the benefit of faster build times isn't worth the hassle. Between the suboptimal performance of the remote interface, syncing files back to my local machine, and all of the little idiosyncrasies of setting up the remote environment, it's still faster to develop locally. I just go grab a glass of water when I start my compile.
This might make more sense if you're working on some C++ monstrosity with millions of lines of code that takes hours to build on a mainframe, but if you're in the region of minutes per build, there's just no benefit.
Or I guess if all you have is a thin client, this might be better than nothing. But then again, you can get an IDE and compiler for most languages on an android tablet. If you have a machine with a processor, it can compile your code.
[+] [-] rvdginste|2 years ago|reply
I am often involved with multiple projects at the same time, but if those projects are set up well (and the toolset is decent), that does not cause any issues. In practice that means that developers always have to take into account that the project itself includes references to the specific versions of the tools that the project needs. And that those tools are -as-much-as-possible- installed in the context of those projects instead of in the context of the whole machine.
A concrete example in node world is to use "nvm" and to always install tools locally in the context of a project instead of globally. In dotnet world, use global.json to specify the version of the tools you want to work with, install dotnet tools locally in the context of a project instead of globally.
I have worked on projects where this is a problem. For example, projects that depend on hardcoded paths. Or projects that depend on hardcoded connection strings for a database. Or projects that do not use something like nvm to specify the version of node/npm they are compatible with.
Each of those things is an issue with the setup of the project and is something that you will bump against when you try to setup a CI/CD pipeline for the project. Although, I agree that it can be useful to use docker containers with a correct version of the tools to create builds in a build pipeline.
The reason that I prefer to develop natively is that it is faster, snappier and more enjoyable. I also believe that it is easier to diagnose issues and investigate performance. Aside from that, renting a virtual machine with the same performance as my development machine, would be a lot more expensive, and I would still have to deal with the lag. I am aware of recent changes that allow you to for example run a JetBrains IDE locally (similar to what VSCode already had for a long time?) that is connected with a remote headless version of the IDE. I do not have any experience with that kind of setup yet, but I assume that would remove most of the lag.
[+] [-] foxbyte|2 years ago|reply
[+] [-] seabrookmx|2 years ago|reply
I used to use WSL 2 or a VM, but on a Linux host LXD is a really nice workflow. I can create a fully isolated Linux instance with all my dev tooling on it (optionally script/automate the install of all my tools), and with nested containers enabled I can even run _Docker in that LXD container_ so that when I type `docker ps` on that instance, and `docker ps` on my host, they each have their own set of containers running. For example, I have an ElasticSearch and Redis instance running in the dev box but Syncthing on the host.
Then I can use snapshots to back up the dev environment, or blow it away completely once it gathers too much cruft all without affecting my host. It's really great.
Pair it with VS Code remote SSH and you have a very feature rich setup with little effort.
[+] [-] m463|2 years ago|reply
I think a Dockerfile as a recipe for an environment is pretty elegant.
I've used LXC with proxmox and managing what's in a container is kind of like being a sysadmin.
[+] [-] swozey|2 years ago|reply
What has been your favorite local dev environment?
I personally docker-compose everything w/ .env overrides and I don't really feel like I've ever needed anything more than this.
[+] [-] black3r|2 years ago|reply
I still wait for properly usable remote work environment that trumps local in performance and convenience.... I feel like the VS Code approach with server-client IDE architecture and the client running as a web application is going in the right direction but not quite there yet.., Especially if you can leverage lots of RAM and GPU power in your workflow and often work from different locations, don't want to carry your laptop around but still want to use the same power, environment & configuration on any machine you work from....
[+] [-] CGamesPlay|2 years ago|reply
[+] [-] easton|2 years ago|reply
Does anyone know if there's a technical limitation to nested virt on EC2 or if it's just so people use .metal? I know Azure recently (1-2 years?) started offering it.
[+] [-] colinchartier|2 years ago|reply
[+] [-] segfaltnh|2 years ago|reply
[+] [-] pphysch|2 years ago|reply
[+] [-] hugodutka|2 years ago|reply
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] therealmarv|2 years ago|reply
[+] [-] debosmit|2 years ago|reply
[+] [-] emptysongglass|2 years ago|reply
[+] [-] ravenstine|2 years ago|reply
Containers have their place, they can be great, but I've tried the whole "containerize everything" approach, and it's no less time consuming than not containers, and now you have another Thing You Have To Learn (TM).
This isn't to say that virtualization is a bad thing for a development environment. It can be a very good thing, actually, but you don't need containers. A development environment, in my opinion, should be well organized, easy for a new developer to pick up, but also scrappy and adaptable to new and interesting situations. Containers, at least in their current conception, are antithetical to this. They're best kept to deployments or running things in a scenario where you know for sure you're not going to tinker with anything. I don't think I've worked on a software project where, at some point or another, I didn't need to reach under the hood in some unconventional way to investigate an issue or simulate some circumstance. Yes, you can do those things with containers, but now you have to deal with that complexity, which can get in the way by merely being confusing and indirect.
Having given up on using containers for development, when I want to use virtualization, I just spin up minimalist instances of Debian under Qemu (using the UTM GUI). This way, I can run a Thing (TM) in isolation, easily open up a shell, or SSH into it from the host shell, or start a GUI from within it if I need to, or install however many services I want within it and open up ports to the host, or have shared folders, etc. etc. etc.
But what if I need multiple of a Thing (TM) running at the same time? In that case, I just clone the VM. No Dockerfiles, no docker-compose.yml, no "volumes", no orchestration, no images failing to build because reasons, no qcow2 files blowing up with Docker For Mac (though that situation has improved), no pruning of orphaned containers/images, no worrying about whether your container should only be running a single process/service, and so on.
The only drawback to this is that, if the host is Linux, then it's indeed kinda dumb to be running other Linux kernels in VMs. But hey, it's 2023 and the difference in performance may not be meaningful depending on what you're doing. I know that, on an M1 Mac, I can run graphical applications at monitor resolution with incredible speed (perhaps moreso with Parallels instead of Qemu), so I'd struggle to care about any performance drawback if my host was Linux on the same hardware.
I'm sure that, for some, containers in development will be totally worthwhile. But if they don't seem worthwhile to you, dear reader, then that's probably because they wouldn't be.
[+] [-] CGamesPlay|2 years ago|reply