We use Mutagen for Garden's hot reloading mechanism. (Garden is a dev tool for K8s and hot reloading enables users to sync changes directly to a prod like dev environment as opposed to doing a rebuild and re-deploy).
It really is a fantastic piece of technology and completely transformed the whole experience (we were using good 'ol rsync before). In particular it works seamlessly across platforms.
This is a MASSIVE endorsement as far as I am concerned
I used Garden in the rsync days and it was already pretty good. The code behind Garden is some of the best TypeScript + Node I've ever seen, so if this was better and more performant than what you already had I'm sold.
I’ve been using Mutagen with Docker for about 3-4 months to work around an (at the time) performance problem with bound volumes. I understand the new experimental FS option in Docker might solve this, but I haven’t tested or migrated my setup yet.
Basically, I’m working on a bare metal kernel in C/C++ on ARM, which only has tooling on Windows or Linux. As I work on a Mac system, I setup a Docker container that makes this trivial for me, including GDB debugging of QEMU on the host machine.
Mutagen was easy to get going, well documented, and has worked perfectly for this. I’m able to make my edits in my Mac IDE, then literally by the time I’ve switched to compilation, the edits are synced to the container, and vice-versa for the build product.
Prior to Mutagen, I was getting 8-10x longer build times since Docker file synchronization was slowing things down. Mutagen syncs slower than Docker, but faster than I can work, so all is well.
Author of Mutagen here, happy to answer any questions you might have, whether it's about its use with Docker, SSH, its historical integration into Docker Desktop, or its related Mutagen Compose[0][1] project.
Hi Jacob. I am one of the founders of Okteto (https://okteto.com/), a remote development platform for Compose and Kubernetes applications. We use Syncthing to sync code between the developer laptop and pods running in Kubernetes. I would love to know your thoughts on the strengths and weak points of Mutagen vs Syncthing for this use case.
Thanks!
I'd love to hear more about anyone who has managed to use mutagen as a docker-compose replacement. I have so far been a bit let down by mutagen in that it doesn't really seem to be living up the promises of really being easier than using docker directly. Its file synchronization concepts also don't always seem to keep up with file watchers and hot reload
I've been looking at something like Lando[0] as a result, and I realize they're not the same thing, it seems like it is more fitting with the role for the development environment, at least.
Hey, sorry to hear that Mutagen didn't pan out for you. I'd be curious to hear if you've tried the Mutagen Compose[0] project or some other mechanism for Compose integration. I'm still iterating on the best approach for integrating Mutagen with various Docker workflows, so I'm always looking for feedback and experience reports.
OT but, I switched away from Lando over to plain docker-compose files, It was much easier to configure things. For example, trying to get HMR working in Lando was just a pain.
Lando is great for getting up and running quickly,outside of the recipes, it's easier to just create your own docker-compose file.
I've been testing Mutagen's file sync through DDEV on a Drupal project and it's pretty nice!
The project is way more snappy than using NFS mounts on the Mac (which was already WAY faster than bind mounts), and DDEV has file ignores set up correctly so the project's media files aren't synced and disk usage isn't doubled.
Yeah, Randy's done a great job with the integration - it seems to work seamlessly for most users as far as I can tell. I'm really excited to get Mutagen v0.14 and beyond into DDEV, because support for fanotify[0] is being added and it's going to make Mutagen's Linux filesystem watching unbelievably faster for container-based development.
Looks like a really handy tool and essentially the logical endpoint of all the various hacky manual file watcher / scripted copy type sync things we end up doing to work across environments if they were developed into a full solution.
We work quite hard to maintain unified production/dev config for our docker-compose setups. Mutagen looks like it could simplify some of these since a whole layer of complexity comes from trying to optionally map ports and volumes in such a way that it works for development, while not being inappropriate in test/production. If instead, mutagen can virtually connect / synchronize these it'll be a lot simpler.
I do wonder a bit how debugging works .... if something is running in a container, it's great I can hot sync my code there, but what happens when I put a breakpoint in the code in my IDE?
Mutagen and mutagen-compose + an EC2 instance work great as a replacement for Docker for Mac. This setup you to develop and run Intel images without performance penalty. Both on M1 and on Intel I’m never going back to Docker for Mac again if I help it.
An important thing to note is that the _worst_ part about Docker for Mac is the lousy disk IO performance for shared drives. This can be almost fully mitigated by using NFS mounts instead[0].
In general Docker for Mac is far from ideal - particularly considering you must allocate a set block of memory to the VM. If only it was the year of the Linux Desktop :grin:
Looks like it handles two-way sync very well, but could it keep more than two participants in sync? I'm sort of "abusing" Syncthing to sync Docker volumes (mostly just configuration and helper scripts) across several hosts. It works well, and the only things I'm missing are file ownership preservation and low-latency "instant"-feeling sync. csync2 gets it right, but it's fragile and needs a lot of babysitting. I'm wondering if Mutagen might be a suitable alternative.
One way you can do this with Mutagen is to use a hub-and-spoke topology, where one copy of the files (say the one on the laptop where you're coding) is the "hub," and then you have multiple two-way (or one-way) synchronization sessions going out to the various "spoke" endpoints. In the two-way case, changes on a spoke endpoint will be propagated out to the other endpoints by first propagating to the hub. Obviously, you can have contention in this case if you have ultra-high-frequency updates to the same sets of of files, though the conflicting operations would have to occur on a smaller timescale than that of a single Mutagen synchronization cycle (which is typically somewhere around 100ms for an average codebase). Also, in the case of contention, Mutagen will simply display the conflicting files to you and you can delete the "losing" copy. You can also set auto-resolution behavior with Mutagen's `two-way-resolved` mode to let the hub win in conflicts, or use `one-way-replica` mode to make all the spoke endpoints replicas of the hub, or some combination of all these things, etc.
Mutagen is great (I really wish Skaffold implemented it for it's sync) and I want to point out that Jacob, the author is the project, is a really awesome and intelligent person - few people in our YC batch deserve the front-page more!
I have been using mutagen for few years. It has been a life saver.
I use it mostly to work on remote projects, mostly for development, rarely working on prod. So there I have my projects on some cloud linux instances and I work locally on windows, mostly phpstorm.
Having the webserver in the cloud and doing the heavy lifting is quite nice, especially if you work with phpstorm, which can be a resource monster for bigger projects, and working on remote projects with phpstorm built in sftp / file sync is a huge pain in the ass, really slow.
With mutagen it feels almost like I work locally. Thanks to the developer for this cool tool!
I've been using Mutagen for years to offload running docker containers during local development from my Mac to a Linux machine and it works great. I'm glad to see more people talking about it.
For integration tests and other code meant to run like external facing code I use my local machine for debugging. If my local machine is too slow (like running the whole integ test suite) I use a cloud VM.
For internal code or slow to build code I use a remote VM hooked up to the internal network. That allows me to build quickly and test in an integration environment.
I use a file synchronization tool like mutagen at work (for ninjas) and I use mutagen on my personal laptop for the same workflow.
I generally prefer using a remote vm for development and only using my laptop for git.
Also you can use spot VMs for development so it can be cheap to get a high CPU vm for building and debugging or a high mem VM for running data science scripts and debugging them.
This looks interesting, but I'm not sure I correctly understand the use case or how exactly this might fit into my development workflow. Can someone enlighten me and elaborate on what this solves and what its use cases are? Thanks.
There's no compiler here - it's a tool for bidirectionally synchronizing files (specifically code, assets, build products, etc.) and forwarding network traffic to and from remote systems.
The idea behind Mutagen is to use your local tools (e.g. editor, IDE, browser, etc.) to work on remote hardware (where "remote" can mean anything you like, e.g. your local system, a Docker container (local or remote), a Raspberry Pi, a cloud server, etc. - or some combination of all of those).
The architecture is designed to be maximally flexible, so you can work between your local system and a remote system, between two remote systems, or in any other topology you want.
In the kind of setup you'd use with mutagen your dev tools and compilers are typically inside the container, while your code and editor are outside the container and being synced inside the container on every change. If the container is running on another machine/cloud service then yeah it could be that your compiler is running outside your machine. But much more likely it's still running as a container on your machine. The typical use case for this is making your dev environment accessible to anyone, just send them a dockerfile and they can almost perfectly reproduce it and get going with building your software (vs. having to laboriously setup the exact version of each dev tool and compiler you use).
I’m sure this feels like an accomplishment but in practice it provides nothing over git/source control as usual, and any number of reliable open or free hosted file sharing options.
What specifically is missing in the current ecosystem of electron state moving technologies that this solves? Why this instead of git and a Wireguard based mesh of devices (to offer my current setup as an example; there are others).
The goal with Mutagen is real-time remote development, not source control or complex networking. It's designed to facilitate making changes to code locally and then having those immediately propagated to a remote system for compilation, execution, etc. Basically, it's designed for the work you do between Git commits, saving you the need to do a full commit, push, pull, evaluate cycle to test your code on a remote system.
Basically, it's like VS Code's remote development extensions, but editor-agnostic and more flexible.
[+] [-] eysi|4 years ago|reply
It really is a fantastic piece of technology and completely transformed the whole experience (we were using good 'ol rsync before). In particular it works seamlessly across platforms.
If anyone's interested in how we use it, it's here: https://github.com/garden-io/garden/blob/master/core/src/plu...
[+] [-] gavinray|4 years ago|reply
I used Garden in the rsync days and it was already pretty good. The code behind Garden is some of the best TypeScript + Node I've ever seen, so if this was better and more performant than what you already had I'm sold.
Tell Jon I said hi =)
[+] [-] incanus77|4 years ago|reply
Basically, I’m working on a bare metal kernel in C/C++ on ARM, which only has tooling on Windows or Linux. As I work on a Mac system, I setup a Docker container that makes this trivial for me, including GDB debugging of QEMU on the host machine.
Mutagen was easy to get going, well documented, and has worked perfectly for this. I’m able to make my edits in my Mac IDE, then literally by the time I’ve switched to compilation, the edits are synced to the container, and vice-versa for the build product.
Prior to Mutagen, I was getting 8-10x longer build times since Docker file synchronization was slowing things down. Mutagen syncs slower than Docker, but faster than I can work, so all is well.
[+] [-] xenoscopic|4 years ago|reply
[0]: https://github.com/mutagen-io/mutagen-compose [1]: https://mutagen.io/documentation/orchestration/compose
[+] [-] lifty|4 years ago|reply
[+] [-] pchico83|4 years ago|reply
[+] [-] vorpalhex|4 years ago|reply
What are the plans for funding the project? It looks like everything is OSS now, will there be any closed source elements in the future?
[+] [-] slightwinder|4 years ago|reply
[+] [-] no_wizard|4 years ago|reply
I've been looking at something like Lando[0] as a result, and I realize they're not the same thing, it seems like it is more fitting with the role for the development environment, at least.
[0]: https://docs.lando.dev/getting-started/
[+] [-] xenoscopic|4 years ago|reply
[0]: https://mutagen.io/documentation/orchestration/compose
[+] [-] bennyp101|4 years ago|reply
Lando is great for getting up and running quickly,outside of the recipes, it's easier to just create your own docker-compose file.
[+] [-] karlshea|4 years ago|reply
The project is way more snappy than using NFS mounts on the Mac (which was already WAY faster than bind mounts), and DDEV has file ignores set up correctly so the project's media files aren't synced and disk usage isn't doubled.
[+] [-] xenoscopic|4 years ago|reply
[0]: https://man7.org/linux/man-pages/man7/fanotify.7.html
[+] [-] pineconewarrior|4 years ago|reply
[+] [-] dutchbrit|4 years ago|reply
[+] [-] zmmmmm|4 years ago|reply
We work quite hard to maintain unified production/dev config for our docker-compose setups. Mutagen looks like it could simplify some of these since a whole layer of complexity comes from trying to optionally map ports and volumes in such a way that it works for development, while not being inappropriate in test/production. If instead, mutagen can virtually connect / synchronize these it'll be a lot simpler.
I do wonder a bit how debugging works .... if something is running in a container, it's great I can hot sync my code there, but what happens when I put a breakpoint in the code in my IDE?
[+] [-] osener|4 years ago|reply
[+] [-] jeffnappi|4 years ago|reply
In general Docker for Mac is far from ideal - particularly considering you must allocate a set block of memory to the VM. If only it was the year of the Linux Desktop :grin:
[0] https://gist.github.com/vschoener/b0b5ae08625744a1f4ad414027...
[+] [-] inportb|4 years ago|reply
[+] [-] xenoscopic|4 years ago|reply
[+] [-] stephen|4 years ago|reply
https://github.com/stephenh/mirror
But dressed up to work with docker + "just" :-) being a professional/polished product, which is impressive. Neat!
[+] [-] erulabs|4 years ago|reply
[+] [-] qxxx|4 years ago|reply
Having the webserver in the cloud and doing the heavy lifting is quite nice, especially if you work with phpstorm, which can be a resource monster for bigger projects, and working on remote projects with phpstorm built in sftp / file sync is a huge pain in the ass, really slow.
With mutagen it feels almost like I work locally. Thanks to the developer for this cool tool!
[+] [-] InsaneOstrich|4 years ago|reply
[+] [-] kyriakos|4 years ago|reply
[+] [-] bliteben|4 years ago|reply
[+] [-] dboreham|4 years ago|reply
[+] [-] mutagen|4 years ago|reply
(not associated)
[+] [-] galaxyLogic|4 years ago|reply
[+] [-] faizshah|4 years ago|reply
For internal code or slow to build code I use a remote VM hooked up to the internal network. That allows me to build quickly and test in an integration environment.
I use a file synchronization tool like mutagen at work (for ninjas) and I use mutagen on my personal laptop for the same workflow.
I’m thinking of trying remote development from jetbrains soon though: https://www.jetbrains.com/remote-development/
I generally prefer using a remote vm for development and only using my laptop for git.
Also you can use spot VMs for development so it can be cheap to get a high CPU vm for building and debugging or a high mem VM for running data science scripts and debugging them.
[+] [-] xwowsersx|4 years ago|reply
[+] [-] xjia|4 years ago|reply
[+] [-] xenoscopic|4 years ago|reply
The idea behind Mutagen is to use your local tools (e.g. editor, IDE, browser, etc.) to work on remote hardware (where "remote" can mean anything you like, e.g. your local system, a Docker container (local or remote), a Raspberry Pi, a cloud server, etc. - or some combination of all of those).
The architecture is designed to be maximally flexible, so you can work between your local system and a remote system, between two remote systems, or in any other topology you want.
[+] [-] qbasic_forever|4 years ago|reply
[+] [-] kamikazeturtles|4 years ago|reply
[+] [-] gr8tnwz|4 years ago|reply
What specifically is missing in the current ecosystem of electron state moving technologies that this solves? Why this instead of git and a Wireguard based mesh of devices (to offer my current setup as an example; there are others).
[+] [-] xenoscopic|4 years ago|reply
Basically, it's like VS Code's remote development extensions, but editor-agnostic and more flexible.
[+] [-] pineconewarrior|4 years ago|reply