I'm more interested in this as part of a CI/CD process for releasing macOS software. For an isolated build worker - yeah, sure, I'll go ahead and disable SIP.
Not every workload is running on an endpoint connected to a human via keyboard and screen.
Fundamentally, containers are about namespace/isolation of a bunch of OS interfaces, so file system functions, network functions, memory management, process functions, etc, can all pretend like they're the only game in town, but crucially without having to virtualize out the kernel.
Does XNU have such namespacing functionality across all its interfaces?
Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.
The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out: they actually run a Linux virtual machine (using Hypervisor.framework/hvf), and have that just provide the container environment.
Is that what this project is doing? But then, how could it run a macOS container?
[1] based on the foundation that Linux, unlike BSDs, has a stable syscall interface!
This introduces a long-standing problem that is really the main issue I have with Docker/Podman/containers on anything other than Linux - you have a dedicated VM that needs memory to be set aside for it exclusively.
This is fine if you have a 32/64GB machine, but less so on an 8GB non-upgradeable laptop.
I get it - memory is relatively cheap these days - and manufacturers that are building memory-limited devices are really only doing it to fleece you on obscene upgrade fees at the time of purchase - but it would be nice if there was a more elegant solution to this on Windows and macOS.
WSL 1 had a solution to this that clearly took a lot of work to put together, wherein they'd have a Linux kernel running side-by-side as a Windows sub-process so that the memory pool was shared. Unfortunately it might have been too much work as they scrapped it entirely for WSL 2 and just used essentially the same VM route.
If anyone knows of any projects trying to work around that problem I'd love to hear about it. If Apple really wanted to bring the development community back on board, focusing on these kind of use cases would be great, sadly it seems someone over there has taken the view that scrapping butterfly keys and the touchbar is "enough".
Say what you will about Microsoft, but they've focused really hard on developer use cases for decades, and it shows.
rund is an experimental containerd shim for running macOS containers on macOS.
rund doesn’t offer the usual level of container isolation that is achievable on other OSes due to limited macOS kernel API.
What rund provides:
- Filesystem isolation via chroot(2)
- Cleanup of container processes using process group
- OCI Runtime Specification compatibility (to the extent it is possible on macOS)
- Host-network mode only
- bind mounts
DfM is more like running the CLI locally to a remote Linux machine, and all it does is conveniently expose /Users in the same place through the VM folder share so that you have the convenient illusion that it happens locally.
If Darwin had process namespacing features it would not make it magically able to run Linux processes.
> Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.
There is more to the container ecosystem than Linux containers; Windows native containers function much the same way (well, in two ways, with VM-backing or the traditional kernel syscall interface, but with Windows syscalls).
I am also interested in the API boundary. It seems that there are too options:
1. Rely on system call stability. This is like Linux containers but unlike Linux macOS doesn't provide a stable system call API. So this would break when the system updates with a change that modifies the system call API.
2. Install the host libraries into the container at runtime. This should provide as much stability as macOS apps usually have. It may also be beneficial as you wouldn't be embedding these into every container.
It seems like 2 would be preferable. However it may be a bit weird when building as the libraries you build against would be updated without the container being aware, but this is unlikely to break anything unless they are copying them to new paths which seems unlikely.
I can't help but feel like this is an X/Y problem. Apps on MacOS shouldn't need containerization to function.
I get the point of isolation for build/test situations. But Apple provides a neat virtualization framework, and you get security + isolation + reproducibility + decent performance.
It seems like if you feel the need to containerize the userspace on MacOS you're using MacOS wrong. It's not the same thing as the Linux userspace, and doesn't have the same kernel features that would let you do so cleanly or performantly.
Orbstack is moving mountains to provide Linux-native perf and support for containers and it still makes me beg the question: why are devs allergic to just using Linux natively? At least I understand why Orbstack is useful, I don't know why containerizing MacOS itself is.
> But Apple provides a neat virtualization framework, and you get security + isolation + reproducibility + decent performance.
You also get limits on how many VMs your machine can run, each VM needs gobs of storage and locked-out RAM blocks, and sharing directories between the host and guest, compared to bind mounts, is something that makes me remember for my root canal dental jobs wistfully.
What's the licensing situation on this? Would I be distributing parts of macOS in my containers? I don't think Apple is OK with that.
Or is this just the fully open source Darwin core? That wouldn't likely be super compatible with a ton of production software? I need more explanation of what is actually going on here because it sounds like a good way to get sued.
> Would I be distributing parts of macOS in my containers?
Unless you're producing fully static binaries (or static enough that they don't bind to non-redistributable things) it'd be a yes (it would not be much of a container if it needed non-packaged things)
There are Cirrus Runners which is a service of managed GitHub Actions Runners powered by M2 chips. But there is no free option for OSS yet. https://tart.run/integrations/github-actions
This one is ridiculous. This should already exist. Until GitHub builds it, you can use GitHub Actions to kick your builds off but run them remotely on Earthly Cloud (https://earthly.dev/). Even the free tier includes arm64 remote runners.
Note: I work at Earthly, but I'm not wrong about this being a good, free, arm64-native workflow for GitHub Actions.
It's intended to prevent malware from changing system files due to rogue permissions or escalation. With SIP enabled, even the root/sudo user doesn't have rights to change these files.
It also refuses to boot a system with drivers that are not signed by Apple, so as to deter malware from using drivers as an attack vector.
It clearly links to the GitHub where you can click to see all contributors
I suppose the answer to your question is “people who want macOS containers”, whoever they are. As far as malware, I’d employ whatever your standard practices are for installing GitHub projects
I bought an Apple Silicon machine after their presentation claiming that they would have first class docker support, but the reality has been that while the first docker worked well as it was translated, now it wants to default to arm containers and it has become very difficult to use because it doesn't want to use Rosetta 2 containers.
The whole point of using docker is to use the same containers in production as you use in development, so having docker default to these random arm containers means that my containers aren't exactly production, because they are arm based and the servers are not.
I understand that docker is the developer of docker software, but I really wish I could just click a button and force intel based containers in docker as the default and have to opt-in to arm.
If anyone has an easy solution to this let me know. I don't want to spend hours and hours figuring out docker on my mac.
At the same time, I don't truly understand why anyone would need to use it. If your preference is to totally work with macOS, then I'm sure this would be perfect for that. Otherwise, what's the advantage?
VMs have really come a long way. Every major OS today has a virtualization framework that makes running another OS extremely performant. Docker on macOS uses a virtual machine, but so what? Performance of individual containers, in my experience, isn't really a problem unless you're doing something with the GPU, and even then there are ways to deal with that. Even a fully-emulated VM using QEMU (without hypervisor or KVM) won't have any noticeable performance penalties in many cases.
IMO, there's a much greater advantage to sticking with Linux. Even if the host isn't Linux, developing and deploying with Linux guests provides a tremendous level of consistency and portability.
But maybe I'll be proven wrong by this project someday soon!
What my dream is that the User Mode Linux is made into a cross-platform userspace binary that translates syscalls transparently between itself and the host. So you might get "drivers" that talk to Windows, Linux, *BSDs, Darwin, it manages memory in an efficient (for the host) way, and enables you to run any kinds of wild experiments with, say, virtualized and passed-through serial devices, USB devices, networking, bind-mounting from the host and image mounts. And yes, containers. All of that without needing host root in most cases.
Of course the drawback would be that the host would see just a fat Linux process and its child processes, much like you can see qemu, but it could be an interesting thing nonetheless, if even for shits and giggles of it.
When macOS runs on Unix kernel and Linux systems are the best supported for containerisation and I assume are much more lightweight than macOS, I personally don't see any reason to run macOS in a container.
Can anyone speak to how the macOS runners on GitHub actions work? It would seem from this post that containers of any kind for macOS are a brand new thing..
the amount of engineering hours wasted making macos usable for backend dev work and then wasted again from inefficiency due to that failure is staggering.
linux is great. macos is great. windows is great too. for their intended purposes.
caveat: this is based on rund. Extract from the readme:
rund is an experimental containerd shim for running macOS containers on macOS.
rund doesn’t offer the usual level of container isolation that is achievable on other OSes due to limited macOS kernel API.
What rund provides:
Filesystem isolation via chroot(2)
Cleanup of container processes using process group
OCI Runtime Specification compatibility (to the extent it is possible on macOS)
Host-network mode only
bind mounts
I use MacOS and am very positive about it. I have lots of reasons to run Linux containers. What are some reasons I might want to run a MacOS container?
I would imagine it would enable you to run MacOS specific pipelines like building a project and have it work semi-portably across different machines/users.
Unrelated to containers themselves: how do you make a patch when no version was released? I mean, people call this "semantic" versioning, but then spit in the face of those semantics...
Original author here. I wanted to clearly indicate early-prealpha-unstable-not-for-production-yet state of this software. Using "1.0.0" and even "1.0.0-alpha" would give false expectations about maturity of this project.
0.1.0 might make more sense, but still not completely (backwards compatible with what?). Then again, it doesn't really matter. It's not like someone's going to accidentally install 0.0.1 because of semantic versioning not being 100% correct.
CI/CD workflows most likely. And devshops that have standadised on docker containers for their stacks (mac-based devs in such places suffer a fair amount of papercuts today). Then I guess there are people that are very security minded that might want to run all userland executables in containers (although this project here is not for them I'd say).
It's sad to see so many negative comments for this. I get it's not an ideal place to start for macOS containers, but it's a start. Apple isn't doing it, so the community has to. Once you have a start, you can iterate on it. It might not be great now, but hopefully this makes it possible in a year or so. Who knows, maybe this is the kick Apple needs, and maybe they'll hire the devs of this project to fully work on this.
Sorry, not disabling SIP for something that I can already do without needing to nobble security policies (and have them reset/impossible due to MDM). If there was user/networking space in Darwin then maybe I'd be interested but...
It’s remarkable that Apple doesn’t have a first party solution to this yet. They used be, or aspire to be, at the forefront of OS research.“The most advanced Unix”.
Some comments were deferred for faster rendering.
highwaylights|2 years ago
"macOS native containers"
Cool, this sounds interesting.
"Disable System Identity Protection."
Eesh.
maggit|2 years ago
(More details here: https://support.apple.com/en-us/102149 )
solatic|2 years ago
Not every workload is running on an endpoint connected to a human via keyboard and screen.
comandillos|2 years ago
jamil7|2 years ago
rakoo|2 years ago
There's probably a hundred things that are not right just yet and they know it, let's not be overly negative
(although the discussion on what it is and what it does definitely is interesting)
conradev|2 years ago
perryizgr8|2 years ago
This is the first thing I do on any Mac OS system before I start using it.
smcleod|2 years ago
satvikpendem|2 years ago
nunez|2 years ago
sacnoradhq|2 years ago
Aaand, it's stillborn. Not happening.
nsonha|2 years ago
dddnzzz334|2 years ago
[deleted]
AceJohnny2|2 years ago
Fundamentally, containers are about namespace/isolation of a bunch of OS interfaces, so file system functions, network functions, memory management, process functions, etc, can all pretend like they're the only game in town, but crucially without having to virtualize out the kernel.
Does XNU have such namespacing functionality across all its interfaces?
Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.
The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out: they actually run a Linux virtual machine (using Hypervisor.framework/hvf), and have that just provide the container environment.
Is that what this project is doing? But then, how could it run a macOS container?
[1] based on the foundation that Linux, unlike BSDs, has a stable syscall interface!
highwaylights|2 years ago
This is fine if you have a 32/64GB machine, but less so on an 8GB non-upgradeable laptop.
I get it - memory is relatively cheap these days - and manufacturers that are building memory-limited devices are really only doing it to fleece you on obscene upgrade fees at the time of purchase - but it would be nice if there was a more elegant solution to this on Windows and macOS.
WSL 1 had a solution to this that clearly took a lot of work to put together, wherein they'd have a Linux kernel running side-by-side as a Windows sub-process so that the memory pool was shared. Unfortunately it might have been too much work as they scrapped it entirely for WSL 2 and just used essentially the same VM route.
If anyone knows of any projects trying to work around that problem I'd love to hear about it. If Apple really wanted to bring the development community back on board, focusing on these kind of use cases would be great, sadly it seems someone over there has taken the view that scrapping butterfly keys and the touchbar is "enough".
Say what you will about Microsoft, but they've focused really hard on developer use cases for decades, and it shows.
robertlagrant|2 years ago
lloeki|2 years ago
Essentially, like this:
https://ericchiang.github.io/post/containers-from-scratch/
https://earthly.dev/blog/chroot/
> The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out
It's not, it's a requirement for running Linux containers: https://news.ycombinator.com/item?id=37656401
DfM is more like running the CLI locally to a remote Linux machine, and all it does is conveniently expose /Users in the same place through the VM folder share so that you have the convenient illusion that it happens locally.
If Darwin had process namespacing features it would not make it magically able to run Linux processes.
plextoria|2 years ago
I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK
> Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.
This project is about running macOS containers on macOS. It's not about running Linux containers.
dwattttt|2 years ago
There is more to the container ecosystem than Linux containers; Windows native containers function much the same way (well, in two ways, with VM-backing or the traditional kernel syscall interface, but with Windows syscalls).
kevincox|2 years ago
1. Rely on system call stability. This is like Linux containers but unlike Linux macOS doesn't provide a stable system call API. So this would break when the system updates with a change that modifies the system call API.
2. Install the host libraries into the container at runtime. This should provide as much stability as macOS apps usually have. It may also be beneficial as you wouldn't be embedding these into every container.
It seems like 2 would be preferable. However it may be a bit weird when building as the libraries you build against would be updated without the container being aware, but this is unlikely to break anything unless they are copying them to new paths which seems unlikely.
unknown|2 years ago
[deleted]
lijok|2 years ago
duped|2 years ago
I get the point of isolation for build/test situations. But Apple provides a neat virtualization framework, and you get security + isolation + reproducibility + decent performance.
It seems like if you feel the need to containerize the userspace on MacOS you're using MacOS wrong. It's not the same thing as the Linux userspace, and doesn't have the same kernel features that would let you do so cleanly or performantly.
Orbstack is moving mountains to provide Linux-native perf and support for containers and it still makes me beg the question: why are devs allergic to just using Linux natively? At least I understand why Orbstack is useful, I don't know why containerizing MacOS itself is.
WesolyKubeczek|2 years ago
You also get limits on how many VMs your machine can run, each VM needs gobs of storage and locked-out RAM blocks, and sharing directories between the host and guest, compared to bind mounts, is something that makes me remember for my root canal dental jobs wistfully.
donatj|2 years ago
Or is this just the fully open source Darwin core? That wouldn't likely be super compatible with a ton of production software? I need more explanation of what is actually going on here because it sounds like a good way to get sued.
slonopotamus|2 years ago
1. This project didn't take explicit permission from Apple to redistribute binaries
2. There are multiple jurisdictions where you don't need to explicitly have such permission, it is implied by law
3. Usage of this software implies you already have macOS system. I'm not a lawyer, but it looks to be covered by section 3 of macOS EULA.
4. There are existing precedents of redistribution of macOS binaries for multiple years aready:
- https://github.com/cirruslabs/macos-image-templates/pkgs/con...
- https://hub.docker.com/r/sickcodes/docker-osx
- https://app.vagrantup.com/jhcook/boxes/macos-sierra
And so on.
lloeki|2 years ago
Unless you're producing fully static binaries (or static enough that they don't bind to non-redistributable things) it'd be a yes (it would not be much of a container if it needed non-packaged things)
The screenshot points out a ghcr.io URL that lands on these packages: https://github.com/orgs/macOScontainers/packages?repo_name=m...
Edit: There's a note here†, so at least there is some consideration for licensing. No idea if it holds ground.
† https://github.com/macOScontainers/macos-jail/blob/9b1d5b141...
therealmarv|2 years ago
fkorotkov|2 years ago
thtmnisamnstr|2 years ago
Note: I work at Earthly, but I'm not wrong about this being a good, free, arm64-native workflow for GitHub Actions.
MuffinFlavored|2 years ago
https://github.com/macOScontainers/rund - new code
https://github.com/macOScontainers/moby - fork, 6 commits
https://github.com/macOScontainers/buildkit - fork, 4 commits
https://github.com/macOScontainers/containerd - fork, 5 commits
Would be interesting to see if they can get moby/buildkit/containerd changes upstreamed
slonopotamus|2 years ago
Other part of containerd changes waits for gods-know-what: https://github.com/containerd/containerd/pull/9054
But I haven't gave up yet.
slonopotamus|2 years ago
https://github.com/moby/buildkit/pull/4059
https://github.com/moby/moby/pull/46558
skibz|2 years ago
highwaylights|2 years ago
It's intended to prevent malware from changing system files due to rogue permissions or escalation. With SIP enabled, even the root/sudo user doesn't have rights to change these files.
It also refuses to boot a system with drivers that are not signed by Apple, so as to deter malware from using drivers as an attack vector.
https://en.wikipedia.org/wiki/System_Integrity_Protection
bzzzt|2 years ago
minipark|2 years ago
Squarex|2 years ago
puquesnrn|2 years ago
It clearly links to the GitHub where you can click to see all contributors
I suppose the answer to your question is “people who want macOS containers”, whoever they are. As far as malware, I’d employ whatever your standard practices are for installing GitHub projects
tmpX7dMeXU|2 years ago
daft_pink|2 years ago
I bought an Apple Silicon machine after their presentation claiming that they would have first class docker support, but the reality has been that while the first docker worked well as it was translated, now it wants to default to arm containers and it has become very difficult to use because it doesn't want to use Rosetta 2 containers.
The whole point of using docker is to use the same containers in production as you use in development, so having docker default to these random arm containers means that my containers aren't exactly production, because they are arm based and the servers are not.
I understand that docker is the developer of docker software, but I really wish I could just click a button and force intel based containers in docker as the default and have to opt-in to arm.
If anyone has an easy solution to this let me know. I don't want to spend hours and hours figuring out docker on my mac.
dmeijboom|2 years ago
cdaringe|2 years ago
xena|2 years ago
xcdzvyn|2 years ago
Hnrobert42|2 years ago
ravenstine|2 years ago
At the same time, I don't truly understand why anyone would need to use it. If your preference is to totally work with macOS, then I'm sure this would be perfect for that. Otherwise, what's the advantage?
VMs have really come a long way. Every major OS today has a virtualization framework that makes running another OS extremely performant. Docker on macOS uses a virtual machine, but so what? Performance of individual containers, in my experience, isn't really a problem unless you're doing something with the GPU, and even then there are ways to deal with that. Even a fully-emulated VM using QEMU (without hypervisor or KVM) won't have any noticeable performance penalties in many cases.
IMO, there's a much greater advantage to sticking with Linux. Even if the host isn't Linux, developing and deploying with Linux guests provides a tremendous level of consistency and portability.
But maybe I'll be proven wrong by this project someday soon!
WesolyKubeczek|2 years ago
Of course the drawback would be that the host would see just a fat Linux process and its child processes, much like you can see qemu, but it could be an interesting thing nonetheless, if even for shits and giggles of it.
bryanlarsen|2 years ago
mavili|2 years ago
jayd16|2 years ago
keepamovin|2 years ago
mawalu|2 years ago
RockRobotRock|2 years ago
nathants|2 years ago
linux is great. macos is great. windows is great too. for their intended purposes.
it’s horseless carriages all the way down.
beders|2 years ago
Trying to break out of that is an exercise in futility.
Can you come up with situations where I would run a container instead of just running an app or sys service?
prmoustache|2 years ago
rund is an experimental containerd shim for running macOS containers on macOS.
rund doesn’t offer the usual level of container isolation that is achievable on other OSes due to limited macOS kernel API.
What rund provides:
blablabla123|2 years ago
da39a3ee|2 years ago
shepherdjerred|2 years ago
[0]: https://github.com/shepherdjerred/macos-cross-compiler
nicexe|2 years ago
sourabhv|2 years ago
speedgoose|2 years ago
crabbone|2 years ago
slonopotamus|2 years ago
servercobra|2 years ago
timcobb|2 years ago
amelius|2 years ago
slonopotamus|2 years ago
xyst|2 years ago
slonopotamus|2 years ago
dingi|2 years ago
shepherdjerred|2 years ago
[0]: https://github.com/shepherdjerred/macos-cross-compiler
dvdplm|2 years ago
figmert|2 years ago
unknown|2 years ago
[deleted]
_joel|2 years ago
lloeki|2 years ago
I would be very curious as to how you already run darwin containers.
The only alternative is spinning a macOS VM (including relying on macOS CI machines as a remote job executor)
unknown|2 years ago
[deleted]
tambourine_man|2 years ago
They’re not even trying, now.
pjmlp|2 years ago
unknown|2 years ago
[deleted]
baq|2 years ago
macbook is the best laptop there is but macos...
can't wait for a stable release of Asahi and permission from corporate to install it even in a VM somehow. probably won't happen, but one can dream.
_joel|2 years ago
unknown|2 years ago
[deleted]
icar|2 years ago