Yo dawg, I heard you liked containers so I put a container inside your container!
In all seriousness though, I wonder if this could solve one of our issues of running mutli-container apps in Azure, which is currently causing us some trouble in regards to VNETs and Private Endpoints. I’m sure it’s likely because we should be using Kubernetes or similar, or at the very least have someone who knows how to actually set the Ops part of DevOps up better than we currently do. Long story short, those things won’t realistically happen, and if we could solve them by running multiple podmans inside a docker docker-container deployed as an azure app then we could likely fix our security issues of not being able to setup VNETs and PEs on multi-container apps because it’s been a “up-coming” feature for at least a year.
I’d be interested to know your thoughts on it. The efficiency doesn’t matter as much, our workloads are around 1-5% of our Azure budget which is basically a blip on the bigger budgets.
This is very useful for running containers in lightweight untrusted environments.
I personally use this setup for my GitLab CI runners, using a mix of podman and buildah to build containers. I can't 100% trust the container build script, so avoiding having to expose a Docker socket to that untrusted code is a critical safety layer.
I realize that Linux wasn't made for containerization originally and that's why it's so so painful to nest containers, run them without root etc. But I wish we'd already live in a world where all binaries (maybe with the exception of command line tools) ran containerized by default and I, as the user, had to grant explicit access to files, network, hardware/devices, etc. Not just for security reasons (though it'd be nice if `npm install` and `pip install` couldn't mess up my entire system) but also to make external dependencies more explicit.
Isn't that what Snap and Flatpak are doing? 1) Include dependencies (there can be shared runtimes but they don't mess up the host), 2) sandboxed by default and you need to grant permissions.
I built Packj [1] sandboxing for securing “pip/NPM install”. It uses strace for sandboxing and blocks access to sensitive files and limits traffic to known-good IP addresses.
It's a shame that multi-ring security never really made a comeback.
The ability to let any ring (such as a user ring) handle the library and system calls of an outer ring (such as a container ring) seems like it would be convenient.
Alternately, some microkernels have a more peer-to-peer setup which could work as well.
It's not so painful and most pain comes from Linux insistence on /proc-based APIs as opposed to something similar to Solaris-descendants where it's a json spec (I think xml originally)
Sounds like android or what a modern linux distro should look like.
Sadly both most used userspace distros and the kernel they come with are borderline dangerous and shouldn't be any trusted by people using them, even if they're technical enough to feel comfortable within their environment.
> Most of this has historically been related to Docker in Docker (DIND), but now, people also want to run Podman in Podman (PINP) or Podman in Docker (PIND).
Kind is super cool! I used it for my thesis, as I needed many nodes without getting super into k8s configuration. I love, love kind. If you're into learning Kubernetes, it's probably the easiest way to get yourself a cluster quickly.
For example, your CI/CD agent is running in a container, and has to build and test containerized software. You can use something like buildah to build your app containers without a container runtime, but you want to run them to run the tests.
Ideally, you would want to do this while remaining platform-agnostic, so your agents will be able to start a container no matter where they are themselves: on bare metal, in Podman, in Docker, in k8s.
Ive been reading every line of this blog for a week trying to get it to work — and failed.
The goal is container driven development. But if all my dev tools are inside a container, how do I iterate on building new containers within a container? This is the core feature I wanted.
But unfortunately, this challenge proved way too difficult to pull off and nothing seemed to work for me.
My guess is these setups heavily depend on the host machine and how podman was setup on it which breaks the entire value of using container driven dev.
I ship containers to customers. Would be nice to run tests against them before releasing a container and it would be nice if those tests themselves could run in a container in order to have a clean test environment.
Ever since we started moving to Docker it's been a massive pain. I'm on a maintenance team doing both legacy (non-Docker) and the occasional Docker-based system that gets handed to us (and then a few months later taken back, it's kind of annoying), and not a single member of my team has gotten the Docker-based systems to work. We all end up with different issues getting Docker to run, or for the different containers to communicate, or being able to browse to the app running in the container. Different systems from different teams also have a tendency to interfere with each other. Two years after this started I finally got an admission from the main development team that every new dev on their team has had similar problems.
On the flipside, LXD containers have "just worked" for everyone who has tried it so far. We've been able to use them on a whim for testing stuff, no problems at all.
So I've been wondering for a while if we could use LXD containers to provide a clean slate to run the Docker containers inside of, maybe then we'd at least all have the same problems, if not be able to solve them entirely.
I use MicroOS (https://microos.opensuse.org/), to keep the base operating system clean you'd install helper tools for constructing containers in a container... so two levels of containers would be very helpful
Being able to run rootful stuff in rootless containers is nice for CI where you need for example to install a bunch of stuff or mount things. There are stuff that requires being root, and you might not want to give a real root access to your CI.
Here's another one. I have an app which spawns subprocesses to do computations. I would love to put the subprocesses in containers to constrain their resource usage (i know that's not the only way to do it, but it's an effective and well-understood way to do it). But i would also like to able to run my application in a container!
Building containers within a CI system, for example. CI jobs run as images, with a DinD service attached, which they can use to build images themselves.
Over complicating the engineering (Docker) involved in getting some x86 code into RAM and on a CPU stack, by over complicating further (DIND.) It's essentially another way to abstract and abstraction, and engineers are finding ways to justify it.
In e.g. the website example, the parent podman container does everything from running the integration tests to converting a video of the test at the end into a slowed down GIF and combines it with screenshots to put in the docs. It orchestrates the child podman containers - playwright + the webapp.
I'm certain that if all this tooling were run on different host machines it would have a myriad of "works on my machine" problems all over the place - whether due to mac, WSL, weird linux distros, github actions idiosyncrasies or whatever.
I'm equally certain that if the tooling were bundled in the app container it would needlessly fatten it with unnecessary and potentially conflicting dependencies. I don't want video conversion tooling installed in my web app even if I do want it to generate my docs.
I just tried this out.
The new systemd directive OpenFile=
opens up the possibility to pass the file descriptor
of a file from the host to a container running in a container.
(using rootless Podman running rootless Podman)
sudo systemd-run --property User=test --property OpenFile=/etc/secretfile.txt --collect --pipe --wait --quiet podman run --security-opt label=disable --user podman --device /dev/fuse quay.io/podman/stable podman run -q alpine sh -c "cat <&3"
[+] [-] devjab|2 years ago|reply
In all seriousness though, I wonder if this could solve one of our issues of running mutli-container apps in Azure, which is currently causing us some trouble in regards to VNETs and Private Endpoints. I’m sure it’s likely because we should be using Kubernetes or similar, or at the very least have someone who knows how to actually set the Ops part of DevOps up better than we currently do. Long story short, those things won’t realistically happen, and if we could solve them by running multiple podmans inside a docker docker-container deployed as an azure app then we could likely fix our security issues of not being able to setup VNETs and PEs on multi-container apps because it’s been a “up-coming” feature for at least a year.
I’d be interested to know your thoughts on it. The efficiency doesn’t matter as much, our workloads are around 1-5% of our Azure budget which is basically a blip on the bigger budgets.
[+] [-] hnlmorg|2 years ago|reply
I’m fact currently using Consul for one such problem as it happens: managing Docker from inside Docker.
[+] [-] mhio|2 years ago|reply
[+] [-] evntdrvn|2 years ago|reply
[+] [-] RandomBK|2 years ago|reply
I personally use this setup for my GitLab CI runners, using a mix of podman and buildah to build containers. I can't 100% trust the container build script, so avoiding having to expose a Docker socket to that untrusted code is a critical safety layer.
[+] [-] storrgie|2 years ago|reply
[+] [-] codethief|2 years ago|reply
[+] [-] PhilippGille|2 years ago|reply
E.g. Snap sandboxing: https://snapcraft.io/docs/security-sandboxing
[+] [-] arcanemachiner|2 years ago|reply
[+] [-] ashishbijlani|2 years ago|reply
1. https://github.com/ossillate-inc/packj
[+] [-] musicale|2 years ago|reply
The ability to let any ring (such as a user ring) handle the library and system calls of an outer ring (such as a container ring) seems like it would be convenient.
Alternately, some microkernels have a more peer-to-peer setup which could work as well.
[+] [-] dilyevsky|2 years ago|reply
[+] [-] kytazo|2 years ago|reply
Sadly both most used userspace distros and the kernel they come with are borderline dangerous and shouldn't be any trusted by people using them, even if they're technical enough to feel comfortable within their environment.
[+] [-] rexsteroxxy|2 years ago|reply
So close to some acronym puns :D
[+] [-] debarshri|2 years ago|reply
[1] https://kind.sigs.k8s.io/
[+] [-] spyremeown|2 years ago|reply
[+] [-] sgsag33|2 years ago|reply
[+] [-] orthoxerox|2 years ago|reply
Ideally, you would want to do this while remaining platform-agnostic, so your agents will be able to start a container no matter where they are themselves: on bare metal, in Podman, in Docker, in k8s.
[+] [-] actionfromafar|2 years ago|reply
Either way, it will be done. Ask IBM. What's the use case for running OS360 VMs inside OS360 VMs?
We repeat the same patterns IBM discovered in the 1960s, but with more building materials.
[+] [-] qudat|2 years ago|reply
The goal is container driven development. But if all my dev tools are inside a container, how do I iterate on building new containers within a container? This is the core feature I wanted.
But unfortunately, this challenge proved way too difficult to pull off and nothing seemed to work for me.
My guess is these setups heavily depend on the host machine and how podman was setup on it which breaks the entire value of using container driven dev.
https://bower.sh/opensuse-microos-container-dev
[+] [-] hkwerf|2 years ago|reply
[+] [-] Izkata|2 years ago|reply
On the flipside, LXD containers have "just worked" for everyone who has tried it so far. We've been able to use them on a whim for testing stuff, no problems at all.
So I've been wondering for a while if we could use LXD containers to provide a clean slate to run the Docker containers inside of, maybe then we'd at least all have the same problems, if not be able to solve them entirely.
[+] [-] zzbn00|2 years ago|reply
[+] [-] Reventlov|2 years ago|reply
[+] [-] twic|2 years ago|reply
[+] [-] Hendrikto|2 years ago|reply
[+] [-] VoodooJuJu|2 years ago|reply
[+] [-] movedx|2 years ago|reply
[+] [-] hitchdev|2 years ago|reply
https://github.com/hitchdev/hitchstory/tree/master/examples
In e.g. the website example, the parent podman container does everything from running the integration tests to converting a video of the test at the end into a slowed down GIF and combines it with screenshots to put in the docs. It orchestrates the child podman containers - playwright + the webapp.
I'm certain that if all this tooling were run on different host machines it would have a myriad of "works on my machine" problems all over the place - whether due to mac, WSL, weird linux distros, github actions idiosyncrasies or whatever.
I'm equally certain that if the tooling were bundled in the app container it would needlessly fatten it with unnecessary and potentially conflicting dependencies. I don't want video conversion tooling installed in my web app even if I do want it to generate my docs.
[+] [-] LeoPanthera|2 years ago|reply
[+] [-] BossingAround|2 years ago|reply
[+] [-] eriksjolund|2 years ago|reply
sudo systemd-run --property User=test --property OpenFile=/etc/secretfile.txt --collect --pipe --wait --quiet podman run --security-opt label=disable --user podman --device /dev/fuse quay.io/podman/stable podman run -q alpine sh -c "cat <&3"
[+] [-] user3939382|2 years ago|reply
[+] [-] onesphere|2 years ago|reply
[+] [-] unknown|2 years ago|reply
[deleted]