Along with the submitter, I am also on the team of maintainers who volunteered to help with maintenance of this project after tteck's sad news that they were entering hospice (1). The team members are all motivated individuals, who are enthusiastic on carrying on tteck's legacy.
We are moving forward in a transparent manner and I am more than happy to answer any questions.
I have been using Proxmox VE for several years now and have most of my services running as docker containers in one VM. This always bothered me because I wanted to be able to control the individual services and their backup jobs using the Proxmox interface. After checking out these scripts I already moved a couple services (Caddy and Wireguard) over to LXC containers and am very impressed by how easy it was to do.
Basically I just wanted to say thanks to everyone involved in making these scripts, it has left me with a great first impression.
Careful. I too thought about this, but docker containers have the following benefits over the LXC scripts:
- Updates and automatic upgrades between major versions.
- The developer who wrote the software created the container (most of the time), this means its a supported environment. Also, as they have the insight into the application and future upgrades the environment has been setup correctly for each version.
If you want to achieve your goal, I'd suggest an LXC with your favourite Linux distro + docker + app container(s) for each app you have. It gives you the same thing, but with the benefits above.
A bit of a tangent. I've been trying to manage libvirt& Unraid through terraform, but have run into issue after issue. I'm about given up, and will just manage the virtual machines manually...
What's the virtualization technology on proxmox?
What's the advantage to using something like this as opposed to terraform or salt stack or Ansible?
It is also worth mentioning that Proxmox uses ZFS making snapshotting quick and Proxmox also has a very good backup system.
If you want to treat your self-hosted applications as "sheep" (1) , then terraform k8s etc. is a better bet.
But if you are happy to manually restore from a backup or snapshot when something goes wrong, or automatically have your LXC container shifted to different hardware if you have a cluster, then Proxmox is for you. The reality is that in a home setup you will spend about as much or less time maintaining your "pets" than than you would your "farm".
Proxmox is more about the management of the hosts and resources on them, including the live migration of VMs between hosts, support for some types of HA and failover.
You can likely manage the configuration of the VMs themselves through terraform or similar in combination with Proxmox if that's your desire.
proxmox is using KVM for virtualization and Linux Containers (LXC) for the containers. I agree that something like terraform and/or ansible would make more sense for an IAC (infrastructure as code) deployment. Most of the people I talk to that use proxmox for a homelab prefer to do things manually and don't bother with any IAC implementation.
For work I'm a firm believer in reproducible environments and IAC. We actually a combination of vagrant, libvirt, and KVM to spin up local clusters for quick testing and development. It works out pretty well, but in my homelab I don't have anything complicated enough to bother setting up terraform/ansible for. Although I imagine if my server crashed I probably wouldn't think that way anymore.
I have been looking into setting up my first Proxmox box, here is my take as a newcomer.
I wanted to do what I think is a very basic and very common setup: Modem > proxmox box > OPNsense VM > physical wifi router via onboard 10Gb NIC + internal network VMs like OMV etc. The goal is to add a full network filter via OPNsense, and allow access to a media sever and backup etc from the internal network.
I see no OPNsense, OMV script is basically contra-indicated because it should be a VM instead of the LXC container, and I don't see any glue scripts to get VMs talking to each other, which is an important part of Proxmox configuration. So it looks like there is room here to get some basic setup scripts for a simple home server either improved or added to the collection.
No it isn't basic and common (it is for me but perhaps not for you and certainly not for most people)
OK, so you want to virtualise a router and firewall. That's fine. I have deployed roughly 200 pfSense firewall/routers as VMs and physical boxes and OPNSense is similar, so I can probably help.
At a minimum you will need two physical interfaces (one will actually do but you will need to know what you are doing!). You need "WAN" and "LAN". OPNSense is still FreeBSD based, I think, so it will not run in a L[inux]XC container for obvious reasons.
Your last paragraph seems rather confused. I don't know what you mean by "glue scripts". VMs communicate via networks
I suggest you try a few experiments to get to grips with virtualisation properly and then move on from there. If you swing by the Proxmox forums with specific issues we'll try to help out but in the end you need to dive in full on ... or not.
There is no OPNSense script I think historically in part because any misconfig could expose the Promox instance to the world. It is easy enough for advanced users to spin up a VM with the ISO. There has been a request for a OPNSense script made recently.
I agree with OMV. It certainly can be used as is, but not usually how people want to use it. A note was added to the script a few days ago.
> I don't see any glue scripts to get VMs talking to each other
There is a Tailscale script which technically helps them talk to each other (over Tailscale) :)
The scripts are designed to setup self contained LCX containers. We are trying to avoid building our own k8s.
I use Proxmox with an OPNSense VM and have multiple NICs - one is dedicated to the fibre ONT. I also use an external wifi mesh.
I have a couple of other vms (unRaid hosting Dockers with sata card passthrough for legacy reasons and a vm for Home Assistant OS) and lots of other LXCs. It works superbly.
I'll definitely look into the docker LXC and Home Assistant VM. I'd been using docker in a VM on proxmox, successfully mind you, but perhaps there's some more efficiency to squeeze...
I decided to run proxmox on my homelab rather than having a k8s setup, and I've come to sort of regret it. LXCs are awesome, but being bound to just them or qemu VMs doesn't fit all of my needs. With Kubernetes I could just add support for lightweight VMs (Firecracker hypervisor, or unikernels or something) with a project like Kata. Proxmox is just not extensible.
It's also just not amenable to automation or reproducible builds in the same way as an established pod manager like Kubernetes: there's no support that I can find for Terraform, and so you're stuck with regular full-disk backups and maybe some Chef/Ansible/Puppet tooling, which I don't want to invest in [re]learning.
Still, very cool resource management and passthrough model, and it's easy to set up and maintain, with a nice control panel.
Also, easy-to-install ZFS makes it hard for me to cajol myself into trying something else. And if I want k8s for play time I can always spin up (a/some) VM(s).
For awhile I ran Docker Swarm with a bunch of SBCs, then k8s, then just a big server running Ubuntu + Cockpit, then Proxmox, until I have finally settled on NixOS.
NixOS has decent container support if necessary, but I've found that its declarative nature means I almost never bother with containers. "Uninstalling" something is generally as simple as "remove it from the config file, rebuild", and it's not hard to do cgroupey stuff if you need to manage memory and the like.
Not to mention that I think NixOS's nginx DSL is wonderful. It's so nice being able to have my proxy configs (along with LetsEncrypt) managed directly (and correctly) by the config environment instead of me writing my own scripts and the like.
(I'm not sure if there are any distributed NixOS things, because I could totally see something neat being built on Flakes)
My homelab has never been simpler and I've never been happier with it.
I've used this[1] Terraform provider together with the Talos[2] distribution for deploying a Kubernetes cluster. I agree that the APIs available with Proxmox are not fully featured, but it more than suits my needs.
I'm running a four node cluster on salvaged SFF machines backing up lvm snapshots to home brewed TruNAS storage and it all makes me happy.
Proxmox doesn’t preclude you from having k8s. You can create VM(s) in Proxmox and then install k8s on them, then run your app workloads in k8s.
You do have to treat Proxmox VMs like “pets, not cattle” since they are more difficult to automate, but that’s the same story as if you were managing your k8s host on bare metal too. The benefit with Proxmox-hosted VMs though is that you can use Proxmox for whole-VM backups and migrations, so you can have the best of both proxmox and k8s!
Different technologies are for different approaches and applications.
It's relatively trivial to use the pve command line utility to create or modify vms in proxmox.
Still, the originating reason of this post is due to a large number of useful scripts to help make things more manageable and maintainable, and the founder of it having to step away, and there being gratitude for their help to make things much more manageable.
But you can just chuck Kubernetes nodes on Proxmox? I have my nodes running on XCP-ng. The beauty of running a hypervisor is maximum flexibility. I can try out different distros etc, either for k8s nodes or otherwise. I run my router on there (opnsense). I can play with stuff like nix and guix and could even install Windows if for some reason I wanted to.
I have a single VM on my proxmox server that I spin up all my docker containers in... This is the simplest thing I could think of in terms of config. I haven't had to wire containers together, though, so maybe I've found the sweet spot for my deployment needs.
Clustering, migration, high availability, backups, Ceph integration, virtual networks as of recent, can do containers as well as VMs to name a few off-hand. The web interface is optional, too. You should check out their webpage for more.
KVM is just kernel side of things, it's not full vmm by itself, you always need some userspace application too. Firecracker, Qemu, Cloud-Hypervisor are some vmms built on top of kvm.
While qemu is common way of using kvm, but running qemu directly is quite annoying. So you have stuff like libvirt and proxmox as wrappers around qemu.
Unfortunately until this day Proxmox doesn’t support full disk encryption oob, despite using the word „Enterprise“ in the first paragraph of their website. Yes, you can go your own way and install on an encrypted Debian, but you will miss important features and are on your own. It all comes down to ZFS not treating FDE as a first class citizen.
Am I right that proxmox takes over your entire machine?
I have been using a combination of docker and lxc/lxd to manage my VMs. But, cockpit (on ubuntu) does not give me a perfect experience for managing running VMS, etc.
I wish there was a good solution for all of this. But, it feels like you need to cobble together a bunch of kibana tools to get true monitoring.
Proxmox runs on top of Debian. The Debian is part of the install but I think you could install the packages separately on an existing Debian install. You could even install cockpit if you want to.
I wanted to delegate management of my raid array to higher level tools since it died on me seemingly for nothing (I was able to recover all the drives but none of the files).
I tried TrueNAS but it's very rigid. Proxmox seems to give you more control over what's installed on the server but it's also quite locked down. Don't remember exactly what was it that pushed me off Proxmox. I think it was that I needed to manage some VMs over LXD API and others over proxmox and I couldn't mix and match, I had to choose one without extra hacks.
Usually, proxmox is the base OS (meaing replaces whatever Linux/Windows/FreeBSD/etc that there was before). It is possible to run Proxmox inside KVM, but that isn't the usual choice.
Tangential question - what are people using their homelab for / what are some interesting or useful projects you've spun up on them? I've been thinking about setting one up but not 100% sure I'd find use out of it :)
At the risk of leaking info to any toes I step on w.r.t my home environment,
- home assistant
- Network Video Recorder
- Jellyfin
- network management such as ubiquiti or omada etc
- vault warden/1pass/other secrets servers
- tailscale or wire guard server
- build server/k8s test environment
- private artifactory or mirror (especially useful if you're using the same distro on a bunch of devices but don't want to overload the actual mirror+improves download times)
- torrents (someone's gotta seed Wikipedia)
- onsite backups
- bastion into your home network (see also: wire guard)
- some people even use it for their router
You could also take a look at the tteck scripts, there's a bunch of cool stuff in there
I have Proxmox on my older (11th gen) Intel NUC - it runs my Unifi network controller for my WiFi APs, an Unbound DNS cache (because my router doesn't support DNS over TLS or DNS over HTTPS), a NAS (using ZFS on some hard drives in a USB 3.2 Gen 2, 6-drive disk chassis - the ZFS managed directly by Proxmox and then a Debian container just doing Samba).
That's all for now but I've just installed Home Assistant but haven't set that up yet. I also intend to try out Jellyfin as a media server and Frigate as a video recorder when I get some cheap cameras.
I started with a home lab a few months back. Its basically an old miniPC running Proxmox virtualization and LCX containers. Mostly it helps me learn various technology I have not much experience with in my work. I run about 20+ services on the home lab. Some of them include:
Open WebUI which can connect to the OpenAI api or a local Ollama LLM. You can also connect various tools to the LLM like a calculator or web search to augment them. The AI has helped me learn how to configure and debug stuff. Like I got step-ca to setup a local certificate authority and give certificates to my various internal services. I played around with configuring Caddy and Nginx along with ACME to the the step-ca. The LLM was even helping be debug my config files.
I'm also using Hoarder for bookmarking and it can use AI to automatically tag your bookmarks. It can even backup the webpages.
I've been using Mealie to clip and save online recipes.
I'm running Uptime Kuma to check if my computers and services are up and if they are down, I'll get a notification.
- home assistant (just a few currently, more soon)
- paperless:absolutely awesome document management system
- immich: image management with automatic synchro of my mobile taken images (ML features)
- tailscale
- StirlingPDF: simple tools for all things PDF
I doubt you have a whole lot of control over this, but this website is misery to use on a phone. Browsing scripts gives you six or ten tiles per page, navigation buttons are only in the header. There are no page numbers or any indication of progress. At a glance, there appears to be no way of filtering or sorting, although there is some arbitrary grouping being applied in the middle of the list. Also the script descriptions only show three lines of text, which is not nearly enough to give a clue to what the script is for.
This is probably the worst implemented list view I've ever seen. Completely useless.
Not sure if we're seeing the same site but this site is better than 90% of the web when accessed via mobile for me. No page number when navigating forward/back through scripts, true, but I doubt you want to go through more than 1 or 2 pages without searching first. Search worked for me and turned up what I wanted on the first hit. Descriptions being 3 lines is pretty much the best it can do with the limited screen space; to improve this I feel you would need dedicated summarized content for mobile, which is a price that most platforms don't want to pay. Your description made me think that this was in the bottom 1% of mobile sites but honestly it's above average.
daqnz|1 year ago
We are moving forward in a transparent manner and I am more than happy to answer any questions.
(1) https://news.ycombinator.com/item?id=42016605
lotophage|1 year ago
I only recently went down the homelab/selfhosted path and the majority of my containers were setup using tteck's scripts.
IseardMi|1 year ago
Basically I just wanted to say thanks to everyone involved in making these scripts, it has left me with a great first impression.
crimsonnoodle58|1 year ago
- Updates and automatic upgrades between major versions.
- The developer who wrote the software created the container (most of the time), this means its a supported environment. Also, as they have the insight into the application and future upgrades the environment has been setup correctly for each version.
If you want to achieve your goal, I'd suggest an LXC with your favourite Linux distro + docker + app container(s) for each app you have. It gives you the same thing, but with the benefits above.
prettyStandard|1 year ago
What's the virtualization technology on proxmox?
What's the advantage to using something like this as opposed to terraform or salt stack or Ansible?
daqnz|1 year ago
If you want to treat your self-hosted applications as "sheep" (1) , then terraform k8s etc. is a better bet.
But if you are happy to manually restore from a backup or snapshot when something goes wrong, or automatically have your LXC container shifted to different hardware if you have a cluster, then Proxmox is for you. The reality is that in a home setup you will spend about as much or less time maintaining your "pets" than than you would your "farm".
(1) I write this from New Zealand
justinclift|1 year ago
You can likely manage the configuration of the VMs themselves through terraform or similar in combination with Proxmox if that's your desire.
beardbound|1 year ago
For work I'm a firm believer in reproducible environments and IAC. We actually a combination of vagrant, libvirt, and KVM to spin up local clusters for quick testing and development. It works out pretty well, but in my homelab I don't have anything complicated enough to bother setting up terraform/ansible for. Although I imagine if my server crashed I probably wouldn't think that way anymore.
heatmiser|1 year ago
- provision VMs with Terraform - configure/maintain your VM with something like Ansible
The provider also allows your to schedule LXC if you'd like to target that instead.
[1]: https://github.com/Telmate/terraform-provider-proxmox
sgc|1 year ago
I wanted to do what I think is a very basic and very common setup: Modem > proxmox box > OPNsense VM > physical wifi router via onboard 10Gb NIC + internal network VMs like OMV etc. The goal is to add a full network filter via OPNsense, and allow access to a media sever and backup etc from the internal network.
I see no OPNsense, OMV script is basically contra-indicated because it should be a VM instead of the LXC container, and I don't see any glue scripts to get VMs talking to each other, which is an important part of Proxmox configuration. So it looks like there is room here to get some basic setup scripts for a simple home server either improved or added to the collection.
gerdesj|1 year ago
OK, so you want to virtualise a router and firewall. That's fine. I have deployed roughly 200 pfSense firewall/routers as VMs and physical boxes and OPNSense is similar, so I can probably help.
At a minimum you will need two physical interfaces (one will actually do but you will need to know what you are doing!). You need "WAN" and "LAN". OPNSense is still FreeBSD based, I think, so it will not run in a L[inux]XC container for obvious reasons.
Your last paragraph seems rather confused. I don't know what you mean by "glue scripts". VMs communicate via networks
I suggest you try a few experiments to get to grips with virtualisation properly and then move on from there. If you swing by the Proxmox forums with specific issues we'll try to help out but in the end you need to dive in full on ... or not.
m463|1 year ago
The router port to the proxmox machine is set up for tagged packets that isolate incoming/outgoing traffic.
After that my VMs and Containers are easily set up to "live" on one or more networks.
For me the firewall rules on the router determine what traffic can be relayed between vlans through the router.
I'm pretty sure you could set up opnsense running in a container or vm to do the same thing, selectively passing traffic from one vlan to another.
daqnz|1 year ago
I agree with OMV. It certainly can be used as is, but not usually how people want to use it. A note was added to the script a few days ago.
> I don't see any glue scripts to get VMs talking to each other
There is a Tailscale script which technically helps them talk to each other (over Tailscale) :)
The scripts are designed to setup self contained LCX containers. We are trying to avoid building our own k8s.
spants|1 year ago
TheSmiddy|1 year ago
I'm confused by what you mean here? Don't they just use the network like any other computer?
I haven't had to do any special configuration to get my VMs to talk to each other.
bigmattystyles|1 year ago
BLKNSLVR|1 year ago
paranoidrobot|1 year ago
I used some of tteck's helper scripts to set up mqtt and zigbee2mqtt LXC containers with a passthrough of the USB zigbee device.
daqnz|1 year ago
WaxProlix|1 year ago
It's also just not amenable to automation or reproducible builds in the same way as an established pod manager like Kubernetes: there's no support that I can find for Terraform, and so you're stuck with regular full-disk backups and maybe some Chef/Ansible/Puppet tooling, which I don't want to invest in [re]learning.
Still, very cool resource management and passthrough model, and it's easy to set up and maintain, with a nice control panel.
KAMSPioneer|1 year ago
However there is a (community) TF module...? https://registry.terraform.io/providers/Telmate/proxmox/late... (I have no experience with it as I typically reach for Ansible).
Also, easy-to-install ZFS makes it hard for me to cajol myself into trying something else. And if I want k8s for play time I can always spin up (a/some) VM(s).
tombert|1 year ago
For awhile I ran Docker Swarm with a bunch of SBCs, then k8s, then just a big server running Ubuntu + Cockpit, then Proxmox, until I have finally settled on NixOS.
NixOS has decent container support if necessary, but I've found that its declarative nature means I almost never bother with containers. "Uninstalling" something is generally as simple as "remove it from the config file, rebuild", and it's not hard to do cgroupey stuff if you need to manage memory and the like.
Not to mention that I think NixOS's nginx DSL is wonderful. It's so nice being able to have my proxy configs (along with LetsEncrypt) managed directly (and correctly) by the config environment instead of me writing my own scripts and the like.
(I'm not sure if there are any distributed NixOS things, because I could totally see something neat being built on Flakes)
My homelab has never been simpler and I've never been happier with it.
tupilaq|1 year ago
I'm running a four node cluster on salvaged SFF machines backing up lvm snapshots to home brewed TruNAS storage and it all makes me happy.
----
[1] https://github.com/Telmate/terraform-provider-proxmox
[2] https://factory.talos.dev/
placardloop|1 year ago
You do have to treat Proxmox VMs like “pets, not cattle” since they are more difficult to automate, but that’s the same story as if you were managing your k8s host on bare metal too. The benefit with Proxmox-hosted VMs though is that you can use Proxmox for whole-VM backups and migrations, so you can have the best of both proxmox and k8s!
j45|1 year ago
It's relatively trivial to use the pve command line utility to create or modify vms in proxmox.
Still, the originating reason of this post is due to a large number of useful scripts to help make things more manageable and maintainable, and the founder of it having to step away, and there being gratitude for their help to make things much more manageable.
globular-toast|1 year ago
ZYbCRq22HbJ2y7|1 year ago
https://linuxcontainers.org/incus/
all2|1 year ago
midtake|1 year ago
eddieroger|1 year ago
zokier|1 year ago
While qemu is common way of using kvm, but running qemu directly is quite annoying. So you have stuff like libvirt and proxmox as wrappers around qemu.
ibuprofn|1 year ago
BodyCulture|1 year ago
Very sad state of affairs.
xrd|1 year ago
I have been using a combination of docker and lxc/lxd to manage my VMs. But, cockpit (on ubuntu) does not give me a perfect experience for managing running VMS, etc.
I wish there was a good solution for all of this. But, it feels like you need to cobble together a bunch of kibana tools to get true monitoring.
pkaye|1 year ago
gchamonlive|1 year ago
I tried TrueNAS but it's very rigid. Proxmox seems to give you more control over what's installed on the server but it's also quite locked down. Don't remember exactly what was it that pushed me off Proxmox. I think it was that I needed to manage some VMs over LXD API and others over proxmox and I couldn't mix and match, I had to choose one without extra hacks.
alsetmusic|1 year ago
jdboyd|1 year ago
unixhero|1 year ago
jamifsud|1 year ago
hughesjj|1 year ago
- home assistant - Network Video Recorder - Jellyfin - network management such as ubiquiti or omada etc - vault warden/1pass/other secrets servers - tailscale or wire guard server - build server/k8s test environment - private artifactory or mirror (especially useful if you're using the same distro on a bunch of devices but don't want to overload the actual mirror+improves download times) - torrents (someone's gotta seed Wikipedia) - onsite backups - bastion into your home network (see also: wire guard) - some people even use it for their router
You could also take a look at the tteck scripts, there's a bunch of cool stuff in there
stephen_g|1 year ago
That's all for now but I've just installed Home Assistant but haven't set that up yet. I also intend to try out Jellyfin as a media server and Frigate as a video recorder when I get some cheap cameras.
pkaye|1 year ago
Open WebUI which can connect to the OpenAI api or a local Ollama LLM. You can also connect various tools to the LLM like a calculator or web search to augment them. The AI has helped me learn how to configure and debug stuff. Like I got step-ca to setup a local certificate authority and give certificates to my various internal services. I played around with configuring Caddy and Nginx along with ACME to the the step-ca. The LLM was even helping be debug my config files.
I'm also using Hoarder for bookmarking and it can use AI to automatically tag your bookmarks. It can even backup the webpages.
I've been using Mealie to clip and save online recipes.
I'm running Uptime Kuma to check if my computers and services are up and if they are down, I'll get a notification.
hansoolo|1 year ago
- home assistant (just a few currently, more soon) - paperless:absolutely awesome document management system - immich: image management with automatic synchro of my mobile taken images (ML features) - tailscale - StirlingPDF: simple tools for all things PDF
grahamj|1 year ago
- Postgres, Maria, Influx, Grafana
- Plex, Arr suite, Transmission
- Ollama, OpenWebUI
- Web change tracker
- Teslamate
evoke4908|1 year ago
This is probably the worst implemented list view I've ever seen. Completely useless.
sigseg1v|1 year ago
mateus1|1 year ago
I’ve just built my first homelab and have favored OpenMediaVault which seems better suited for my use.
unknown|1 year ago
[deleted]
unknown|1 year ago
[deleted]