For my home lab I started with k3s on VMs which I eventually migrated to a k3s with etcd for HA. I added Raspberry Pi nodes to force myself to deal with multi-architecture builds of my own code in Jenkins and deployments. Some of the Pi's have only wifi and some Ethernet so that got me into node affinity for deploying workloads. At some point I added some bare metal Intel machines to the mix. So now whether it's a Pi, VM, or bare metal machine all I need to do is a base OS install with ssh server, add it to my Ansible inventory and it will be up on my cluster in a few minutes.
I've found Raspberry Pis to be too unreliable for my homelab needs due to constant crashing and random performance drops. Instead, I bought two used 2012 Mac Minis with 16GB RAM, 1TB on eBay for about $120 each. I installed Proxmox on them, and it’s been a much better experience. It's nice not having to deal with ARM. They look decent and, most importantly, are super quiet. So can even use them in the living room. Absolutely zero issues with this setup—boring hardware wins.
It's certainly fun to mess with the hardware, but my personal preference would be to use VMs.
It's cheaper, in my opinion, to get a refurbished mini desktop from Lenovo or whatever. They're "real" PCs, and I've found the raspberry pi hardware limitations to be onerous. It's just really nice to have SATA, NVMe, and PCI Express lanes/slots. For the price of a couple of PIs, you can get a powerful CPU with loads of RAM.
I listened to this kind of preaching and got myself a Lenovo ThinkCentre with Ryzen 5. What everyone forgets to mention is that these are loud. Fans are small and they spin fast. And fanless used mini pc market is kind of dry.
The relatively exotic Cortex ARM, along with reliability, challenges with USB & microSD, and (at least here in AU) cost & availability challenges for the 'larger' boards, really put me off heading too eagerly down the Pi path.
A few weeks ago I picked up an ex-enterprise server with 256GB ECC RAM, 24/48 Xeon cores, and a handful of disks - for AUD$1000 (~ 600 USD)
Yes, it's a noisy box, and yes 2.5" SAS disks are expensive & small - so you want storage separately - but the same applies to a RPi or NUC based lab.
It draws about 100W while idle (and in a home lab environment it takes some effort to make it sweat)
Proxmox installed and runs like a dream.
I'm sticking with Nomad over k8s (or variants) for container orchestrator across several Debian VM's in the one box -- so I'm effectively relying on ECC, dual-PSUs, and hardware RAID5 for my ersatz HA.
I am currently streaming my work on migrating HomelabOS from a docker-compose based implementation to a k8s (actually k3s) based implementation. With Longhorn backing it, I'm pretty excited by the possibilities. It will stay as generic as HomelabOS is, so it will be deployable on anything from a pi on up to big cloud machines.
I won't be doing any 3d printing or anything like that though, so what the author here is doing looks fun in it's own right!
Wasn't aware of HomelabOS, looks pretty interesting!
How is your Longhorn performing? I tried setting it up on my nodes but with Gigabit networking and possibly the Pis pretty average CPUs I would get pretty awful performance both in distributed volumes (with replicas etc) but also on strict-local ones (for reasons I haven't yet figured out).
I've done the same thing with Pi 3Bs/4s basically for the same reasons. I definitely ran into the limitations of the SDCards and set up USB drives and NAS storage class for better performance. I ended up running my own docker registry on my NAS and running all of the images through that before deploying to the k3s cluster. I also hooked up container scanning and automated it the deployments through ansible.
Things I would do differently are using NixOS or bootable containers (CentOS) (side note, bootable NixOS container would be a killer app) and writing my own helm charts instead of fully customizing my manifests and doing the deployments from ansible, and would recommend against Raspberry Pis for the compute as the 3's and 4's don't support limits, e.g. cpu or ram limits, and I wasn't able to set up firecracker containers correctly on the Pis.
I'm also exploring hyperconvergence infrastructure (HCI) as that seems more like my ultimate goal for homelab stuff.
As far as Helm vs Ansible, I'm using Ansible to deploy the basics (bootstrap control plane & worker nodes, networks plugin) and then everything is deployed with IaC (Pulumi) which installs Helm releases.
If you're looking at HCI Proxmox is amazing for homelabbing, with a 3 node cluster or 2 nodes with a qdevice. I use PBS for automated backups and RAIDZ and while there's quite a learning curve and initial setup time actual cluster maintainance is pretty hands off.
With enterprise gear it becomes outdated and then you replace it at your leisure, whereas with consumer equipment it dies so you need to replace it. The disvantage of that is the noise and power consumption so it's a tradeoff you need to consider.
I've been using a Raspberry Pi to serve as a kubernetes master, with a few Intel NUCs to run as the workers, and I've loved it. The whole setup is small and cheap to run; I installed everything into a switch depth rack in my basement. I'm now trying to figure out how to add some GPU to the cluster, and I'm undecided whether to build a switch depth 3U server or go another route. Most people I've talked to with homelabs are running a full depth rack, so they can easily accommodate full size equipment, but I have size constraints so I can't fit something that large.
> These servers were difficult to maintain, monitor, upgrade etc and a number of solutions were developed by system administrators to work with them. It all though felt very cumbersome
There’s a lot of irony saying this about bare metal vis-à-vis k8s. Maintaining individual servers isn’t difficult, doing it at scale with high availability and the requirement of upfront investment (hardware, colo, staffing, etc.) is. Doing k8s at scale isn’t a walk in the park or a cheap date either, though.
It most definitely is not, you are right! I think the main difference IMO is that by paying that price with k8s I at least have something that resembles an application platform and I can easily ship containers, deploy my app and not deal with hardware so much as I'd have to do in the past deploying to servers.
But, as mentioned, I only see it as a developer. My end product with k8s resembles something that's closer to my development tools than what I'd have maintaining a series of servers and using other tools to deploy apps onto.
Cargo culting the processes required for extreme use cases outside of that context is only good for learning to get paid money to do it in those contexts in the future. In and of itself it is very silly. For a human person a process running on an OS on running on a computer is far, far less work, maintenance, and complexity with better performance and longer lifetime.
We all never find nirvana if we don't try. Maybe someone does really get it slick and right.
Eventually this system probably will become quasi stable, sustaining. And for some folks, replicable & practicable. The author will probably soldier through a good amount of the pain then it'll tick along. Doom isn't certain. (But it is probable!)
> a process running on an OS on running on a computer
Such a process can be deployed/backed up/restored in minutes by anyone with basic sysadmin skills and can scale to serve nontrivial user bases while costing much less than SaaS.
I tried doing a homelab with six Nvidia Jetson Nanos using K8S, maintained it for about a year, and I have no desire to ever do that again. I ended up just buying a single rack mount server and using that for two years, and now I bought a mini gaming computer which I use as a single server. Maintaining the k8s cluster was becoming a second job that actually costs me money, that I enjoyed less and less every day, and it made me dread actually using any aspect of my server, meaning that when something broke it would take me a long time to actually muster up the strength to fix anything. My home server runs a Transmission server, Jellyfin, Apache Kafka, Apache Spark, Cassandra, and RabbitMQ, with 32 gigs of RAM, and it works fine.
Distributed systems are cool and they're fun to play with, but the combinatorial explosion of maintenance shouldn't be underestimated. If you're making something that needs to serve 10,000+ users, then it's probably worth it, but homelabs generally aren't that. Generally a homelab situation has like a Plex/Emby/Jellyfin server, a torrent server, a reverse proxy, maybe some kind of message queuing solution, and it generally only has like four concurrent users.
Obviously if your goal is to learn K8s, then doing it with a bunch of Raspberry Pis isn't a bad idea at all, and try to have fun doing it. However, I would warn anyone thinking that they're going to make their NAS a k8s cluster, you're likely going to regret it. I recommend buying a slightly beefier computer and just installing NixOS or something.
I do dread that exact scenario you're mentioning. I know it is a possibility but I'm hoping that should this day arrive I'll be able to put it to rest and hopefully have learned stuff along the way.
I have pretty much landed on a similar simple solution for my homelab. A minipc with dietpi + docker + dockge + bunch of homelab apps using docker compose. Important config backups on a S3 compatible bucket in cloud + photos backup on connected USB HDD.
I opted for bundlewrap over Ansible/TF/etc, and keepalived over MetalLB.
I also didn’t add any kind of real storage or networking: they’re just using WiFi.
After I got it set up and working, and was forced to read up on the high-level k8s concepts, I never really did anything much with it again, but the learning was valuable. :-)
kinda annoying that on this blog post, if i press the up arrow cursor button or page up button it takes me to the top of the page , instead of scrolling up
the page down and cursor down button work as expected
--
kinda found the bug, when you first open the blog, the cursor stays on the top site navigation menu, as long as the cursor is up there, page up or cursor up will take you to the top of the page
click anywhere on the page that will change the cursor position, and now page up and cursor up work as expected
And than your network stops working for couple of days, and you are @#£#&. Or your backups did not work for couple of months, and you find out the hard way!
I have some hardware for fun and learning, but it is well isolated. Using K8s for home infrastructure is bad idea.
sprior|1 year ago
domrdy|1 year ago
srameshc|1 year ago
bloopernova|1 year ago
It's cheaper, in my opinion, to get a refurbished mini desktop from Lenovo or whatever. They're "real" PCs, and I've found the raspberry pi hardware limitations to be onerous. It's just really nice to have SATA, NVMe, and PCI Express lanes/slots. For the price of a couple of PIs, you can get a powerful CPU with loads of RAM.
gdrulia|1 year ago
Jedd|1 year ago
A few weeks ago I picked up an ex-enterprise server with 256GB ECC RAM, 24/48 Xeon cores, and a handful of disks - for AUD$1000 (~ 600 USD)
Yes, it's a noisy box, and yes 2.5" SAS disks are expensive & small - so you want storage separately - but the same applies to a RPi or NUC based lab.
It draws about 100W while idle (and in a home lab environment it takes some effort to make it sweat)
Proxmox installed and runs like a dream.
I'm sticking with Nomad over k8s (or variants) for container orchestrator across several Debian VM's in the one box -- so I'm effectively relying on ECC, dual-PSUs, and hardware RAID5 for my ersatz HA.
NickBusey|1 year ago
I won't be doing any 3d printing or anything like that though, so what the author here is doing looks fun in it's own right!
s3rg4fts|1 year ago
How is your Longhorn performing? I tried setting it up on my nodes but with Gigabit networking and possibly the Pis pretty average CPUs I would get pretty awful performance both in distributed volumes (with replicas etc) but also on strict-local ones (for reasons I haven't yet figured out).
I am now considering using something like https://github.com/rancher/local-path-provisioner, since I mainly intend to use Longhorn for DBs that handle fault tolerance / backups etc on their own.
Kerbonut|1 year ago
Things I would do differently are using NixOS or bootable containers (CentOS) (side note, bootable NixOS container would be a killer app) and writing my own helm charts instead of fully customizing my manifests and doing the deployments from ansible, and would recommend against Raspberry Pis for the compute as the 3's and 4's don't support limits, e.g. cpu or ram limits, and I wasn't able to set up firecracker containers correctly on the Pis.
I'm also exploring hyperconvergence infrastructure (HCI) as that seems more like my ultimate goal for homelab stuff.
s3rg4fts|1 year ago
Wasn't aware of NixOS, looks pretty interesting but I'm not sure about how easy / reliable it'd be to run it on a Pi 5 (https://wiki.nixos.org/wiki/NixOS_on_ARM/Raspberry_Pi_5). I'll be keeping an eye on it though!
As far as Helm vs Ansible, I'm using Ansible to deploy the basics (bootstrap control plane & worker nodes, networks plugin) and then everything is deployed with IaC (Pulumi) which installs Helm releases.
mysteria|1 year ago
With enterprise gear it becomes outdated and then you replace it at your leisure, whereas with consumer equipment it dies so you need to replace it. The disvantage of that is the noise and power consumption so it's a tradeoff you need to consider.
timmahoney|1 year ago
moribvndvs|1 year ago
There’s a lot of irony saying this about bare metal vis-à-vis k8s. Maintaining individual servers isn’t difficult, doing it at scale with high availability and the requirement of upfront investment (hardware, colo, staffing, etc.) is. Doing k8s at scale isn’t a walk in the park or a cheap date either, though.
s3rg4fts|1 year ago
But, as mentioned, I only see it as a developer. My end product with k8s resembles something that's closer to my development tools than what I'd have maintaining a series of servers and using other tools to deploy apps onto.
superkuh|1 year ago
Cargo culting the processes required for extreme use cases outside of that context is only good for learning to get paid money to do it in those contexts in the future. In and of itself it is very silly. For a human person a process running on an OS on running on a computer is far, far less work, maintenance, and complexity with better performance and longer lifetime.
jauntywundrkind|1 year ago
Eventually this system probably will become quasi stable, sustaining. And for some folks, replicable & practicable. The author will probably soldier through a good amount of the pain then it'll tick along. Doom isn't certain. (But it is probable!)
elevation|1 year ago
Such a process can be deployed/backed up/restored in minutes by anyone with basic sysadmin skills and can scale to serve nontrivial user bases while costing much less than SaaS.
Classic sysadmin is seriously underrated.
2OEH8eoCRo0|1 year ago
tombert|1 year ago
I tried doing a homelab with six Nvidia Jetson Nanos using K8S, maintained it for about a year, and I have no desire to ever do that again. I ended up just buying a single rack mount server and using that for two years, and now I bought a mini gaming computer which I use as a single server. Maintaining the k8s cluster was becoming a second job that actually costs me money, that I enjoyed less and less every day, and it made me dread actually using any aspect of my server, meaning that when something broke it would take me a long time to actually muster up the strength to fix anything. My home server runs a Transmission server, Jellyfin, Apache Kafka, Apache Spark, Cassandra, and RabbitMQ, with 32 gigs of RAM, and it works fine.
Distributed systems are cool and they're fun to play with, but the combinatorial explosion of maintenance shouldn't be underestimated. If you're making something that needs to serve 10,000+ users, then it's probably worth it, but homelabs generally aren't that. Generally a homelab situation has like a Plex/Emby/Jellyfin server, a torrent server, a reverse proxy, maybe some kind of message queuing solution, and it generally only has like four concurrent users.
Obviously if your goal is to learn K8s, then doing it with a bunch of Raspberry Pis isn't a bad idea at all, and try to have fun doing it. However, I would warn anyone thinking that they're going to make their NAS a k8s cluster, you're likely going to regret it. I recommend buying a slightly beefier computer and just installing NixOS or something.
s3rg4fts|1 year ago
mirchiseth|1 year ago
darkwater|1 year ago
zellyn|1 year ago
I opted for bundlewrap over Ansible/TF/etc, and keepalived over MetalLB.
I also didn’t add any kind of real storage or networking: they’re just using WiFi.
After I got it set up and working, and was forced to read up on the high-level k8s concepts, I never really did anything much with it again, but the learning was valuable. :-)
systems|1 year ago
kinda annoying that on this blog post, if i press the up arrow cursor button or page up button it takes me to the top of the page , instead of scrolling up
the page down and cursor down button work as expected
--
kinda found the bug, when you first open the blog, the cursor stays on the top site navigation menu, as long as the cursor is up there, page up or cursor up will take you to the top of the page
click anywhere on the page that will change the cursor position, and now page up and cursor up work as expected
englishspot|1 year ago
sed3|1 year ago
I have some hardware for fun and learning, but it is well isolated. Using K8s for home infrastructure is bad idea.
ta988|1 year ago
unknown|1 year ago
[deleted]