[s3rg4fts]

I think the 3/4s have a lot of limitations indeed. The 5s are a bit more powerful, so I'm expecting your experience would be better.

Wasn't aware of NixOS, looks pretty interesting but I'm not sure about how easy / reliable it'd be to run it on a Pi 5 (https://wiki.nixos.org/wiki/NixOS_on_ARM/Raspberry_Pi_5). I'll be keeping an eye on it though!

As far as Helm vs Ansible, I'm using Ansible to deploy the basics (bootstrap control plane & worker nodes, networks plugin) and then everything is deployed with IaC (Pulumi) which installs Helm releases.

[Kerbonut]

I built mine before the 5 was released. I ended up running Ubuntu server on the nodes and configure them all using ansible playbooks (installing tailscale, k3s, updates, OS tweaks, etc.). I started looking at helm but there is so much inconsistency using community helm charts. I think writing my own would have been a better approach instead of templatizing my manifests and playbooks and doing it that way, however it is very easy to stand up a new service (assuming it's only 1-2 pods). If I end up DRY'ing my deployments it could end up being not too bad as a distinct deployment method from IaC or helm.

How do you like Pulumi? It seems similar to AWS CDK...

[s3rg4fts]

If you're templatizing manifests, kubespray does this pretty well I think. At least for the basics, it's pretty helpful so far. But indeed, I'm looking into deploying more things with Helm if possible.

Most services I've been using so far offer official Helm charts. But I get your point, it can be cumbersome and if there isn't an official one, then they can be pretty undocumented / hard to work around.

I haven't used CDK, but the concept is definitely similar. I think Pulumi most likely has wider support, since it's based on Terraform and even if you don't have a provider available on Pulumi you can "port it" (although never tried it, not sure if it works well). I like how it stores the state for you and secrets as well, saves quite a bit of trouble.