To everyone saying that Kubernetes is unnecessary, try implementing autoscaling, service discovery, secrets management, and autohealing in a vendor independent way without it.
Of course none of that is necessary for a self hosted home lab, but neither is gitops.
This is a very nice example of how to set stuff up properly.
OP, I would love to see Robusta (https://robusta.dev) as part of this too. It's definitely in line with your vision of automating everything, as it let's you automate the response to alerts and other events in your cluster. (Disclaimer: I'm one of the maintainers)
Oh. I’ve actually done that. It’s not as hard or complicated as you claim.
This was before kubernetes, arguably learning kubernetes is harder than implementing auto discovery and auto healing because you have to learn a lot about cluster semantics, the various network providers, service accounts, role based access control, integrations with secrets manager or the very disparate implementations of custom operators which may operate in your environment.
Kubernetes is a platform, I don’t doubt it solves a lot of problems. But it’s a complicated wee beastie, and I say that as a person who did it the easy way and used GKE on Google Cloud
Have you considered something like Tailscale so you can securely access it from outside your home? I've been thinking about spinning up my own home server that way, seeing as Tailscale makes it easy to securely access it from my phone when I'm out and about.
Exposing Home Assistant over it allows me to do things like turn my lights off when I'm away from home, without exposing stuff to the public internet.
https://www.blog.akhil.cc/shelly-dimmer
It's not quite the same as a VPN, but in addition to Tailscale and some of the alternatives suggested in other comments, it may be worth looking at using HTTPS client certificates.
Done right, you get a pretty simple second line of defence - you can validate the client has an appropriate certificate (running your own CA is pretty straightforward for small-scale home use or for a small group of users). Without such a certificate, users can't access the web service.
If your goal around a VPN is a second line of defence against the application's own authentication logic failing, client certificate authentication might be worth a look. If your threat model needs to cover a major issue in your web server, you might still want to stick with a VPN-based setup.
(You can of course do both, and bind services to an internal-only IP that you can only reach via a VPN, then have certificate auth on that too if you so desire)
I just stumbled on a self hosted project similar to tailscale: https://github.com/gravitl/netmaker. There is also https://github.com/juanfont/headscale which is an open sourced implementation of the Tailscale coordination server. Tailscale "just works" and makes things incredibly easy for personal use, but it's awesome that there are also similar self-hosted options out there
I'm using Tailscale on my home network. Right now my two most interesting use cases are 1) I have it installed on my Home Assistant VM so I can access it wherever I am without having to futz with internal vs external URLs, and 2) I have a few Raspberry Pi's scattered around the house running zwavejs2mqtt which doesn't have any authentication built in (i.e. anyone on my wifi could theoretically mess with my zwave network). I told it to listen on each Pi's Tailscale address and use ACLs to lock it down so just Home Assistant can talk to the websocket API.
Isn't headscale a clearly better option, since it removes the need to trust and depend on external sources? It the same software essentially, but if we're talking about self hosting, headscale is just inherently better, since it actually is self hosted.
I found the Tailscale Android app to take quite a big hit on my battery when used as an always-on VPN, so I use cloudflared on a VPS to create a Google authenticated tunnel to home assistant (over Tailscale).
I enjoy reading about homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. This is just another "how I installed kubernetes" thing only without any real scale behind it. There are other ways to deploy and run things, and it takes people toying around with alternatives for something unique to spring up.
My "homelab" is just debian stable, with a moratorium against containerization. Systemd keeps the services going. Updates are manual. It's a bit incredible how clean and easy such a set-up can be. Like there's almost nothing to say. It's extremely reliable and just keeps ticking.
I actually did use to have a bunch of virtualization and kubernetes and crap, but got rid of it because it ate literally half the systems' resources, in a death by a thousand containers-type of way. There was also a lot of just jank, stuff was constantly breaking and I was always trying to piece together why from half a dozen out-of-date tutorials. Felt like herding cats.
I've used everything under the sun and for homelab use IMO nothing beats using docker-compose and salt.
1. Unless you have crazy dynamic scaling needs, there's literally no point in using k8s or nomad
2. Unless you want RBAC and fine tuned ACL, theres literally no point in using k8s or Nomad
I hate this idea that you need a Ceph cluster and a 3 node HA cluster to have a home lab.
Not saying don't do it because you want to play/test/learn any of those platforms, just that if you are going to run a server service maybe consider if you are doing it for learning a task orchestration service or just want to run XYZ reliably.
Personally I don’t use Kubernetes for the scalability, but because I haven’t seen anything else remotely maintainable (and yes, Kubernetes still has lots of room for improvement).
I don’t want to build a good chunk of Kubernetes myself just to host my software. It’s nice to be able to Google for solutions to my problems or to ask on various forums/chat rooms (or in many cases, not have the problems in the first place because someone else committed the fixes to Kubernetes long ago).
> homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like.
I've had great success with Ansible+Proxmox. The proxmox CLI interface is good enough that you can basically do DIY terraform like setup straight against CLI without any agents / plugins etc. (I know declarative vs imperative but hopefully the comparison makes sense)
Lately I've been toying with local gitlab-CI building and deploying images directly into the environment so that any changes to the code redeploy it (to prod...this is home after all haha)
Considered terraform too but not a big fan of their mystery 3rd party plugin provider system
Yeah I’m with you. I run a rack at home with servers running a few applications orchestrated with a bit of Puppet, it’s 100x more reliable than K8s, I’m not having to do weird hacks to make persistent storage work, edit dozens of YAML files and run a whole host of dodgy control plane services.
Sadly, I leaned this the hard way by setting up a 4-node k8s cluster and realizing bare-metal support is abysmal and the maintainers simply don’t care about you if you’re not running a cluster on AWS or GCP. Unfortunately my day job still involves k8s and people give me strange looks when I suggest metal so substantially less complex, cheaper, more performant, and easier to orchestrate and maintain.
I'm currently building out my single-node "homelab" with coreos + podman + systemd. Most of the config is pulled from 1 ignition file during coreos installation. Its been a learning experience and but also very lightweight.
My setup I just have ansible deploy to a couple of hosts running Arch. Generally I just use OS packages and have ansible deploy appropriate configs then for some software where that's more hassle I use podman rootless containers. Such software includes e.g. Graylog, which wants particular versions of Elasticsearch and mongo, or postgres which requires a manual data migration for annual version upgrades
k8s and nomad are on my "maybe eventually later" list though.
This is a very good example of how you can save yourself the mental hurdle of remembering how to configure something etc.
I can only suspect how much time, trial and error this must've taken. This is my main issue with IaC. The concept really lends itself to any kind of modern infra, however I'm really put off by the sheer amount of time it takes me to whip out a bunch of Ansible playbooks and helper scripts, and on top of that make sure that all of them are idempotent.
Maybe I'm doing something wrong and this should be easy?
Pretty sure you're not doing anything wrong - well, if you are then I am too :-)
What's often overlooked (I believe) is that when you're doing this work in your day job you've for existing infra to support your infra along with other folks you can rely on to for help.
With home infra, you first must invent the universe (apologies to Carl). Having built 3 variations of home infra (Swarm on x64/Beelink, K3s on Pi, K3s on x64/Odroid) I've gained a strong admiration for anyone who take this on irregardless of there relative success.
What I've learnt over time is to add as little accidental complexity as is possible which I think is what you're getting at. One incarnation of the Pi K3s was provisioned by Ansible (on it's own stand alone Pi that the cluster would netboot from). Was Ansible better that imaging the usb drives manually and running a small script on each of the nodes? - probably a wash. I did however learn a bit of Ansible.
Today's stack is way too complex, and fragile. The entire stack relies on the goodwill of other projects to continue to be maintained/not have breaking changes. Even with the initial setup cost s(h)aved there will be a continuous maintenance burden. What you get in abstraction and automation you pay back in babysitting the additional connecting interfaces.
Projects are making this easier. For example moving from large mono config files to using conf.d/ directories where you can drop in extra files and manage each one independently is great for IaC.
While it's challenging up front I do enjoy being able to freshly install the latest version of Fedora, run my playbook and be more or less up and running.
It feels cleaner and more reliable (at least until this week when a broken selinux policy made it to stable) rather then trying to upgrade packages across major release versions in place.
<hat material="tinfoil"> If I've somehow acquired secret hidden malware or someone has opened up something to come back in later that's also going to get flushed out at least once every six months.
Awesome! hajimari is neat, haven't seen that before. I'm building a startup that's trying to put a good chunk of this in a box for folks to buy and plug-n-play self-hosting (as much as that is possible). Since you're running k3s already, you might checkout our templates at https://kubesail.com/templates for more home-hosting apps :)
This is downvoted but I think it is true, self hosted services don't get nearly as much love as they need. It takes a huge amount of effort, monitoring, and knowledge on the specific tools to be able to host them securely and detect intrusions. I attempted self hosting for years but decided it ultimately wasn't worth it and that I have very little ability to prevent against attacks or detect that they had happened.
Maybe they want to practice. I installed proxmox and I'm pretty much set up, but my needs are pretty huble, just a couple of VMs and maybe containers in the near future.
Pretty sweet. I have been called a devops person by others around me (and I am hesitant to proudly identify as one) because this homelab is more impressive and modular than a lot of stuff I have seen colleagues and I put together for professional stuff.
Well done. I was not aware of the Cloudflare solution. Is this something someone can use, _with_ their Cloudflare Access offering, for personal dev/lab envs without breaking the bank?
Cloudflare Tunnels (formerly Argo Tunnels) are a free complement to Cloudflare Access. It's an easy way to expose internal servers to the public internet, which you can then lock down with Cloudflare Access.
Very cool, this is like a perfect little kubernetes development environment. The combo of tekton + gitea + k8s is really nice and just a shame it takes bodging all the lego pieces together manually right now. I wish there were a one click exe install like Docker for Desktop that gave people an entire environment like this on their desktop. Like a little private github but so much more powerful.
If you're looking for some nice stuff to develop on an environment like this, check out VS Code and Google's cloud code addon: https://marketplace.visualstudio.com/items?itemName=GoogleCl... It's not totally tied to their GCE cloud offerings and is actually just a VS Code integration of their Skaffold tool (a fantastic inner dev loop tool for k8s). It works great with minikube or any other k8s cluster, presumably this one too. You can very quickly get up and running with code in your IDE now running and debugging on your k8s environment.
Although it's still incomplete and not one click yet, that's the direction I'm heading in: anyone can try my homelab on their PC, and if they want to use it on their real hardware, they can continue with the full install.
For people who self host with like setups or even simpler - how you guarantee network access? where I live internet might be shaky at times, and for that I can't access my home setup while away. How do you deal with that?
Regarding my server connection: I can't help you there, my ISP has only had one incident of downtime I've noticed in the seven years of being with them.
Regarding my client connection while away from home: If my client doesn't have a connection, cloud services don't work any better. But a lot of apps (e.g. bitwarden, some jellyfin clients, etc) are smart enough to work offline to some extent via operating on cashed resources.
I recently wanted to give my current RPi4 home server a GitOps makeover, migrating services (Home Assistant, Nextcloud and others) to k3s in the process.
What has been an obstacle is the availability of officially maintained Docker images for some of the components I've been wanting to use - afaict neither Argo CD nor Rook have official armv7/aarch64 images (though it seems Argo will release one soon).
Until then, I'll hold off on that pet project until I get my hands on a reasonably priced x86 SFF PC (the ThinkCentre M700 Tiny from TFA looks interesting!).
FWIW, FluxCD has full armv7 and aarch64 support (it definitely works on a Raspberry Pi 4, the earlier models with 1GB of RAM too, although depending on the size of your git repos that may not be enough memory, 4GB-8GB certainly is for Flux.)
How many Rpi4 servers are we talking? I have never run Rook before (though I ran Ceph, when it was called deis-store way back before it was cool) and I always remember that you needed at least 3-5 servers with at least one dedicated disk for OSD per each to make it work well, if at all!
I'm looking at synology-csi right now for my homelab, and wondering if there is hope that I can stop using local-path-storage this way, it looks a little bit outdated, so I'm afraid that may keep me from running the latest version of Kubernetes on it... unless I hack the support into it myself :gasp: surely that's gonna void the warranty
I always found the hurdle with self-hosting to be maintaining, not the initial setup. Things like upgrading, in order to keep getting security fixes, and verifying everything works after the upgrade, are what has taken the most effort and time in the past for me.
This looks like a great setup by the author, but difficult to maintain in the long run without significant time investment.
Docker and docker-compose make maintaining stuff dead simple to me. In like 95% of the cases it's just a matter of changing the version in a YAML file and running one command, and in the remaining 5% of the time I'm not concerned at all with some downtime because I'm (mostly) the only user.
True, it's not easy. In the past I used to run everything on Debian stable for that reason. And after every two years or so, I'd book two days for the upgrade, with some time offline.
The problem was, at some point Debian stable got so distant from modern infrastructure you had to patch it from the beginning with newer versions, so reluctantly I switched to testing (that was years ago). I was surprised to find out things were working just fine.
The problem is today everybody distributes software in their own way. Especially web software - native packages are more and more rare. So automatic updates that work are indeed a challenge.
The question I have whenever I see something like this is what the upgrade process looks like. How much time do you spend per month upgrading dependencies, how many mailing lists you're subscribed to, etc. When the log4j thing blew up, how long did it take you to gain confidence that you were no longer impacted?
Doesn't really matter, as long as it's fun. Some people play games, so people enjoy re-configuring their setup (I do enjoy it too). It's not a business / prod env setting.
Probably no worse than the alternatives. Just about everything is going to be easy to upgrade except possibly the Kubernetes masters (and even that isn’t so bad if you can spare a bit of downtime).
This needs a companion guide about how to set up the host machine. Which Linux distro to choose, how to set it up / harden it, nftables / firewall, public key login, etc.
Several people in this thread stating they are working on solutions to assist others learning modern cloud/devops stacks. I’ll throw my name in the hat too. I have a single node home lab running enterprise cloud/devops software stack. I have been working on a project to deploy on-prem cloud (OpenStack) in disconnected environment. I’ve had to learn how to build and host all of the infrastructure required to do this without the luxury of internet access. I realized the same thing can be simulated very cheap in a modern computer with enough cores and memory to run the stack as a single bare metal host. My build cost $1300 2 years ago but the knowledge gained has paid me back 5 figures worth of raises in my career in 16 months.
Having my own personal CloudBox has allowed me to experiment and fail fast. I am ahead in experience and knowledge than the rest of my team as a result. I have a tool they do not. I realize it would be better if we all had a tool like this.
So that’s the pitch, a single node “cloud in a box’, the CloudBox for IT students or professionals to learn any aspect of IT.
Now I just need lots more time to actually turn my prototype into a product.
You are unlikely to find what you're looking for. I'm going to dig into this article a bit, but what you're talking about is inherently hard, and delivers tremendous value to businesses. There are a lot of people doing it for money, and someone builds something that starts to actually work, they get acquired, or they take their project and make it enterprise-y (because that's where the funding is).
There's flatcar and k3os and fedora coreos and talos and lokomotive. There are maybe a dozen others as well, but those are the ones I know something about.
The real problem is that the orchestration of PXE boot, DHCP, name services, server registration, cluster bootstrapping, while simultaneously building a light distribution that makes the right calls on persistence, security, etc. is just *really hard*.
I took at a stab at it myself (failed startup) and have a custom distribution based on alpine, but the amount of work to go from there to everything is so large that it's tough to take on if you're small (and there is the constant desire to go enterprise because of the money)
(Repo owner here) I've already automated that with Ansible, including Linux installation (Rocky Linux). There's no hardening yet but that's on the roadmap.
I wonder how the fan noise is on those M700s. These little Lenovo machines are very pleasant. My only wish with my M93 is that the temperature would stay below 50’C and keep it 100% silent.
It's sad to see so many people that are dismissive of this setup, because they have some preconceived notions about Kubernetes and other tooling that's used. What has happened to curiosity? Of course nothing about this is "necessary", but the fun is in trial and error. In my experience running Kubernetes in my homelab and colo, every issue has been a learning opportunity, and was usually because of my own mistakes.
What do you use to provision Kubernetes persistent volumes on bare metal? I’m looking at open-ebs (https://openebs.io/).
Also, when you bump the image tag in a git commit for a given helm chart, how does that get deployed? Is it automatic, or do you manually run helm upgrade commands?
Now all we need is robots hooked into a Ci/CD pipeline that turn on computers and initiate the process.
And a few more years to automate computer production, let the computers find bugs & optimize computers, and we have a full circle of life.
This is cool, but are you doing this as learning experience? For a homelab it seems severely overkill. I understand it all being self-hosted, but it could be drastically simpler if you adopt a few SaaS products (using free-tier)
(Repo owner here) Yes, I'm mostly doing this as a learning exercise; there's still a lot of work to be done before I can rely on it to host my services.
> it could be drastically simpler if you adopt a few SaaS products (using free-tier)
For some people (including me), the risk involved in any SaaS product suddenly either dropping or imposing unworkable restrictions on free tier is high enough to make the extra work involved in self-hosting worth it. (Granted, my current self-hosting setup is a lot simpler than the one described in this article, but even if mine were more complex I would still say the same thing.)
I've been burned too mamy times. Once a product realizes that x% of its free tier uses only feature X they suddenly put that feature behind a paywall. Now I"ve got to research and setup an alternative or pay 20$ for some minor convenice I set it up for.
Most free tiers requires a card on file. I shudder to think at having to check dozens or hundreds of such services regularly to make sure they have not sent out an email about how we are now charging you. Then I don't notice for 6 months and find out I spend $150 on something I could have setup myself in less than an hour that will never charge me.
I have had hundreds of such products subscribed at work. At least every year or so an entire day gets burnt updating credit card info from logging in and looking up the bespoke way every single site requires the updates to be done for payments.
The services that suddenly shut their doors and again require me to research and setup an alternative.
Services get acquired all the time then switch to pay only and they have your CC on file already for easy billing.
Bravo, nice work! I am certainly going to use this as a model for my personal setup. Right now I have a few different hosts running different hypervisors, but would like to consolidate on kube and an iaas tool.
I've been eyeing off OpenStack for sometime for a similar use case, but I must admit it is an uninformed opinion. What am I missing between the two solutions (OpenStack vs this specific custom solution)?
Overkill probably, but useful practise for when working on something of a scale where it will make a significant difference. And they might just like playing with these things. A home lab does not have to be DayJob practical!
Talking about overkill, he also use 4 physical machines each with a 4 core CPU. But its not about what you need, its about what you want to learn. So if you want to learn about cluster/orchestrating i think its warranted.
Some people just like to tinker and try other things even if it's overkill. I have a k8s cluster on DO to host shitty side projects, it's pretty nice to just `kubectl apply -f manifest.yml` and be done with it, especially since I'll probably wind up tearing everything I just set up anyway.
I fully agree that setting up Kubernetes is a lot of work, but I think people forget how much work it is to set up anything maintainable for more than a few services running on a handful of services.
The nice thing about Kubernetes is that it’s probably not much harder (if at all) than the alternative (assuming you already know which tools to use to compose your bespoke platform) and you can easily find online resources and support (never mind all of the problems you don’t run into because they’re already solved in Kubernetes).
I would rather choose something like xgp-ng as a hypervisor for a home lab environment such a this, instead of a custom one-off kubernetes environment.
Or just a total DIY xen or KVM hypervisor on debian or centos approach. But with considerably more RAM in the host machine than 16GB (like 64-128GB).
the kubernetes sort of makes sense if the purpose is also for the person who owns/operates it to test kubernetes things, learn kubernetes, or use it as a home lab of some larger similar environment that they have elsewhere.
This is definitely cool, but it also highlights a huge problem we have with software nowadays.
There are 19 stacks in this repository. 19 pieces of software that require their own maintenance, JUST TO RUN YOUR APPLICATIONS! The amount of extra work required just to host the software that views your pictures, plays your videos, and allows chat with people is absolutely insane.
I'm not talking about this particular repo; it's just indicative of how complicated things have become that you must do the equivalent of building and maintaining your own operating system just to get even simple things done. And I belive that it's unnecessarily complicated (once again, not this repo, but rather the state of affairs in general). We're at an inflection point in the industry, and haven't come out the other side yet.
I could post my fully automated self hosting infrastructure, but it would be boring. Few bash ssh scripts and LEPP stack. None of this complexity and yet 10+ years uptime. Interesting nonetheless but think i'll pass on this. I have done these setups at cloudproviders for clients; that is definitely the risk averse version; if something goes wrong every month with AWS, it's just 'ok'; if something goes wrong every 5 years with your non-cloud, it's panic & heads need to roll (even though it costs a fraction).
It's not just running an app. He's building a setup that supports FANG-level infrastructure with autoscaling, self healing, etc.
Maybe it's unnecessary for tiny companies, but if you're dealing with infrastructure at scale then the complexity is unavoidable. The only question is whether you want to invent it in house or use a robust system that someone else built properly.
> There are 19 stacks in this repository. 19 pieces of software that require their own maintenance, JUST TO RUN YOUR APPLICATIONS!
There's tools like Yunohost [0] that can save a lot of work and are focused on making the subsequent maintenance burden easier too. There's an application catalogue [1] with a lot of popular self-hosted apps to choose from and new ones being packaged all the time. I haven't used it myself yet, but hear very good experiences from people that have. Especially those that lack the more sysadmin tech skills and just wanna click-install & go.
There's also Freedombox [2]. I'd be interested to know about other such tools and your experience with them.
Most of that is kubernetes, which is completely unnecessary for self hosting purposes. It's mostly for learning, or for the convenience of hosting multiple different services, trying many different containerized applications.
If your only goal is to serve Nextcloud, Plex or whatever to your family, you can get away with much less than that.
I agree that the number of different technologies here is shocking, but you get a lot more value as a result than "just to run your applications". Most notably that includes CI/CD, scaling, and redundancy, all with IaC. These things used to require a lot more administration and development effort.
Plugging together all these different tools has become so much work though that in many organizations the platform team(s) who are mainly occupied with doing just that take up a lot of engineering resources.
In my opinion the next evolutionary step would be for all of this to be bundled and abstracted away. Funnily enough, we pretty much have that product already with autoscaled "serverless" cloud services, GitHub Actions/Azure DevOps pipelines etc.
The biggest problem is probably that things like AWS Lambda, SQS, and DDB lack versatility and user friendliness. If we get some improvements on that front, many organizations might opt for that instead of dealing with their own K8s deployments.
Even better would be if we had something like a stripped-down version of OpenStack, just focussed on doing "serverless" right, and rebuilt from the ground up.
my main objection to this is choosing host machines that don't meet the criteria of anything like serious server-grade hardware.
if this is for a home lab where any one of the services run on it are not actually going to affect you if it goes belly up? or the whole host machine? sure, okay, but that's self hosting a home lab, not self-hosting actual infrastructure...
clearly the hardware shown in the image is meant to be small, not noisy, and not take up a lot of space, and operate in somebody's house.
but the people I know who seriously self host all their stuff have it on something like a few old Dell R620 1U servers with 1+1 dual hot swap power supplies, RAID-1 for the operating system boot, RAID-1 or RAID-6 (or something like mdadm raid6) for storage drives (again all hot swap capable), etc.
note that I am not talking about a lot of money here, you can buy the hardware I described in the preceding paragraph for $300 on eBay and then add your own choice of SSDs.
and not in a house, but in something like a small to medium sized ISP's colocation environment, with UPS, generator backed power, etc. also attached to network equipment and a DIA circuit that's considerably more serious than a tp-link unmanaged switch attached to a residential internet service.
All overkill. If you want something more reliable, get some late model HP or Dell thin clients and put Debian on them. Usually you can pick them up for <$75.
I setup something like this to control some farm equipment when I was in college 20 years ago. The farmers’ son called me a few months ago because they were having an issue with the equipment. Turned out oily dust coated the thing and got inside - I had him pop the device open (old Wyse box running Slackware iirc) clean up the goop and hit it with rubbing alcohol. He dried it and everything is working again.
khuedoan|4 years ago
I was surprised to find this on Hacker News, I wanted to wait until the stable release before posting on HN, but thank you for posting :)
This project is still in alpha stage, but please feel free to critique; I'd appreciate it.
Edit 1: After reading some of the comments, I want to clarify a few things:
- Because it is currently in the alpha stage, I do not host anything important on it.
- This is also my learning environment, I use Kubernetes in my day job.
- Yes it is overkill ;)
Edit 2: Wording
jmakov|4 years ago
hunta2097|4 years ago
Mine is based around VMs using K3OS but I have also been looking at thinkstations and other cheap Intel-T CPU workstations.
How do you find disk performance and reliability?
vimsee|4 years ago
sandGorgon|4 years ago
nicbou|4 years ago
nyellin|4 years ago
Of course none of that is necessary for a self hosted home lab, but neither is gitops.
This is a very nice example of how to set stuff up properly.
OP, I would love to see Robusta (https://robusta.dev) as part of this too. It's definitely in line with your vision of automating everything, as it let's you automate the response to alerts and other events in your cluster. (Disclaimer: I'm one of the maintainers)
dijit|4 years ago
This was before kubernetes, arguably learning kubernetes is harder than implementing auto discovery and auto healing because you have to learn a lot about cluster semantics, the various network providers, service accounts, role based access control, integrations with secrets manager or the very disparate implementations of custom operators which may operate in your environment.
Kubernetes is a platform, I don’t doubt it solves a lot of problems. But it’s a complicated wee beastie, and I say that as a person who did it the easy way and used GKE on Google Cloud
simonw|4 years ago
Have you considered something like Tailscale so you can securely access it from outside your home? I've been thinking about spinning up my own home server that way, seeing as Tailscale makes it easy to securely access it from my phone when I'm out and about.
smasher164|4 years ago
Exposing Home Assistant over it allows me to do things like turn my lights off when I'm away from home, without exposing stuff to the public internet. https://www.blog.akhil.cc/shelly-dimmer
g_p|4 years ago
Done right, you get a pretty simple second line of defence - you can validate the client has an appropriate certificate (running your own CA is pretty straightforward for small-scale home use or for a small group of users). Without such a certificate, users can't access the web service.
If your goal around a VPN is a second line of defence against the application's own authentication logic failing, client certificate authentication might be worth a look. If your threat model needs to cover a major issue in your web server, you might still want to stick with a VPN-based setup.
(You can of course do both, and bind services to an internal-only IP that you can only reach via a VPN, then have certificate auth on that too if you so desire)
mcescalante|4 years ago
zrail|4 years ago
chaxor|4 years ago
omnicognate|4 years ago
kinduff|4 years ago
I really like the service, works like a charm. Sometimes I use one of my servers as an exit node when I'm out as a VPN.
mlangenberg|4 years ago
rhizome|4 years ago
Orou|4 years ago
candiddevmike|4 years ago
marginalia_nu|4 years ago
I actually did use to have a bunch of virtualization and kubernetes and crap, but got rid of it because it ate literally half the systems' resources, in a death by a thousand containers-type of way. There was also a lot of just jank, stuff was constantly breaking and I was always trying to piece together why from half a dozen out-of-date tutorials. Felt like herding cats.
nullwarp|4 years ago
1. Unless you have crazy dynamic scaling needs, there's literally no point in using k8s or nomad 2. Unless you want RBAC and fine tuned ACL, theres literally no point in using k8s or Nomad
I hate this idea that you need a Ceph cluster and a 3 node HA cluster to have a home lab.
Not saying don't do it because you want to play/test/learn any of those platforms, just that if you are going to run a server service maybe consider if you are doing it for learning a task orchestration service or just want to run XYZ reliably.
throwaway894345|4 years ago
I don’t want to build a good chunk of Kubernetes myself just to host my software. It’s nice to be able to Google for solutions to my problems or to ask on various forums/chat rooms (or in many cases, not have the problems in the first place because someone else committed the fixes to Kubernetes long ago).
Axsuul|4 years ago
Havoc|4 years ago
I've had great success with Ansible+Proxmox. The proxmox CLI interface is good enough that you can basically do DIY terraform like setup straight against CLI without any agents / plugins etc. (I know declarative vs imperative but hopefully the comparison makes sense)
Lately I've been toying with local gitlab-CI building and deploying images directly into the environment so that any changes to the code redeploy it (to prod...this is home after all haha)
Considered terraform too but not a big fan of their mystery 3rd party plugin provider system
oceanplexian|4 years ago
Sadly, I leaned this the hard way by setting up a 4-node k8s cluster and realizing bare-metal support is abysmal and the maintainers simply don’t care about you if you’re not running a cluster on AWS or GCP. Unfortunately my day job still involves k8s and people give me strange looks when I suggest metal so substantially less complex, cheaper, more performant, and easier to orchestrate and maintain.
discardedrefuse|4 years ago
Macha|4 years ago
k8s and nomad are on my "maybe eventually later" list though.
extinctpotato|4 years ago
I can only suspect how much time, trial and error this must've taken. This is my main issue with IaC. The concept really lends itself to any kind of modern infra, however I'm really put off by the sheer amount of time it takes me to whip out a bunch of Ansible playbooks and helper scripts, and on top of that make sure that all of them are idempotent.
Maybe I'm doing something wrong and this should be easy?
earleybird|4 years ago
What's often overlooked (I believe) is that when you're doing this work in your day job you've for existing infra to support your infra along with other folks you can rely on to for help.
With home infra, you first must invent the universe (apologies to Carl). Having built 3 variations of home infra (Swarm on x64/Beelink, K3s on Pi, K3s on x64/Odroid) I've gained a strong admiration for anyone who take this on irregardless of there relative success.
What I've learnt over time is to add as little accidental complexity as is possible which I think is what you're getting at. One incarnation of the Pi K3s was provisioned by Ansible (on it's own stand alone Pi that the cluster would netboot from). Was Ansible better that imaging the usb drives manually and running a small script on each of the nodes? - probably a wash. I did however learn a bit of Ansible.
pishpash|4 years ago
artdigital|4 years ago
Especially cloud setups that just run containers are relatively easy to get idempotent with terraform
khuedoan|4 years ago
cube00|4 years ago
While it's challenging up front I do enjoy being able to freshly install the latest version of Fedora, run my playbook and be more or less up and running.
It feels cleaner and more reliable (at least until this week when a broken selinux policy made it to stable) rather then trying to upgrade packages across major release versions in place.
<hat material="tinfoil"> If I've somehow acquired secret hidden malware or someone has opened up something to come back in later that's also going to get flushed out at least once every six months.
erulabs|4 years ago
khuedoan|4 years ago
Also I believe the tag filter is malfunctioning: https://kubesail.com/templates-by-tag/Media
goodpoint|4 years ago
And you cannot do when you deploy tools worth millions of lines of code.
Complexity matters. Those popular products make sense only if you have a 20 engineers is your team or you don't care about reliability.
Gigachad|4 years ago
spaniard89277|4 years ago
616c|4 years ago
Well done. I was not aware of the Cloudflare solution. Is this something someone can use, _with_ their Cloudflare Access offering, for personal dev/lab envs without breaking the bank?
LilBytes|4 years ago
I'm using it for personal use at the moment and I'm considering changing to a paid user to my friends and family can access Emby over the internet.
I did have a few dramas getting in to work for my LxC environments but nothing a quick Google resolved for me.
altano|4 years ago
qbasic_forever|4 years ago
If you're looking for some nice stuff to develop on an environment like this, check out VS Code and Google's cloud code addon: https://marketplace.visualstudio.com/items?itemName=GoogleCl... It's not totally tied to their GCE cloud offerings and is actually just a VS Code integration of their Skaffold tool (a fantastic inner dev loop tool for k8s). It works great with minikube or any other k8s cluster, presumably this one too. You can very quickly get up and running with code in your IDE now running and debugging on your k8s environment.
khuedoan|4 years ago
Although it's still incomplete and not one click yet, that's the direction I'm heading in: anyone can try my homelab on their PC, and if they want to use it on their real hardware, they can continue with the full install.
surfsvammel|4 years ago
I have done some home lab going on over the years, and find that trying to do more on less W has been the most fun.
khuedoan|4 years ago
comprev|4 years ago
For a single server deployment docker-compose is very useful.
JamesAdir|4 years ago
Macha|4 years ago
Regarding my client connection while away from home: If my client doesn't have a connection, cloud services don't work any better. But a lot of apps (e.g. bitwarden, some jellyfin clients, etc) are smart enough to work offline to some extent via operating on cashed resources.
killingtime74|4 years ago
AdrianoKF|4 years ago
What has been an obstacle is the availability of officially maintained Docker images for some of the components I've been wanting to use - afaict neither Argo CD nor Rook have official armv7/aarch64 images (though it seems Argo will release one soon).
Until then, I'll hold off on that pet project until I get my hands on a reasonably priced x86 SFF PC (the ThinkCentre M700 Tiny from TFA looks interesting!).
yebyen|4 years ago
How many Rpi4 servers are we talking? I have never run Rook before (though I ran Ceph, when it was called deis-store way back before it was cool) and I always remember that you needed at least 3-5 servers with at least one dedicated disk for OSD per each to make it work well, if at all!
I'm looking at synology-csi right now for my homelab, and wondering if there is hope that I can stop using local-path-storage this way, it looks a little bit outdated, so I'm afraid that may keep me from running the latest version of Kubernetes on it... unless I hack the support into it myself :gasp: surely that's gonna void the warranty
3np|4 years ago
Running you own registry is super simple.
https://docs.docker.com/registry/deploying/
unknown|4 years ago
[deleted]
cyfex|4 years ago
This looks like a great setup by the author, but difficult to maintain in the long run without significant time investment.
input_sh|4 years ago
hdjjhhvvhga|4 years ago
The problem was, at some point Debian stable got so distant from modern infrastructure you had to patch it from the beginning with newer versions, so reluctantly I switched to testing (that was years ago). I was surprised to find out things were working just fine.
The problem is today everybody distributes software in their own way. Especially web software - native packages are more and more rare. So automatic updates that work are indeed a challenge.
k8sToGo|4 years ago
akkartik|4 years ago
zenlot|4 years ago
khuedoan|4 years ago
throwaway894345|4 years ago
mikesabbagh|4 years ago
All this work is nice and beautiful, the problem will come when you try to update different components.
christophilus|4 years ago
It amazes me how much of the internet doesn’t run on a single node, when it probably could.
gigel82|4 years ago
If anyone has one handy, I'd appreciate a link.
Art9681|4 years ago
Having my own personal CloudBox has allowed me to experiment and fail fast. I am ahead in experience and knowledge than the rest of my team as a result. I have a tool they do not. I realize it would be better if we all had a tool like this.
So that’s the pitch, a single node “cloud in a box’, the CloudBox for IT students or professionals to learn any aspect of IT.
Now I just need lots more time to actually turn my prototype into a product.
dmayle|4 years ago
There's flatcar and k3os and fedora coreos and talos and lokomotive. There are maybe a dozen others as well, but those are the ones I know something about.
The real problem is that the orchestration of PXE boot, DHCP, name services, server registration, cluster bootstrapping, while simultaneously building a light distribution that makes the right calls on persistence, security, etc. is just *really hard*.
I took at a stab at it myself (failed startup) and have a custom distribution based on alpine, but the amount of work to go from there to everything is so large that it's tough to take on if you're small (and there is the constant desire to go enterprise because of the money)
indigodaddy|4 years ago
khuedoan|4 years ago
gorgoiler|4 years ago
ianunruh|4 years ago
unknown|4 years ago
[deleted]
nodesocket|4 years ago
Also, when you bump the image tag in a git commit for a given helm chart, how does that get deployed? Is it automatic, or do you manually run helm upgrade commands?
khuedoan|4 years ago
When you make a change in git, it is automatically deployed without the need for human intervention.
endofreach|4 years ago
Good first version, i am excited for the beta!
jaimex2|4 years ago
midasuni|4 years ago
rambambram|4 years ago
turtlebits|4 years ago
b33j0r|4 years ago
I knew I’d want to change the LAN architecture and the services, and everything; but it was definitely intended as a learning experience for me.
Kinda looking forward to doing it over again with my new/refreshed skillset. Automated everything.
khuedoan|4 years ago
> For a homelab it seems severely overkill
Isn't that the point of homelab? ;)
pdonis|4 years ago
For some people (including me), the risk involved in any SaaS product suddenly either dropping or imposing unworkable restrictions on free tier is high enough to make the extra work involved in self-hosting worth it. (Granted, my current self-hosting setup is a lot simpler than the one described in this article, but even if mine were more complex I would still say the same thing.)
yosito|4 years ago
citizenpaul|4 years ago
Most free tiers requires a card on file. I shudder to think at having to check dozens or hundreds of such services regularly to make sure they have not sent out an email about how we are now charging you. Then I don't notice for 6 months and find out I spend $150 on something I could have setup myself in less than an hour that will never charge me.
I have had hundreds of such products subscribed at work. At least every year or so an entire day gets burnt updating credit card info from logging in and looking up the bespoke way every single site requires the updates to be done for payments.
The services that suddenly shut their doors and again require me to research and setup an alternative.
Services get acquired all the time then switch to pay only and they have your CC on file already for easy billing.
whalesalad|4 years ago
mrslave|4 years ago
Lamad123|4 years ago
unixhero|4 years ago
divbzero|4 years ago
throwaway894345|4 years ago
quocanh|4 years ago
Do you experiment with your tech stack? Swapping things in and out to see which apps are best?
khuedoan|4 years ago
I think the OP discovered my homelab through that post.
iamgopal|4 years ago
ekianjo|4 years ago
dspillett|4 years ago
z3t4|4 years ago
status_quo69|4 years ago
throwaway894345|4 years ago
The nice thing about Kubernetes is that it’s probably not much harder (if at all) than the alternative (assuming you already know which tools to use to compose your bespoke platform) and you can easily find online resources and support (never mind all of the problems you don’t run into because they’re already solved in Kubernetes).
walrus01|4 years ago
Or just a total DIY xen or KVM hypervisor on debian or centos approach. But with considerably more RAM in the host machine than 16GB (like 64-128GB).
the kubernetes sort of makes sense if the purpose is also for the person who owns/operates it to test kubernetes things, learn kubernetes, or use it as a home lab of some larger similar environment that they have elsewhere.
the_duke|4 years ago
But Kubernetes exists for a reason and makes a lot of things easier.
If you are familiar with the ecosystem, k3s is a great foundation for self-hosted setups.
Especially thanks to the many helm charts and operators that exist.
josephd79|4 years ago
kstenerud|4 years ago
There are 19 stacks in this repository. 19 pieces of software that require their own maintenance, JUST TO RUN YOUR APPLICATIONS! The amount of extra work required just to host the software that views your pictures, plays your videos, and allows chat with people is absolutely insane.
I'm not talking about this particular repo; it's just indicative of how complicated things have become that you must do the equivalent of building and maintaining your own operating system just to get even simple things done. And I belive that it's unnecessarily complicated (once again, not this repo, but rather the state of affairs in general). We're at an inflection point in the industry, and haven't come out the other side yet.
tluyben2|4 years ago
nyellin|4 years ago
Maybe it's unnecessary for tiny companies, but if you're dealing with infrastructure at scale then the complexity is unavoidable. The only question is whether you want to invent it in house or use a robust system that someone else built properly.
I elaborated more in my comment below
dgellow|4 years ago
Systemd units and a bit of sh is enough if you just want your applications.
rapnie|4 years ago
There's tools like Yunohost [0] that can save a lot of work and are focused on making the subsequent maintenance burden easier too. There's an application catalogue [1] with a lot of popular self-hosted apps to choose from and new ones being packaged all the time. I haven't used it myself yet, but hear very good experiences from people that have. Especially those that lack the more sysadmin tech skills and just wanna click-install & go.
There's also Freedombox [2]. I'd be interested to know about other such tools and your experience with them.
[0] https://yunohost.org
[1] https://yunohost.org/en/apps
[2] https://freedombox.org/
legends2k|4 years ago
INTPenis|4 years ago
If your only goal is to serve Nextcloud, Plex or whatever to your family, you can get away with much less than that.
tiarafawn|4 years ago
Plugging together all these different tools has become so much work though that in many organizations the platform team(s) who are mainly occupied with doing just that take up a lot of engineering resources.
In my opinion the next evolutionary step would be for all of this to be bundled and abstracted away. Funnily enough, we pretty much have that product already with autoscaled "serverless" cloud services, GitHub Actions/Azure DevOps pipelines etc.
The biggest problem is probably that things like AWS Lambda, SQS, and DDB lack versatility and user friendliness. If we get some improvements on that front, many organizations might opt for that instead of dealing with their own K8s deployments. Even better would be if we had something like a stripped-down version of OpenStack, just focussed on doing "serverless" right, and rebuilt from the ground up.
unknown|4 years ago
[deleted]
walrus01|4 years ago
if this is for a home lab where any one of the services run on it are not actually going to affect you if it goes belly up? or the whole host machine? sure, okay, but that's self hosting a home lab, not self-hosting actual infrastructure...
clearly the hardware shown in the image is meant to be small, not noisy, and not take up a lot of space, and operate in somebody's house.
but the people I know who seriously self host all their stuff have it on something like a few old Dell R620 1U servers with 1+1 dual hot swap power supplies, RAID-1 for the operating system boot, RAID-1 or RAID-6 (or something like mdadm raid6) for storage drives (again all hot swap capable), etc.
note that I am not talking about a lot of money here, you can buy the hardware I described in the preceding paragraph for $300 on eBay and then add your own choice of SSDs.
and not in a house, but in something like a small to medium sized ISP's colocation environment, with UPS, generator backed power, etc. also attached to network equipment and a DIA circuit that's considerably more serious than a tp-link unmanaged switch attached to a residential internet service.
Spooky23|4 years ago
I setup something like this to control some farm equipment when I was in college 20 years ago. The farmers’ son called me a few months ago because they were having an issue with the equipment. Turned out oily dust coated the thing and got inside - I had him pop the device open (old Wyse box running Slackware iirc) clean up the goop and hit it with rubbing alcohol. He dried it and everything is working again.