top | item 24474309

Raspberry Pi as a local server for self hosting applications

356 points| christian_fei | 5 years ago |cri.dev

314 comments

order
[+] anderspitman|5 years ago|reply
The hardware is there (RPi + USB storage). The server software is there (NextCloud, Plex, n8n, etc). What isn't there is the plumbing. The next logical step after this blog post is making your services accessible to your phone over the public net. You'll immediately find yourself mired in domain name registration, VPS management, TLS cert management, dyndns, port forwarding, hole punching, etc etc.

There are lots of great tools that solve some of these problems. I have yet to find one that solves all of them.

I think we need something like Namecheap + CloudFlare + ngrok, designed and marketed for self-hosters and federators. You simply register a domain and run a client tool on each of your machines that talks to a central server which tunnels HTTPS connections securely to the clients.

Mapping X subdomain to Y port on Z machine should take a couple clicks from a web interface.

[+] phaer|5 years ago|reply
> The next logical step after this blog post is making your services accessible to your phone over the public net. You'll immediately find yourself mired in domain name registration, VPS management, TLS cert management, dyndns, port forwarding, hole punching, etc etc.

You'll need don't need any of that with Onion Services. Tor does not only anonymize, but offers easily configurable services with NAT punching, an .onion Domain and e2e crypto for free. And setting them up is easy enough https://community.torproject.org/onion-services/setup/

You'll just need tor or a tor browser to access those services, but that shouldn't be a problem for many self-hosting setups

[+] cguess|5 years ago|reply
> I think we need something like Namecheap + CloudFlare + ngrok, designed and marketed for self-hosters and federators. You simply register a domain and run a client tool on each of your machines that talks to a central server which tunnels HTTPS connections securely to the clients.

PageKite (https://www.pagekite.net) is what it sounds like you're looking for. It'll set you up with a url, SSL, and a tunnel in about 30 seconds. Highly configurable for your own domain if you'd like, multiple ports etc.

[+] colechristensen|5 years ago|reply
With a couple of fiber providers I've been lucky enough (it wasn't luck, I chose housing based on ISP availability) to get a business class gigabit with a static ipv4 address and ipv6 for ~$100/mo, solves lots of problems.

Plex does the plumbing for you, I think NextCloud might too.

Doing DNS just seems like another thing to setup which is just fine. Pay namecheap, then pay a big boy DNS provider (I like dnsmadeeasy) then register some domains.

I don't really want a solves everything tool because I don't see a way for it not to be really opinionated and hide everything behind its own abstraction which isn't really any better than the interface it is hiding. Maybe a series of how-to whitepaper kinds of thing to build up the requisite knowledge to figure these things out.

I'm not a fan of the old school configuration hell where you have to spend hours/days/weeks trying to figure out the correct set of software and config options to do something right, but I'm equally not a fan of completely canned solutions that hide everything in favor of a single button to push. I'm not a technician, I don't need to have everything done for me, but I do appreciate tools where right configuration interface is provided. That is, sane defaults, well documented options, meaningful errors and sanity checking, and options given in the right way.

[+] angrygoat|5 years ago|reply
I've got IPv6 from my ISP - I gave my pi a static v6 IP, set up cloudflare and told it that v6 IP, and cloudflare handles making it available to the v4 internet. Has worked pretty well so far, just for tinkering stuff, nothing 'production'.
[+] z3t4|5 years ago|reply
Most home routers have functionality to do port forwarding and dyndns. Certbot for TLS/cert. Just get a free domain from the dyndns service. No VPS needed if you got your own hardware/Rasbery. Make sure you have backups. Just Dont expect a $30 PC to be without issues.
[+] chrios|5 years ago|reply
Use IPv6.

It doesn't have the address exhaustion that caused providers to implement CGNAT, and dynamic IPv4 addresses.

No need for VPS management, DynDNS, port forwarding, hold punching. You still need public DNS, but you can use public DNS as your internal zone as well (no need for split DNS). You also still need PKI, so maybe setup a reverse proxy for SSL termination with a wildcard certificate.

[+] xyzzy_plugh|5 years ago|reply
Or just setup Tailscale, which takes about two minutes.
[+] Polylactic_acid|5 years ago|reply
I tried running nextcloud on an rpi. It just doesn't cut it. I had the 4gb model and nextcloud runs but its a horrible experience. You go on the web UI and click a photo and it takes 10 seconds to load. Moved my server to a ryzen 5 based setup and now everything is instant. I'm not sure what the limiting factor on the rpi was because the ram and cpu usage was low. Perhaps it was memory or storage speed.
[+] jeroenhd|5 years ago|reply
There's a lot of solutions sibling comments have already brought up, but I don't know if it should be this automagical. Keeping services up to date requires effort, money, or a big reduction in freedom of what you can do with your server.

There's a full-automatic mail server program, maininabox, that tries to be this instant "just make it work" system. The result of the project is that the host OS was severely outdated for years because upgrading configuration automatically is difficult and because the system manages DNS for you, adding a new subdomein to your server is more of a challenge than it should be.

Similarly, automatic service install and management tools like Plesk, cPanel, ISPconfig have been around forever but they always provide some limitation. I think Sandatorm.IO is a quite recent tool of this sort that runs Docker so you have a bit more control.

All of these still require occasional maintenance though. If you can't figure out how to point a DNS name and a wildcard to your IP, then I'm not sure if you should be exposing services on the internet like that. If you don't update for a while your nice, powerful server Raspberry Pi might suddenly be DDOS'ing random websites without you even knowing about it, and all you can do to prevent that is to keep your (limited) software stack updated.

All attempts to make this easy for the general public have so far shown that people don't like to press the update button; even rebooting Windows is a risk some people just aren't willing to take, which is why Microsoft had to force reboots in Windows 10. With that kind of risk out there, freely connecting whatever to the web and forgetting about it, I'm glad there's some technical requirements before you can host something.

[+] gramakri|5 years ago|reply
https://cloudron.io can do most of this minus the hole punching . The port forward ing is very router specific. I think maybe there is some upnp interface for this but not sure how widely it is supported.
[+] johnjungles|5 years ago|reply
Check out KubeSail! Not affiliated in anyway. They make it super easy to do the plumbing, networking and have a kubernetes cluster on a raspberry pi.

If you ever wanted to learn k8s without spending $80\month on a cluster, best way to learn it!

[+] Aperocky|5 years ago|reply
> Mapping X subdomain to Y port on Z machine should take a couple clicks from a web interface.

route53 can work like that, it also has a cli version. (But you can't get the domain there).

[+] RealStickman_|5 years ago|reply
I'm currently thinking of using a reverse proxy through a wireguard tunnel. That should work also for non-static home ip addresses. (I already habe the domain and VPS)
[+] shifto|5 years ago|reply
> Mapping X subdomain to Y port on Z machine should take a couple clicks from a web interface.

This is already the case. Routers have mostly easy webinterfaces nowadays en the same goes for DNS options at any domainname provider. What people need is a bit of knowledge. It takes me a few mouse clicks in a webinterface to do this because I know what I am doing. Yeah you could dumb down anything to a single button but I don't think we should want that.

[+] d1plo1d|5 years ago|reply
I tunnel everything through webRTC. It's a bit exotic but it gets you a direct bidirectional data connection to the self hosted device. You can put all users' self hosted content through a single domain name & SSL cert or you could have subdomains automatically provisioned for each device.

I'm using this WebRTC method for 3D printers at https://tegapp.io

[+] homedrive|5 years ago|reply
HomeDrive ( https://www.homedrive.io ) is plumbing exactly this! We are currently only hosting Nextcloud, but we plan to support more apps and custom dockers. It is as easy as plugging the box into the home router.

There are still many features to implement, but we are working towards "easy self-hosting at home", and looking for early adopters.

[+] wwn_se|5 years ago|reply
Cloudflare -> Router (only allows 80 443 trafic from cloudflare ips) -> nginx -> all selfhosted services (wiki, hass...)

Problems with the "easy" one click ones is that they tend to not be very secure. If they are supposed to be public access. Plex uses their cloud to secure the access and Synology to

[+] bashallah|5 years ago|reply
I need this.

Bought a namecheap registry for a small nfp.

Been swamped with how-to’s and learning things just to learn what to search for...

I could use something else that does it all...

But I want a level of authority none of those offer.. without the technical insight of “is this everything/enough”.

[+] christian_fei|5 years ago|reply
I can relate, thought about setting up a Caddy server to route through the different services (also nginx would be fine). Have to try it out and probably make a list of services in a HTML document returned on port 80/443
[+] antognini|5 years ago|reply
One thing to watch out for when doing something like this is that the Raspberry Pi will by default put your file system on the SD card it boots from. SD cards aren't meant to support a lot of write/erase cycles, so it's easy to end up with a corrupt SD card after a few months to a year depending on what you're doing on your Pi.

A workaround that can save you some headaches here is to only boot from the SD card (which means you're effectively only ever reading from the card), and then mount a filesystem on an external SSD drive. There are a couple of good guides here [1] [2].

[1]: https://www.stewright.me/2019/10/run-raspbian-from-a-usb-or-...

[2]: https://www.pragmaticlinux.com/2020/08/move-the-raspberry-pi...

[+] gerdesj|5 years ago|reply
Very true for a given value of true 8) You need to evaluate all the components. The RPi itself is a decent piece of kit, well tested and safe to use. Get a decent power supply to it - either a RPi branded one or at least a decent mobile charger from a brand that you trust.

I generally use a decent USB stick nowadays. RPi 4 from about a month or so ago onwards will do this out of the box. You can also put a second USB stick in and clone the thing every now and then.

You can PXE boot them as well (citation needed) and that brings nfs and iSCSI to bear. That's my long term plan for fleets of them.

For the semi casual user, I recommend the dual USB stick combo. Quite easy to set up and you can always whip out the backup and test it on another device.

[+] rovr138|5 years ago|reply
Latest Pi’s can boot from external media. It’s possible to boot from an SSD
[+] ocdtrekkie|5 years ago|reply
I've been using a RasPi for a Pihole for years and this is a constant peeve. No matter how many precautions it takes, it eventually dives. RasPi is neat for a lot of things but I'm not convinced it's an ideal selfhosting platform. By the time you invest in the necessary addons, you might as well have gotten a used actual server.
[+] charwalker|5 years ago|reply
Newer Pi's do support booting to USB drives (flash or SATA adapter) and I think the firmware update enabling that on the 3/3b/4/4b allows SATA hats and others to work but I haven't read into or tested any of them.

Booting to USB protects the install better, allows easier access or setup on a PC, and I have more of them around to set up random OS or systems to test. I have several pis around and each that can boots to USB including the pihole systems at my parents place and grandma's apartment.

Plus, the SD copier tool in Raspbian works with USB drives (and VHDs) so I set up the pihole for my grandma, cloned the usb drive, then sent both drives to her with the pi and called to walk her through plugging things in then used my existing remote tool on her laptop to finish set up. Now she has a backup USB drive that may need updates but is ready to go if the existing USB fails with pihole ready and everything. Plus once I update the backup drive I can clone it to the corrupt USB and she can store that as backup.

I know many people have hard lines for supporting friends/family. I've taken it in stride with super useful setups for relatives (pihole and a remoting tool if I set up their laptop/desktop) and beer or similar cost for friends. One recently spilled warm garlic sauce in their laptop so I pulled it apart while they scrubbed and wiped and some local beer is cheaper than a new laptop any day.

[+] bscphil|5 years ago|reply
You can also netboot them, which adds a little latency but in terms of speed is likely even better than the SD card, now that the Pis have real Gbit.
[+] christian_fei|5 years ago|reply
Good points, thanks!

Just stumbled upon this today by coincidence, will definitely follow the suggestion, cheers

[+] gonehome|5 years ago|reply
How do people access these servers off of their home network (or do they not?).

That seems like most of the value to me, hosting some service you can access from anywhere without having to use Digital Ocean.

It seems like most residential ISPs don't provide a static IP and some block port 80? I think forcing ISPs to allow home users to serve traffic via some standard method would go a long way to enabling a more decentralized web.

I know Zero Tier, and Tailscale exist - but I don't really understand how they work (and I think they require intermediate server access anyway so might as well use Digital Ocean?).

I'd like a future where you could sell users a raspberry pi running a service they can just plug into their home switch and access it securely from anywhere.

[+] liability|5 years ago|reply
I recently thought about getting an RPi4 but ultimately spent about a hundred dollars more to get a cheap NUC. It was a bit more expensive but it seems like a more robust platform. A real AC adapter instead of USB (apparently the RPi4 kind of botched it in some revisions? What I read wasn't confidence inspiring), takes normal SO-DIMM ram and a typical SSD, doesn't have a reputation for overheating.. it seems generally more straight forward with fewer 'gotchas.'
[+] codezero|5 years ago|reply
I haven't been into computer hardware lately, but I decided to pick up a NAS, and was pleased to learn that they're now just a complete computer - I've started using VMs living on my NAS to do this kind of thing, which is quite nice. Synology's interface is not bad either, but I imagine others have come up with even better ways to use these systems.

Obviously a RPi is a way cheaper way to get a lot of the same work done though :)

[+] nicbou|5 years ago|reply
I'm using my old Thinkpad T510 as a home server. It's been running for 7+ years already. I only need to dust the vents once in a while.

It has PiHole, Nextcloud, my humble little Netflix clone, and a few other things. If you use ffmpeg a lot, you ought to have more power than the RPi offers. I often SSH into it to use it as a SOCKS proxy in other countries.

[+] _lacroix|5 years ago|reply
Setting up a pi-hole DNS server for my wifi network was one of the best decisions I've ever made. Horrifying to see what percentage of traffic is on the ad server blacklist though...
[+] flyinghamster|5 years ago|reply
It really wasn't until I got myself a Pi 4 that I really appreciated what an improvement it was over the earlier generations. I have the Pi 4 and a 3B+ running BOINC, crunching away on World Community Grid[1], and the 4 is at least twice as fast at completing work units (it's too early for RAC numbers to settle down, yet). The Cortex-A72 is a huge step up from the A53. [ETA: both are actively cooled.]

Also, the Pi 4 eliminates the USB2 bottleneck the old Pis have, and has a couple of USB3 ports.

[1] Be sure to boot with arm_64bit=1 in config.txt or you will get no work units.

[+] yegle|5 years ago|reply
For anyone with a home server and has the need to remotely access your self-hosted websites, https://pomerium.io has been a wonderful piece of software in my stack.

You can safely expose your self-hosted websites to the internet and without the hassle of needing to have a VPN connection first.

[+] oblib|5 years ago|reply
A DigitalOcean vps can be a pretty inexpensive and easy option.

I've done this with a Pi and Dyndns. It's pretty easy to setup but not as good (for me) as a DO vps because my home ISP limits data heading out. I would have to purchase a business plan to fix that and it still wouldn't be better or cheaper than what DO and others can provide.

A Pi can be used for development on your home network and it excels at that. And the older RPis can run [1]CouchDB and be configured to "Live Sync" with a CouchDB running on a commercial VPS. That too is a pretty easy to setup and it provides some pretty nice options. For example, you can make your app use the Cloud based CouchDB while you're out and about and it will sync your data with your local CouchDB. Then when you get home you can turn off the cloud access and even delete your data on the cloud DB.

1. The latest version of CoudhDB (v3.0+) doesn't run on the new ARM based Pi 4.

[+] StillBored|5 years ago|reply
The pi4 even overclocked isn't a great number cruncher, and I don't think the gpu acceleration has landed yet..

So I might expect it to be on par with that old of a macbook but not beat it by nearly 2x, particularly if the macbook is being accelerated. (despite having 2x the core count) Which makes me think the MBP may be suffering from some serious thermal throttling, which wouldn't be uncommon on machines of that vintage.

I also assume the call line is:

https://github.com/christian-fei/raspberry-pi-time-lapse/blo...

which is noticeably missing the -hwaccel switch, which means its probably not using the GPU on the mac..

[+] newbie789|5 years ago|reply
This is related because I'm exclusively interested in building a local-network-only media server to serve my firestick and a laptop, but are plex (nice interface, for some reason requires a sign-in to their service, some sort of paid features) or emby (similar) the only options?

If I for example wanted to just access my media library through firestick and windows with a kinda neat interface without paying anybody or making an account on a third party website, is there a solution?

I suppose I could just use VLC, which is fantastic software that isn't particularly beautiful and get used to it, but I'd like a somewhat more "netflix-style" interface for navigating my content within a single rpi server on my network.

[+] gramakri|5 years ago|reply
I have tinkered with RPI a lot in my previous life, I used to maintain the Qt eglfs QPA plugin. Back then, they were quite under powered CPU wise. Are the recent versions powerful enough to host websites and data for every day use? Like say, is it powerful to host a website, couple of blogs, a nextcloud/syncthing instance and say Emby/Jellyfin/Plex? Most importantly, I want to hear about setups that people are using for every day use and not just learning.

(For context, we get a lot of requests to port Cloudron to ARM/RPI but I am still not sure if these are just hobbyists/tinkerers or something people use everyday.)

[+] sneak|5 years ago|reply
Note that if you got all excited about n8n when learning about it from this webpage, as a potential open source Zapier:

n8n is not open source, despite being source available. The author goes to pretty great lengths to avoid confronting this fact.

https://github.com/n8n-io/n8n/issues/40

[+] valiant-comma|5 years ago|reply
For simple, web-based speedtests on a local LAN/WLAN, I like librespeed[1]. Really helps identify subpar WiFi coverage (for example), better than just signal strength. Runs nicely on a Pi, up to line rate (1 Gbps) on a Pi 4.

[1] https://github.com/librespeed/speedtest