For the Pine family of SBCs I highly recommend installing Tow-Boot - https://tow-boot.org/ - on the SPI flash memory to allow yourself much better boot options, including booting directly from NVMe so you don't need to keep the MicroSD card plugged-in.
Does this have a mechanism for automatic redundant OS upgrades? I just built a Yocto-based distribution for a board based on rk3399, but the currently integrated U-Boot is not in the best state. This could be a great alternative if it really is a bit easier to integrate/build upon.
Yes, I considered that and agree that it would have been nicer! I didn't pursue it for this project because my jury-rigged SD boot was working fine and I wanted to move on to other parts of the system.
Does the U-boot in the SPI-NOR not support booting from NVMe? It might also be possible to patch that in from mainline if it exists. You can also often provide a “boot script” in the vfat partition that overrides the boot config in non volatile memory. This was something Freescale did with the i.MX6 that became a relatively standard thing for vendor-supplied U-boot.
The hard way? Copy bootloader from somewhere, partition, extract readymade rootfs, setup bootloader, reboot. Sounds more like the Arch way. :)
The only ARM specific thing here is probably the need to use a DTB.
This just shows that manual Linux installation on random ARM board is not more complex than on x86_64. Perhaps even simpler, since you're just extracting a pre-made rootfs instead of using a package manager during installation.
Right? With that article title you figure the author had written his own bootloader in Typescript then transpiled to Rust, ultimately cross compiling to his Arm64 target from a homebrewed x86 CPU fabricated in his garage.
For real though, what the author did is much harder than downloading and booting an official OS image from Pine. The article also documents all the successful steps and skips any missteps or debugging, making the process look very simple (not a criticism, I thought it was an excellent read). Maybe those missteps took place during previous projects, but suffice to say that you don’t make booting a non-standard image look easy without expending significant effort, at some time.
I was hoping for something more akin to what the word "building" usually implies - something a bit more physical. If nothing else, the author made a box for it ;)
The RockPro64 is a good board with lots of expandability. I run NetBSD/aarch64eb on one to build all of NetBSD's pkgsrc packages (26,000). It performs well with an m.2 NVMe, and has been rock solid.
Of course, for anyone using the RockPro64, if you plan to do lots of processor intensive work like compiling, you'll either need a very large heat sink (no Flirc cases for these, unfortunately) or you'll need active cooling. Without good cooling, it'll throttle.
This seems to be low-power entry-level stuff. I'm curious, is there anything more serious - but less serious than some proper rack server hardware?
Currently I'm running a home server on EPYC 3251 mini-ITX board, which I use to route 1GbE WAN and 10GbE LAN, serve as a NAS, and run a bunch of services all without it breaking a sweat, and leaving plenty of headroom shall I want to run more stuff there. It sits on my desk in a small-ish cubic Supermicro chassis and barely makes any noise beyond the normal HDD screeching. And it's an entry-level server-oriented board so I have proper LOM without having to throw in an IPKVM.
I would fancy an ARMv8 machine - just for fun of it (and possibly better performance per watt) - but I think I can't get anything comparable from a RPi-level hardware. But the next "step" I see when searching for ARM servers are those fan-screaming behemoths you put in a rack in a proper server room, which is something I dread for a homelab, as I don't have a dedicated room for it. I've had a pleasure of WfH involving setting up some PowerEdges in my living room, was fun but extremely noisy. So I wonder, where are the middle grounds?
At this point, if you want a quiet, high-performance ARM system for home, Apple is worth a look, even if that means your "server" storage is plugged into Thunderbolt (and keeping in mind the Apple premium for RAM and internal storage).
It's a middle ground between Raspberry Pi and a 128 core they sell to cloud providers. For the money, you can probably get more work done with an amd64 workstation, unless you're paying someone to generate your electricity by riding a bicycle or something. (Cooling and power matter to cloud-scale datacenters, but not really for one computer in a room that you use to generate your income.)
I set up Proxmox on one of my Pi 4s (using an external SSD) and am quite happy with it. Runs four different LXC containers (one of which is a public-facing ActivityPub server for testing) and gives me zero headaches, so am currently looking for a beefier alternative that has a proper M.2 slot and at least 16GB of RAM...
I do wish that alternative boards had better OS support (especially the Rockchip ones, which tend to have weird kernel builds, etc),
The beefier alternative you're looking for is possibly the Radxa Rock5B, which has proper M.2 slot and comes in a 16GB version. Hardware support for it isn't entirely mainlined yet, but a lot of development is happening week by week. Debian runs well on it.
I moved from a Pi setup to a second hand Lenovo M900 tiny PC, with 24GB of RAM and nvme drive and it works great. The power efficiency is obviously not as good as the Pi but it's a reasonable trade off
It’s a bit of a sledgehammer approach, but a Nuc with Proxmox is pretty excellent. You can even use 10gbe via thunderbolt (or the pci slot on the larger Nucs).
Somewhat tangential but is Arch really suitable for servers? Most Arch users I know still prefer Debian for servers. Yet I know at least one company that uses them for servers, which surprised me. I know the Arch breaking meme is overblown but for a server I'd still want something with less moving parts.
That big release upgrades provide more hassle than benifits was also observed by Google, hence they switched to rolling release. The reason Debian breaks more at release changes though is probably more due to them patching and modifying software, which they have sometimes have to change/drop with a new SW release. Or you have a hard time deploying a newer SW version on top of the old binaries. Arch follows upstream very close, which maybe increases the times things could/have to be reconfigured, but it still mostly means running vimdiff against config.conf and config.conf.pac{new,save}. Sure Debian is more reliable if you want do want to really change deployed systems, but if your company strategy is to keep up with upstream, Arch may work better than it's reputation.
And if you'd need stability once, you could just set Archive on a specific day as package mirror on your cache server.
If this was going to be used for some "big and serious" application, maybe different choices would have been made. Hopefully it was clear from the post that my goals here were the exact opposite!
In my own anecdotal experience of running a hobby server on Arch for several years, I haven't experienced anything to make me think the distro is unsuitable for server work.
> Somewhat tangential but is Arch really suitable for servers?
I think we all use the software we choose to use in order to use the software we choose to use. When we build a cluster, it's often done in order to build a cluster.
Depending on what you use it for, you'll have to babysit your servers more and you'll not be able to do it on your own timeline.
Eg. if major postgresql update comes, you'll have to upgrade your DB cluster very soon. If major update to some program requires configuration changes, or if scripting language has deprecations that you've ignored for years, etc. you'll run into trouble, too.
I've been running a few Arch Linux servers for ~5 years and it's been quite pleasant. Being able to use the latest features in various programs or scripting languages is a very nice benefit.
Run wathever on sd card on its most basic thing. Run lxc and mount storage on /var/lib/lxc.
Well… at least its what I’ve wish I could say.
Truth is I’m using manjaro (arch based) on a similar board and then one day after an upgrade they just decided to migrate from eth0 to the current naming scheme based on nic driver. Had to plug in a monitor and keyboard to fix the situation. Home stuff so its all good in my case.
If you think the price is high, I would point out that the SSD I used cost €200 new when I purchased it back in mid 2022. A used 120 GB SSD by contrast can be had for maybe €10 which alone would explain the difference in cost.
Now if 120 GB is enough for your application, that's a good value so more power to you.
It's a nice write up. but as much as I love whole-drive filesystems, in this case I would have used a partition table and used a fixed partition for the swap space. Not only is it (slightly) more efficient, it's also simpler than using a btrfs subvolume and remembering to +C the swapfile.
I think the general approach is very good and could probably be used for the VisionFive 2 RISC-V SBC as well.
I've been impatient for ARM64 hardware to become easy to use for home servers. I've got an RPi 4 (at retail prices, no less!) and it is quite good but I want more options. The previous stories I've read of RockPro64 have been much worse, it was nice to read this went relatively easily.
You could do this for cheaper (and with more ram) using a raspberry pi 4 with a usb nvme ssd, it’s got gigabit ethernet and is arm64. Sure you have two less cores than this solution but it’s
more likely to be supported over time and once you get the SD card out of the mix the I/O is solid. I’ve been surprised by how much the SD card throughput was limiting the experience.
I run arch Linux arm on mine and it’s a fantastic little device. I wonder if these boards are way faster or just more of the same. I guess the pcie expansion makes this more extensible.
RK3399 is completely FOSS, from bootloader, to firmware, to Linux drivers and mainlined. It will be supported as long as someone wants to run software on it.
I have this same board running freebsd as a NAS. It has been a pretty great experience overall. My main gripe has been that I would really like to be able to run an nvme drive for a cache at the same time as SATA ports, and I haven't found any cost effective PCIE2.0 x4 switches that I could use for that purpose. There is an x1 switch for use with RP4, but it is a shame to lose all the bandwidth.
I'm looking forward to someone making a NAS board on the new RK3588 since that has enough connectivity for everything I want.
Issue with FreeBSD on RockPro64 and similar boards is that the scheduler doesnt support big.little configurations therefore the OS doesnt make distinctions between the 2 A72 cores and the weaker A55s.
I had issues with SD card that may be related to durability. A lot of it was mitigated by moving the /var/log folder to a tmpfs (if you don't care much for the logs, or are using something to ship them to another machine, you really don't mind them not being written to durable storage).
What's with arm sbc users not setting up any disk encryption? I have yet to see a guide / tutorial / experience reports that do not totally skip the subject.
Do it the easy way with Oracle cloud’s free tier and get Arm Ampere A1 CPUs, 24GB RAM, and 10TB egress with the hard part being creating an account which requires a credit card.
That uboot is pretty old, but not as old as the one on my armada 8040 boards... they can't even boot a modern kernel properly without having to compile uboot and ATF and upgrading the firmware.
Got myself a Pi, a plastic box, a memory card, a big usb key, wrote my own SMTP server in super lean no-libc C (c89 with benign bit of c99/c11), put a devuan GNU/linux (NOT debian with its toxic trashy bloat and kludge of systemd).
I did the same thing with a nanomimal http server to serve static content and maybe dynamic in the future: a noscript/basic (x)html http server for maps (which uses openstreet map tiles), which does provide proper map display in links2, with a font not too big, and with harmless html tables.
Configured the "server" to restart everything if something is detected missing (you know, cron with SH scripts and certainly not bash scripts).
It has been running for years. I never had to modify the code of my smtp server, yet (and I run IPv4 and native IPv6 provided by default to millions of clients by my ISP, I think it has been the case for more than a decade, may be wrong about this one though). I am kind of surprise it was not already pown by some trashy hackers.
The main issue: spamhaus block lists, they are hostile to all self-hosted people and they don't provide a irc server, or a non blocked email to be removed from their lists (which are unfortunately used by too many open source related companies/project, which is a mistake). Basically, they force ppl to use one of google/apple super heavy javascripted web engine (no better than the default security checks from cloudflare). Yes, those ppl are seriously worse than spam itself, hope they will fix that (they are a shaddy swiss-andoran company...).
Did you know you cannot send an email to redhat(IBM now) people using an ipv6 smtp? yeah...
And it is coming: I'll move everything to a similar RISC-V mini-computer because I am aware of the super toxic IP tied to arm64 ISA (same for x86_64), that will be the first step, the 2nd step will be to hand compile (=assembly programming with near Zero-SDK) all of them and forget this C syntax too complex and those horribly massive and complex compilers, not stable on the long run (thanks ISO, gcc extensions and c++). And with all that, I would not be surprise to port to 64bits RISC-V assembly a minimal IPv6 stack... and maybe more.
> The main issue: spamhaus block lists, they are hostile to all self-hosted people
Allow me to correct that for you.
There is nothing wrong with spamhaus. They provide one of the best anti-spam options amongst all the commercial providers.
Spamhaus have many lists, I suspect the one you are referring to is the PBL, in their words "DNSBL database of end-user IP address ranges which should not be delivering unauthenticated SMTP email to any Internet mail server except those provided for specifically by an ISP for that customer's use.".
We are in 2023, I think it is beyond any sort of doubt by now that a significant proportion of spam and phishing mails originates from home internet connections because people can't be bothered to keep their computers up to date and virus free, so they become part of a botnet.
So the fact of the matter is that even if Spamhaus PBL did not exist, someone else (or the MX operators themselves) would very soon fill their place by blocking the very same ranges.
Added to which, most home ISPs don't even provide reverse DNS ... so again, even if Spamhaus PBL did not exist, you would likely STILL find yourself being blocked by other measures that most sensible sysadmins implement on their servers.
Hell, many home ISPs just block outbound port 25 these days anyway !
Your efforts are commendable, but you're not correct about Spamhaus and being forced to use Google / Apple.
For starters, nobody is ever forced to use a web browser with email. I'm OK with the fact that pine will parse some of the HTML so I don't see all the silly tags in most email, but it will never follow a link, at least.
If your IPv4 and/or IPv6 is on a Spamhaus list and you can't get it / them removed, likely because you're in a pool of residential IPs, and likely in part because you can't control the PTR, then you can always smarthost through any reasonable provider.
I've been self-hosting email for a quarter of a century, and I'd never blame anyone else if I tried to send email from a residential pool of IPs and it didn't work.
Not sure what this has to do with setting up a nice little ARM server, besides your observation that the ARM architecture is licensed, but here we are :)
daneel_w|3 years ago
fellowmartian|3 years ago
Timon3|3 years ago
jforberg|3 years ago
throwaway173738|3 years ago
megous|3 years ago
The only ARM specific thing here is probably the need to use a DTB.
This just shows that manual Linux installation on random ARM board is not more complex than on x86_64. Perhaps even simpler, since you're just extracting a pre-made rootfs instead of using a package manager during installation.
anthomtb|3 years ago
For real though, what the author did is much harder than downloading and booting an official OS image from Pine. The article also documents all the successful steps and skips any missteps or debugging, making the process look very simple (not a criticism, I thought it was an excellent read). Maybe those missteps took place during previous projects, but suffice to say that you don’t make booting a non-standard image look easy without expending significant effort, at some time.
m463|3 years ago
quectophoton|3 years ago
EDIT: Probably yes. I see this term appears in the article.
johnklos|3 years ago
The RockPro64 is a good board with lots of expandability. I run NetBSD/aarch64eb on one to build all of NetBSD's pkgsrc packages (26,000). It performs well with an m.2 NVMe, and has been rock solid.
Of course, for anyone using the RockPro64, if you plan to do lots of processor intensive work like compiling, you'll either need a very large heat sink (no Flirc cases for these, unfortunately) or you'll need active cooling. Without good cooling, it'll throttle.
https://klos.com/~john/rockpro64.jpeg
e40|3 years ago
drdaeman|3 years ago
Currently I'm running a home server on EPYC 3251 mini-ITX board, which I use to route 1GbE WAN and 10GbE LAN, serve as a NAS, and run a bunch of services all without it breaking a sweat, and leaving plenty of headroom shall I want to run more stuff there. It sits on my desk in a small-ish cubic Supermicro chassis and barely makes any noise beyond the normal HDD screeching. And it's an entry-level server-oriented board so I have proper LOM without having to throw in an IPKVM.
I would fancy an ARMv8 machine - just for fun of it (and possibly better performance per watt) - but I think I can't get anything comparable from a RPi-level hardware. But the next "step" I see when searching for ARM servers are those fan-screaming behemoths you put in a rack in a proper server room, which is something I dread for a homelab, as I don't have a dedicated room for it. I've had a pleasure of WfH involving setting up some PowerEdges in my living room, was fun but extremely noisy. So I wonder, where are the middle grounds?
flyinghamster|3 years ago
jrockway|3 years ago
It's a middle ground between Raspberry Pi and a 128 core they sell to cloud providers. For the money, you can probably get more work done with an amd64 workstation, unless you're paying someone to generate your electricity by riding a bicycle or something. (Cooling and power matter to cloud-scale datacenters, but not really for one computer in a room that you use to generate your income.)
rcarmo|3 years ago
I do wish that alternative boards had better OS support (especially the Rockchip ones, which tend to have weird kernel builds, etc),
daneel_w|3 years ago
djhworld|3 years ago
lostlogin|3 years ago
pella|3 years ago
Orange Pi 5 16GB RK3588S (8 Core 64 Bit, 2.4GHz Frequency), PCIE Module External WiFi+BT,SSD Gigabit Ethernet Single Board Computer,Run Android Debian OS (M.2 PCIe2.0!)
https://www.aliexpress.com/store/group/OPI-5/1553371_4000000...
( via https://news.ycombinator.com/item?id=33739176 )
Review:
"""
"Orange Pi 5 Review – Powerful, No WiFi" https://jamesachambers.com/orange-pi-5-review/
Pros:
- 4 GB and 8 GB RAM variants cost under $100
- M.2 slot supports high speed NVMe storage
- RAM options from 4 GB all the way up to 32 GB available
Cons
- No WiFi or Bluetooth included (requires either adapter for the M.2 slot or a USB adapter to get WiFi/Bluetooth capabilities)
- No eMMC option
- PCIe speeds are limited to 500MB/s (PCIe 2.0, benchmarks show closer to 250MB/s write or PCIe 1.0 performance) — this is slower than SATA3
"""
kryptocannon|3 years ago
andai|3 years ago
bionade24|3 years ago
And if you'd need stability once, you could just set Archive on a specific day as package mirror on your cache server.
jforberg|3 years ago
In my own anecdotal experience of running a hobby server on Arch for several years, I haven't experienced anything to make me think the distro is unsuitable for server work.
rbanffy|3 years ago
I think we all use the software we choose to use in order to use the software we choose to use. When we build a cluster, it's often done in order to build a cluster.
megous|3 years ago
Eg. if major postgresql update comes, you'll have to upgrade your DB cluster very soon. If major update to some program requires configuration changes, or if scripting language has deprecations that you've ignored for years, etc. you'll run into trouble, too.
I've been running a few Arch Linux servers for ~5 years and it's been quite pleasant. Being able to use the latest features in various programs or scripting languages is a very nice benefit.
irusensei|3 years ago
Well… at least its what I’ve wish I could say.
Truth is I’m using manjaro (arch based) on a similar board and then one day after an upgrade they just decided to migrate from eth0 to the current naming scheme based on nic driver. Had to plug in a monitor and keyboard to fix the situation. Home stuff so its all good in my case.
speed_spread|3 years ago
znpy|3 years ago
A few weeks ago I bought an used intel nuc7 with a 7th gen core i5… for 150€.
It came with a 120gb ssd, 4gb ram and a power brick.
I still don’t see the value in this SBCs used as home servers.
jforberg|3 years ago
If you think the price is high, I would point out that the SSD I used cost €200 new when I purchased it back in mid 2022. A used 120 GB SSD by contrast can be had for maybe €10 which alone would explain the difference in cost.
Now if 120 GB is enough for your application, that's a good value so more power to you.
tyingq|3 years ago
Though there are reasons to specifically want an ARM64 machine for builds, etc.
FullyFunctional|3 years ago
I think the general approach is very good and could probably be used for the VisionFive 2 RISC-V SBC as well.
NelsonMinar|3 years ago
I've been impatient for ARM64 hardware to become easy to use for home servers. I've got an RPi 4 (at retail prices, no less!) and it is quite good but I want more options. The previous stories I've read of RockPro64 have been much worse, it was nice to read this went relatively easily.
older|3 years ago
vincnetas|3 years ago
jforberg|3 years ago
rubatuga|3 years ago
sgt|3 years ago
tbrock|3 years ago
I run arch Linux arm on mine and it’s a fantastic little device. I wonder if these boards are way faster or just more of the same. I guess the pcie expansion makes this more extensible.
detrites|3 years ago
The slight problem with the deservedly often-recommended RP4 is that for most people it's so hard to come by it effectively doesn't exist.
johnklos|3 years ago
No, you can't, unless you know of some source of retail priced Raspberry Pi 4s.
In some basic tests (compiling, ffmpeg), the Pi 4 and the Rock Pro 64 are within a small percentage of difference in performance.
megous|3 years ago
Tepix|3 years ago
MobiusHorizons|3 years ago
I'm looking forward to someone making a NAS board on the new RK3588 since that has enough connectivity for everything I want.
irusensei|3 years ago
https://wiki.freebsd.org/arm/RockChip#Known_issues
jraph|3 years ago
I recommend using an SSD and not an SD card though.
rbanffy|3 years ago
prmoustache|3 years ago
EnigmaCurry|3 years ago
_joel|3 years ago
prmoustache|3 years ago
graton|3 years ago
M95D|3 years ago
After about a month I had a barely working uboot built from unpatched official sources.
After two months I still didn't have a bootable kernel built from unpatched official sources.
- with power regulator drivers, the board powers itself off while booting
- without power regulator drivers, it boots the kernel, but there's no power to usb, ethernet and wifi.
What I learned: To stay away from Rockchip.
voidmain0001|3 years ago
https://www.oracle.com/cloud/free/
lockhouse|3 years ago
nubinetwork|3 years ago
megous|3 years ago
sylware|3 years ago
I did the same thing with a nanomimal http server to serve static content and maybe dynamic in the future: a noscript/basic (x)html http server for maps (which uses openstreet map tiles), which does provide proper map display in links2, with a font not too big, and with harmless html tables.
Configured the "server" to restart everything if something is detected missing (you know, cron with SH scripts and certainly not bash scripts).
It has been running for years. I never had to modify the code of my smtp server, yet (and I run IPv4 and native IPv6 provided by default to millions of clients by my ISP, I think it has been the case for more than a decade, may be wrong about this one though). I am kind of surprise it was not already pown by some trashy hackers.
The main issue: spamhaus block lists, they are hostile to all self-hosted people and they don't provide a irc server, or a non blocked email to be removed from their lists (which are unfortunately used by too many open source related companies/project, which is a mistake). Basically, they force ppl to use one of google/apple super heavy javascripted web engine (no better than the default security checks from cloudflare). Yes, those ppl are seriously worse than spam itself, hope they will fix that (they are a shaddy swiss-andoran company...).
Did you know you cannot send an email to redhat(IBM now) people using an ipv6 smtp? yeah...
And it is coming: I'll move everything to a similar RISC-V mini-computer because I am aware of the super toxic IP tied to arm64 ISA (same for x86_64), that will be the first step, the 2nd step will be to hand compile (=assembly programming with near Zero-SDK) all of them and forget this C syntax too complex and those horribly massive and complex compilers, not stable on the long run (thanks ISO, gcc extensions and c++). And with all that, I would not be surprise to port to 64bits RISC-V assembly a minimal IPv6 stack... and maybe more.
traceroute66|3 years ago
Allow me to correct that for you.
There is nothing wrong with spamhaus. They provide one of the best anti-spam options amongst all the commercial providers.
Spamhaus have many lists, I suspect the one you are referring to is the PBL, in their words "DNSBL database of end-user IP address ranges which should not be delivering unauthenticated SMTP email to any Internet mail server except those provided for specifically by an ISP for that customer's use.".
We are in 2023, I think it is beyond any sort of doubt by now that a significant proportion of spam and phishing mails originates from home internet connections because people can't be bothered to keep their computers up to date and virus free, so they become part of a botnet.
So the fact of the matter is that even if Spamhaus PBL did not exist, someone else (or the MX operators themselves) would very soon fill their place by blocking the very same ranges.
Added to which, most home ISPs don't even provide reverse DNS ... so again, even if Spamhaus PBL did not exist, you would likely STILL find yourself being blocked by other measures that most sensible sysadmins implement on their servers.
Hell, many home ISPs just block outbound port 25 these days anyway !
johnklos|3 years ago
For starters, nobody is ever forced to use a web browser with email. I'm OK with the fact that pine will parse some of the HTML so I don't see all the silly tags in most email, but it will never follow a link, at least.
If your IPv4 and/or IPv6 is on a Spamhaus list and you can't get it / them removed, likely because you're in a pool of residential IPs, and likely in part because you can't control the PTR, then you can always smarthost through any reasonable provider.
I've been self-hosting email for a quarter of a century, and I'd never blame anyone else if I tried to send email from a residential pool of IPs and it didn't work.
Not sure what this has to do with setting up a nice little ARM server, besides your observation that the ARM architecture is licensed, but here we are :)
disdi89|3 years ago
kristianpaul|3 years ago
jforberg|3 years ago
jbirer|3 years ago
swtyshinytimmy|3 years ago
swtyshinytimmy|3 years ago