> By "everything else", I also mean WireGuard. Did you know that if your machine gets far enough out of sync, that'll stop working, too? I had no idea that it apparently includes time in its crypto stuff, but what other explanation is there?
WireGuard only requires a monotonic clock, because it periodically rotates keys to provide the forward secrecy. Peer clocks are otherwise not required to be synchronized [1]. I guess the clock had a higher rate than usual and it couldn't be corrected due to the lack of RTC?
The Pi ecosystem is coming apart at the seams. Talk about falling off a pedestal.
By the time you buy everything you need to make a Pi 5 even remotely usable and reliable today, you'll have spent as much as a small form factor PC. What do you need?
1. Special power supply - Yes, it's USB-C but it requires a high amperage 5V supply instead of accepting a higher voltage like most USB-C stuff
2. Active cooler - Pi 5 throttles immediately without it; it's not an optional component
3. NVMe Expansion HAT - Seriously why the fiddly little PCIe header? Put an NVMe slot on the bottom and let people adapt *that* connector
4. RTC Battery - At least put a damn capacitor on the board
5. Enclosure - I'm halfway expecting the pi foundation to print some dashed cut lines on the cardboard box so they can say it comes with a case =]
At the end of this you get a system that can run ... raspbian
Not like the Pi Foundation couldn't have contributed to uboot or the upstream kernel or anything before thier hardware was released. Pretty much nothing released for "Raspberry Pi" runs on a Pi 5 at this point in time.
They better try to salvage something before putting out the expected CM5, but I'm not expecting much.
They seem to be obsessed with desktops, and ultra high performance homelab NASes.
They could have just use any old less powerful chip, like, whatever has at least 3B performance is fine, and used the money saved for stuff we actually need.
Maybe ditch those awful micro HDMIs, it's kind of a nasty connector and USB C exists.
Onboard lithium charging is a few cents. Better yet, onboard LTO charging, but it seems nobody does that yet. Why just stop at an RTC battery when you could have a few minutes of real backup power to help stop SD corruption?
While you're at it, add an onboard I2C FRAM or battery backed SRAM.
Instead of NVMe why not hire a few devs to go around fixing stuff that can't run well from the SD card?
The raspberry pi is still a great and easy way to control GPIO lines. IMO that's it's strength, it was never a particularly capable desktop computer or server for any kind of load. The raspberry pi was underpowered compared to a "real computer" when it came out and that's still fine for many applications.
The power supply is becoming less special over time. My first result for "usb c power supply" on amazon is a $25 anker 317 that supports PPS and will do 5 amps at any voltage. Not that that's an amazing price, but the feature is spreading.
But I agree that the price to get a happy Pi is pretty harsh compared to a tiny PC.
I find it hard to believe that it can be worse than RPi 3, which could try to draw up to 15W => 3A under the highest loads, while USB 2 could officially only go up to 2.5 W = 0.5A at 5V, resulting in countless corrupted SD cards. (15 W being the minimum maximum for USB C.)
I ran into a similar problem on a Raspberry Pi, where my program would ignore data because it was deemed too old.
It turns out a Pi uses fake-hwclock. This tool stores a timestamp regularly in a file. On boot the Pi reads from that file to set the time. When the Pi has been turned off for longer then the initial time can be hours or days out of sync.
The Pi then takes a couple of seconds to obtain a valid time via NTP, but my program ran quickly enough to work with the old timestamp. After a couple of minutes it would typically work again.
It's unfortunate that using DHCP to get rough time isn't more popular, especially for devices without RTC. Sure DHCP can be spoofed and should be considered more of a last resort, but it sure would be better than nothing and ending up in this kind of a situation where nothing works anymore.
Speaking of RTC battery, I've recently come to the realization that I have to make sure that BIOS battery is not absolutely dead in always-on PC boxes.
Background: I use an x86 box as home router. I've changed the configuration in the BIOS that it should automatically boot up on power. However if the BIOS battery is dead, the config will be lost and it will revert to default settings, which is not to boot on power.
But there is no way to know how much juice is left in the BIOS battery, thus there is no way to issue warnings that the battery should be replaced soon. If there is power loss AND the battery is dead, when the power comes back up again, the router will just stay off indefinitely until it's manually turned on.
That's when I realized why most consumer routers do not feature RTC nor do they require batteries.
> But there is no way to know how much juice is left in the BIOS battery,
True, but its voltage can be used as an estimate. Most PC MB still around monitor the supply voltages, some also of the battery feeding the RTC. If you have the (Debian) package 'lm-sensors' installed, look for
sensors[<pid>]: Vbat: +2.91 V (min = +2.70 V, max = +3.63 V) ALARM
Some unsolicited solutions, maybe not for you but somebody in a similar situation: maybe replace it with a supercapacitor (needs some wiring changes, otherwise it won't ever be charged, should last long enough for most power outages), or use a stack of coin cells in parallel (difficult due to physical dimensions, and they'll still go dead at some point). You can also short two wires on the ATX supply to automatically turn on the computer when power is supplied (which may not be good enough if you have to press a key during boot to dismiss the low battery warning). Or hack the BIOS :)
Voltage check on battery is part of regular PC maintenance. Like cleaning dust, checking all fans are spinning, capacitors are not getting bigger, checking for weird sounds in PSU, overnight memtest...
I know I’ve brought this up before, but I can’t remember exactly where.
It would be nice if DHCP could provide the current time as one of the pieces of information it provides with a lease. If you’re already trusting it for your IP address, you might as well trust it with the current UTC time: The current 64-bit timestamp would be enough to get timestamp-sensitive things working.
> If you’re already trusting it for your IP address, you might as well trust it with the current UTC time
I don't follow! How does trusting DHCP with IP address automatically mean I should trust it with the current UTC time?
Time requires higher degree of trust than IP.
I may not care what my IP address looks like but I might care a lot about what the current UTC time is and I might want this to come from a more trustworthy device.
> It would be nice if DHCP could provide the current time as one of the pieces of information it provides with a lease. [0]
DHCP can provide the IP address of one or more NTP servers to consult for time queries. I use this feature so that all of the machines on my LAN (at least the ones that can be configured to ask for and/or use this information) have the same general idea of what time it is.
I trust a free wifi hotspot to give me an IP (traffic will be protected by TLS), but not with the time. E. g. time from the past can be used to trick into accepting an expired TLS certificate. But for most wifi hotspots it will be unintentionally wrong because such devices often unmonitored and not maintained until they completely break.
If your DHCP server supports bulk queries (RFC6926: https://www.rfc-editor.org/rfc/rfc6926.html ) then responses to such a query from a dhcp client include a timestamp in the form of an 32-bit integer count of seconds since the epoch.
I don’t know how widely this DHCP RFC is supported though.
It would be nice if DHCP (or maybe wifi?) could provide the location as well, so that you could relay the airplane's geo position to the connected clients.
I had this happen before. I used it as an excuse to learn how to setup NTP with a GPS receiver. I made a little blog on it if anyone is interested in the results. Be sure to click the sandwich menu for some real-time data: https://www.developerdan.com/ntp/
Quick question, because I've followed a number of how tos before that don't, if you disconnected all network connectivity, would it sync to the PPS source? Second, have you tested this, I only ask because 1# I found a lot of time sync was really slow, and #2 some of them reported that they had sub-microsecond time sync but when tested they were several milliseconds out.
Anyway, I solved these problems with sbts-aru, but maybe it's interesting for you to test if you haven't already done this.
Interesting! What is the accuracy like when lacking "PPS precision"/Clayface? I'm wondering about the magnitude by which it is off (seconds, minutes, or other).
PPS turned up quickly on Wikipedia, so that one is readily answered
> PPS signals have an accuracy ranging from 12 picoseconds to a few microseconds per second, or 2.0 nanoseconds to a few milliseconds per day based on the resolution and accuracy of the device generating the signal.
Most SD cards for sale are not suitable for write-intensive random-access workloads. However, they do make SD cards that are intended for this, I've had very good luck with cards marked "high-endurance." They cost more, but they work.
I've always been amazed that the Foundation never mandated the use of validated brands or types of SD cards.
Not that it matters much to me, I've had to retire all of my Pis earlier than the 4. It seems that all of my Raspberry Pis prior to the 4 all had some sort of design issue with their power regulation. After a couple years of constant operation, they just start locking up randomly around once every day or so. I've had everything from a Pi 1 through 3 exhibit this behavior. I spent WEEKS troubleshooting this and conclusively ruled out wonky SD cards, power supplies, connected devices, proximity to other equipment, and every other factor I can think of. I found many threads with other people reporting the same issues but the Foundation refuses to acknowledge it as a problem. I only run Pi 4's now, and am gradually switching over to Intel N100-based systems for low-power operation.
Same with "real" servers. For a while there, it was trendy to boot ESXi from an SD card stuck to the system board and use all of the 2.5" disks for just data storage, no OS. It worked great... until it didn't. It turns out that writing logs and swap to an SD card roasts it after 2-4 years pretty consistently.
Indeed, except if you run an in-memory overlay filesystem this might change dramatically for you. I have several Pi's that run for years and years without SD card corruption this way. Normally, however, as in my sbts-aru project, I don't just do that but also create separate data partitions that are read-write. It appears that even though there are some partitions on the card that are read-write, if they are not the system one it is also much more resilient to fatal corruption.
I heard that this is caused by not supplying enough power to the Pi, did you use the official powerbrick? I have had corruptions happen a couple of times but I have never tried with a good powersource if it actually does a difference.
Using SD cards for an OS is a really bad idea in general. You may make it work with a read only filesystem, but if you don't sooner or later you will have issues.
I dunno, I've always found that to be a... less impressive part of the blog. When the author hits problems in her stack, anyone who would have caught it should shut up and stop being THE ONE. Fine, we've all run into people with big skills and bigger egos. But... the other half of her blog posts are her making snide remarks about how stupidly other people have set stuff up. To her (ex-)coworkers, she is THE ONE. So it leaves a sour taste in my mouth.
> I figured they must be running DNSSEC on that zone (or some part of it), and it must have a "not-before" constraint
Since clients will attempt to resolve ntp.org in order to actually sync their clock, there is a good probability that some clients will be way off.
Enabling dnssec on that zone was probably not without important drawbacks? I wonder if the operators thought about that potential pitfall. Seems like they might be doing a disservice to their core mission of allowing devices to sync their clock.
If you _don't_ enable dnssec on ntp.org, then a mitm can intercept the dns request to redirect to an attacker-owned timeserver with a time in the past. Then the host can have old and expired (without loss of generality) keys/certificates replayed against it.
If memory serves though, Raspbian used to not even have `fake-hwclock` by default and even more Pis would end up with this "wildly wrong time near the epoch and can't bootstrap DNS/NTP" failure mode. You'd sometimes also see it with VMs too (esp if they were doing a pure-software clock that ran slower than realtime, instead of patching clock calls to the real hardware clock on the host).
If you don't have an RTC I'd recommend having a tlsdate with some bounding heuristics to prevent extreme clock fixation from a mitm. You can relatively cheaply hit a large number of public servers that are likely to have good times available over TLS and trust the common result. You validate the certs without considering the notbefore stamps and then if you're feeling aggressive validate them after you've managed to approximate a date from the cohort. I know there are commercial packages that do this, I'm not sure about OSS ones.
Roughtime would be far better, but essentially there's no broad deployment of it yet.
Ideally something good would be picked by Raspbian and delivered in the distro as standard.
It's surprising that it was left out. Ie, the workaround makes sense, but this is a peripheral that is sometimes built into MCUs themselves. For example, most (all?) STM32s include an RTC... with the caveat that if you are using them for canonical use cases, you will need external hardware in the form of a dedicated 32kHz oscillator, and possibly a battery.
For a lot of micro-controller timing uses, they aren't required; for example, hardware timers based on the primary clock source, or the Cortex-M systick, but for maintaining accurate dates and times over long periods, the RTC is the right tool. It can also output the dates and times in a convenient format as well, as long as you find named register fields convenient!
An RTC really needs a battery, and even a smaller battery like a 1220 takes up a fair bit of board space -- space that could be used for another connector or more functionality. And even if you have an RTC, you still have to handle the edge case where the battery dies or is removed and the clock resets. Critical low-level internet services probably shouldn't require clients to already have the correct time.
I use an external RTC with a Raspberry Pi for some sensor systems I work with. Its purpose is to give a reasonable chance of having the correct time for recording data in remote locations where no internet connection is available at startup. For a Pi that only provides network services (and thus always needs a working internet connection), I probably wouldn't think to bother.
RTC is just an electronically readable Casio watch, it's not even responsible for keeping real time while OS is running. All it does is always incrementing regardless of motherboard power state so to provide time as OS loads.
Or... Install one of these on a Pi somewhere in your network and use that as a time source for all of your devices. Problem solved for all your machines.
If you are running Bind or SSL/x.509 certs, than an RTC is pretty much a requirement on the raspberry pi. Network time leaps over NTP even when working (first big leap was allowed etc.) can still bork a lot of services.
Local stratum 1 GPS time (even with PPS pin) servers based on Pi's can suffer similar issues when they lose satellite lock (weather etc.)
Having said that, I've never personally seen my stratum #1 Pi loose sync due to weather and I run about 4 of em here. I also use a cheap 7 euro ublox 7m. So long as you position it or an antennae with a reasonable view of the sky that shouldn't happen I reckon.
Why all the complex solutions when it's so easy and cheap now to install a stratum #1 time server, even a Pi 3 will suffice. It works on a Pi Zero, but then you only have a wifi link.
The sbts-au project make's it incredibly easy to setup. git lone and then install. One command, it's that simple.
The wireguard problem is a pain in the neck. Even if you're happy for your system to not have a realtime clock, when it does come online you'd want wireguard to not start until after the clock has synced to a timeserver. The below is what I'm doing at the moment, but I can't say I'm sure it's working - haven't seen wireguard start before the time is synced since doing it, but it could be probabilistic:
Well... I see where you are going with this but is it necessary? What happens if the clock gets it's correct time sync back, will not wireguard come back online by itself ?
Then the real solution might involve ensuring good clock sync. Using a GPS synched time source is good for this, such as this one :) https://github.com/hcfman/sbts-aru
That was my thought as well - if we know certain things need time to be in sync, can we use systemd dependencies to enforce that? I'm not sure it would help with OP's problem because that was more or less a circular dependency (time sync needed name lookup which needed time to be in sync), but it could mitigate some of the effects or at worst make the problem more explicit/obvious.
Unfortunately a Raspberry Pi is a bit ill suited for production environments. Id recommend an RTC module. Otherwise this might be helpful: https://github.com/hcfman/sbts-aru
Tangential: I have a pi 4 running openwrt and a wireguard interface that is routed over the WiFi access point. If you connect to this network it's as if you are connecting in another country. I have outbound traffic only going over wireguard. If wireguard is down then no internet (to avoid any leakage). However if pi loses power the clock resets, wireguard can't handshake with that much clock skew, NTP can't connect without wireguard. So I need to manually update clock after a power failure.
I guess I need requests to the NTP server to go over WAN directly. Always seemed like a hassle to get this to work with openwrt zones and stuff. My eternal gratitude to anyone who shoves me in the right direction...
If you “always” have Internet access, you should really just place ntpd onto the physical network where it belongs. Otherwise, the options are all mentioned in the article (and might be used in your distribution).
— Restoring more sane shutdown timestamp from file instead of just using 1980.
— Making blind single use ntpdate request to some public server to set local time after the network is available, but before any other application uses it. There might be appropriate hooks in your init system if you don't want to make it an independent step.
— The same, but using some local time source. It is not widely known that even plain old Windows desktops, while not having a complete NTP server, can be trivially configured to announce time over some form of SNTP. If you have any computer with a real clock running nearby, you can probably rely on it. Next step router is also a good potential candidate.
As for compartments and access control, you can try to use basic VLANs (one for ntp traffic, and only it, and one for VPN) or maybe some two-step network configuration process (get time with one configuration, reset and proceed with other services).
All of that is not a problem, the problem is what to do when something fails. Time server becomes unavailable, intermittent internet connectivity, server you use for pings is available, but others are not, etc. You can choose to stop completely if there is no time source, and wait, but that might render the system unavailable for remote configuration. You can try to restart the device or restart the network router automatically (though only a fixed number of times, restart loop will surely break something), but that won't help if the problem gets fixed some time later. You can also use a fallback configuration that doesn't depend on correct time (e. g. only start an ssh server so you can connect to it).
> I guess I need requests to the NTP server to go over WAN directly. ... My eternal gratitude to anyone who shoves me in the right direction...
Do you have something on the LAN side that could run an NTP server? I'm thinking you could add your laptop to your Pi's list of NTP servers, then just start that service up temporarily as-needed to give the Pi a time that's good enough, then it can get actual correct time sync from somewhere else once Wireguard is working.
I also use wireguard a lot. I didn't know that clock sensitivity could be an issue here. I've never seen an issue before. Now that I know it could be one I'll make a point of deploying all cellular based systems with a GPS synched time source as well and avoid that sort of problem.
Yeah you got it. Set the listening interface to eth0 or whatever the wan interface is. Might need to get full ntpd instead of the busy box version. The Interface option here: https://www.ntp.org/documentation/4.2.8-series/miscopt/
I’m actually setting up a router with a rpi4 right now, only using Fedora IoT instead of OpenWrt. It’s a bit more assembly required lol
I’m not sure this is related or not but if there’s one thing I’ve learned about complex objects (engines, computers, software, sewing machines, etc), it’s that every _custom_ thing you do to it has some kind of order of magnitude impact on the overall complexity of the machine, and thus, you should avoid customizing it. At least if your primary concern is usability vs solving specific problems.
It’s definitely served me well in a wide variety of disciplines.
There is not much of an allowance any more for devices that are not on the internet 24/7 without fail.
Multi-booting PCs with Windows and Linux over the years, I have seen the time sync problem go from nonexistent to show-stopper.
For devices that you only need to connect to the internet occasionally or sporadically (so that's what you do), that's where I noticed it most.
Linux sets the RTC to UTC, then when you reboot to Windows, Windows uses the RTC as local time. Which for me is 5 or 6 hours different from UTC, depending on Daylight Savings Time.
Plus when Daylight Savings comes around, each OS wants to make a 1 hour correction the first time you boot it after that date. With multi-booting this can add up to more than one hour difference too.
Didn't used to be so bad, if you were a few hours (days or longer for many PCs) away from the actual time, things did not fail for this reason.
Eventually only one hour off was OK for a while, now nope.
There are just so many more obstacles to smooth reliable operation, and weak links in a more extensive chain that must remain perfectly strong. Otherwise the chain is broken, you are disconnected, and the weak link is too many sections away from you now to be within reach.
Complete perfection is required more so than ever, while at the same time, your efforts to approach perfection are being made more difficult.
Two nights ago I rebooted my router and it wasn't able to update its clock. The logs were filled with "ntp: start NTP update" every 5 seconds.
Well, it was something related to using AdGuard's (94.140.14.14) DNS and pool.ntp.org. I added Google's (8.8.8.8) to resolv.conf temporarily to get the clock synced. But I'm still not sure what the root cause was.
Its weird that people work OK without knowing exactly what time it is. Sure it helps us but if a person lost track of the date for a few days in a city they would still be able to perform basic functions. We only have had timekeeping devices for a tiny part of our history but we invented alot of stuff without precise synchrony.
Isn’t that just because individual human activities aren’t very precise, and for the most part, the position of the sun in the sky is enough for an individual to broadly synchronise themselves with the rest of society.
Unless someone is actually aiming to do a time coordinated task with others, you really need to know the time, or even the day-of-week to do stuff, because most human things in world aren’t actually occurring in small time sensitive windows.
My raspberry pi is also currently dead, something it didn't like about updating the OS caused it to be bricked. All updates were done within the OS, so didn't expect anything bad to happen.
I could reflash the OS, but I also "don't have the time". I'll just use my laptop until I do have the time someday to resurrect it.
Quite likely installing the sbts-aru as a base project would have prevented this as well. Loads of SD cards get corrupted historically without apparent good reason. Using a memory-based overlayFS file system solves so many of these problems. sbts-aru, as well as setting up a stratum #1 time server sets up a resilient memory overlayFS as a base.
I guess that's what the article meant here: "This particular Pi doesn't have a real-time clock. The very newest ones (5B) do, but you have to actually buy a battery and connect it. "
This is the perennial problem with trying to secure NTP. I've also seen people suggest that we just use HTTPS for time sync, not realizing that if your clocks are too far off you can't make a HTTPS connection.
Plus the time to learn a few modprobe commands. Perhaps run a few commands echoing numbers to random sysfs files. Perhaps learn WTF a 'Device Tree Overlay' is - a concept that's completely absent on x86 systems. You see, this is an 'I2C' device, so it doesn't plug and play like USB and PCIe devices do. It is much simpler, and hence for some reason more difficult. Perhaps your device tree overlay needs to be complied? Or maybe not. Perhaps you'll edit your modules file and your kernel command line. Of course that won't involve update-grub, why would you think that?
Don't forget to periodically check on your settings, just in case an OS update or something has disabled one of those settings.
Oh and of course you won't be able to fully test this over SSH, if the network is up NTP will have set your clock whether the RTC is set up right or not. As we've learned from the article, if you run into clock problems you might not be able to connect remotely.
Then simply invent a time machine so you can do all of this in the past, before the pi SD card fails and it drops off the network.
lifthrasiir|1 year ago
WireGuard only requires a monotonic clock, because it periodically rotates keys to provide the forward secrecy. Peer clocks are otherwise not required to be synchronized [1]. I guess the clock had a higher rate than usual and it couldn't be corrected due to the lack of RTC?
[1] https://www.wireguard.com/papers/wireguard.pdf#page=7 "In fact, it does not even have to be an accurate timestamp; it simply must be a per-peer monotonically increasing 96-bit number."
josephg|1 year ago
gorkish|1 year ago
By the time you buy everything you need to make a Pi 5 even remotely usable and reliable today, you'll have spent as much as a small form factor PC. What do you need?
At the end of this you get a system that can run ... raspbianNot like the Pi Foundation couldn't have contributed to uboot or the upstream kernel or anything before thier hardware was released. Pretty much nothing released for "Raspberry Pi" runs on a Pi 5 at this point in time.
They better try to salvage something before putting out the expected CM5, but I'm not expecting much.
eternityforest|1 year ago
They could have just use any old less powerful chip, like, whatever has at least 3B performance is fine, and used the money saved for stuff we actually need.
Maybe ditch those awful micro HDMIs, it's kind of a nasty connector and USB C exists.
Onboard lithium charging is a few cents. Better yet, onboard LTO charging, but it seems nobody does that yet. Why just stop at an RTC battery when you could have a few minutes of real backup power to help stop SD corruption?
While you're at it, add an onboard I2C FRAM or battery backed SRAM.
Instead of NVMe why not hire a few devs to go around fixing stuff that can't run well from the SD card?
tdeck|1 year ago
Dylan16807|1 year ago
But I agree that the price to get a happy Pi is pretty harsh compared to a tiny PC.
BlueTemplar|1 year ago
I find it hard to believe that it can be worse than RPi 3, which could try to draw up to 15W => 3A under the highest loads, while USB 2 could officially only go up to 2.5 W = 0.5A at 5V, resulting in countless corrupted SD cards. (15 W being the minimum maximum for USB C.)
beryilma|1 year ago
The tooling for light software development and hobby embedded hardware development has been great so far.
mplaisier|1 year ago
zokier|1 year ago
Previous discussion on similar thing: "Encrypted DNS + NTP = Deadlock" https://news.ycombinator.com/item?id=34177331
riobard|1 year ago
Background: I use an x86 box as home router. I've changed the configuration in the BIOS that it should automatically boot up on power. However if the BIOS battery is dead, the config will be lost and it will revert to default settings, which is not to boot on power.
But there is no way to know how much juice is left in the BIOS battery, thus there is no way to issue warnings that the battery should be replaced soon. If there is power loss AND the battery is dead, when the power comes back up again, the router will just stay off indefinitely until it's manually turned on.
That's when I realized why most consumer routers do not feature RTC nor do they require batteries.
guenthert|1 year ago
True, but its voltage can be used as an estimate. Most PC MB still around monitor the supply voltages, some also of the battery feeding the RTC. If you have the (Debian) package 'lm-sensors' installed, look for
in /var/log/syslog.qiqitori|1 year ago
qmarchi|1 year ago
Hrnrurj|1 year ago
You should do it every year or two...
CaliforniaKarl|1 year ago
It would be nice if DHCP could provide the current time as one of the pieces of information it provides with a lease. If you’re already trusting it for your IP address, you might as well trust it with the current UTC time: The current 64-bit timestamp would be enough to get timestamp-sensitive things working.
mapreduce|1 year ago
I don't follow! How does trusting DHCP with IP address automatically mean I should trust it with the current UTC time?
Time requires higher degree of trust than IP.
I may not care what my IP address looks like but I might care a lot about what the current UTC time is and I might want this to come from a more trustworthy device.
simoncion|1 year ago
DHCP can provide the IP address of one or more NTP servers to consult for time queries. I use this feature so that all of the machines on my LAN (at least the ones that can be configured to ask for and/or use this information) have the same general idea of what time it is.
[0] See subsection 8.3 here: <https://www.rfc-editor.org/rfc/rfc2132#section-8>
citrin_ru|1 year ago
pja|1 year ago
I don’t know how widely this DHCP RFC is supported though.
flemhans|1 year ago
unknown|1 year ago
[deleted]
lightswitch05|1 year ago
hcfman|1 year ago
Anyway, I solved these problems with sbts-aru, but maybe it's interesting for you to test if you haven't already done this.
boneitis|1 year ago
PPS turned up quickly on Wikipedia, so that one is readily answered
> PPS signals have an accuracy ranging from 12 picoseconds to a few microseconds per second, or 2.0 nanoseconds to a few milliseconds per day based on the resolution and accuracy of the device generating the signal.
antx|1 year ago
https://github.com/semuconsulting/PyGPSClient
shawabawa3|1 year ago
I only ever use USB storage on my pi's now and not had a single issue since
bityard|1 year ago
I've always been amazed that the Foundation never mandated the use of validated brands or types of SD cards.
Not that it matters much to me, I've had to retire all of my Pis earlier than the 4. It seems that all of my Raspberry Pis prior to the 4 all had some sort of design issue with their power regulation. After a couple years of constant operation, they just start locking up randomly around once every day or so. I've had everything from a Pi 1 through 3 exhibit this behavior. I spent WEEKS troubleshooting this and conclusively ruled out wonky SD cards, power supplies, connected devices, proximity to other equipment, and every other factor I can think of. I found many threads with other people reporting the same issues but the Foundation refuses to acknowledge it as a problem. I only run Pi 4's now, and am gradually switching over to Intel N100-based systems for low-power operation.
yabones|1 year ago
hcfman|1 year ago
Moru|1 year ago
constantcrying|1 year ago
cbhl|1 year ago
For those of you who didn't read the fine article, I've quoted it here:
> As usual, this post is not a request for THE ONE to show up. If you are THE ONE, you don't make mistakes. We know. Shut up and go away.
yjftsjthsd-h|1 year ago
smashed|1 year ago
Since clients will attempt to resolve ntp.org in order to actually sync their clock, there is a good probability that some clients will be way off.
Enabling dnssec on that zone was probably not without important drawbacks? I wonder if the operators thought about that potential pitfall. Seems like they might be doing a disservice to their core mission of allowing devices to sync their clock.
growse|1 year ago
Is my clock more likely to be accurate after NTP.org got signed? It looks like it's less likely, on average.
cbhl|1 year ago
If memory serves though, Raspbian used to not even have `fake-hwclock` by default and even more Pis would end up with this "wildly wrong time near the epoch and can't bootstrap DNS/NTP" failure mode. You'd sometimes also see it with VMs too (esp if they were doing a pure-software clock that ran slower than realtime, instead of patching clock calls to the real hardware clock on the host).
raggi|1 year ago
Roughtime would be far better, but essentially there's no broad deployment of it yet.
Ideally something good would be picked by Raspbian and delivered in the distro as standard.
the__alchemist|1 year ago
For a lot of micro-controller timing uses, they aren't required; for example, hardware timers based on the primary clock source, or the Cortex-M systick, but for maintaining accurate dates and times over long periods, the RTC is the right tool. It can also output the dates and times in a convenient format as well, as long as you find named register fields convenient!
AdamH12113|1 year ago
I use an external RTC with a Raspberry Pi for some sensor systems I work with. Its purpose is to give a reasonable chance of having the correct time for recording data in remote locations where no internet connection is available at startup. For a Pi that only provides network services (and thus always needs a working internet connection), I probably wouldn't think to bother.
01HNNWZ0MV43FF|1 year ago
That was actually the easier hardware platform, because it either had good NTP sync or it didn't reach our C&C servers.
The difficult one was the platform that had an RTC. Those would slooowly drift out of sync if NTP was broken. Slowly and silently.
numpad0|1 year ago
hcfman|1 year ago
https://github.com/hcfman/sbts-aru
Problem solved.
hcfman|1 year ago
mkesper|1 year ago
Joel_Mckay|1 year ago
Local stratum 1 GPS time (even with PPS pin) servers based on Pi's can suffer similar issues when they lose satellite lock (weather etc.)
Have a great day =)
hcfman|1 year ago
Why all the complex solutions when it's so easy and cheap now to install a stratum #1 time server, even a Pi 3 will suffice. It works on a Pi Zero, but then you only have a wifi link.
The sbts-au project make's it incredibly easy to setup. git lone and then install. One command, it's that simple.
benmmurphy|1 year ago
tptacek|1 year ago
cjbillington|1 year ago
hcfman|1 year ago
Then the real solution might involve ensuring good clock sync. Using a GPS synched time source is good for this, such as this one :) https://github.com/hcfman/sbts-aru
yjftsjthsd-h|1 year ago
wodenokoto|1 year ago
bjackman|1 year ago
dmvdoug|1 year ago
semireg|1 year ago
malivvan|1 year ago
silasdavis|1 year ago
I guess I need requests to the NTP server to go over WAN directly. Always seemed like a hassle to get this to work with openwrt zones and stuff. My eternal gratitude to anyone who shoves me in the right direction...
ogurechny|1 year ago
— Restoring more sane shutdown timestamp from file instead of just using 1980.
— Making blind single use ntpdate request to some public server to set local time after the network is available, but before any other application uses it. There might be appropriate hooks in your init system if you don't want to make it an independent step.
— The same, but using some local time source. It is not widely known that even plain old Windows desktops, while not having a complete NTP server, can be trivially configured to announce time over some form of SNTP. If you have any computer with a real clock running nearby, you can probably rely on it. Next step router is also a good potential candidate.
As for compartments and access control, you can try to use basic VLANs (one for ntp traffic, and only it, and one for VPN) or maybe some two-step network configuration process (get time with one configuration, reset and proceed with other services).
All of that is not a problem, the problem is what to do when something fails. Time server becomes unavailable, intermittent internet connectivity, server you use for pings is available, but others are not, etc. You can choose to stop completely if there is no time source, and wait, but that might render the system unavailable for remote configuration. You can try to restart the device or restart the network router automatically (though only a fixed number of times, restart loop will surely break something), but that won't help if the problem gets fixed some time later. You can also use a fallback configuration that doesn't depend on correct time (e. g. only start an ssh server so you can connect to it).
nix0n|1 year ago
Do you have something on the LAN side that could run an NTP server? I'm thinking you could add your laptop to your Pi's list of NTP servers, then just start that service up temporarily as-needed to give the Pi a time that's good enough, then it can get actual correct time sync from somewhere else once Wireguard is working.
hcfman|1 year ago
pynne|1 year ago
I’m actually setting up a router with a rpi4 right now, only using Fedora IoT instead of OpenWrt. It’s a bit more assembly required lol
dclowd9901|1 year ago
It’s definitely served me well in a wide variety of disciplines.
fuzzfactor|1 year ago
Multi-booting PCs with Windows and Linux over the years, I have seen the time sync problem go from nonexistent to show-stopper.
For devices that you only need to connect to the internet occasionally or sporadically (so that's what you do), that's where I noticed it most.
Linux sets the RTC to UTC, then when you reboot to Windows, Windows uses the RTC as local time. Which for me is 5 or 6 hours different from UTC, depending on Daylight Savings Time.
Plus when Daylight Savings comes around, each OS wants to make a 1 hour correction the first time you boot it after that date. With multi-booting this can add up to more than one hour difference too.
Didn't used to be so bad, if you were a few hours (days or longer for many PCs) away from the actual time, things did not fail for this reason.
Eventually only one hour off was OK for a while, now nope.
There are just so many more obstacles to smooth reliable operation, and weak links in a more extensive chain that must remain perfectly strong. Otherwise the chain is broken, you are disconnected, and the weak link is too many sections away from you now to be within reach.
Complete perfection is required more so than ever, while at the same time, your efforts to approach perfection are being made more difficult.
champtar|1 year ago
aimor|1 year ago
Well, it was something related to using AdGuard's (94.140.14.14) DNS and pool.ntp.org. I added Google's (8.8.8.8) to resolv.conf temporarily to get the clock synced. But I'm still not sure what the root cause was.
doubloon|1 year ago
avianlyric|1 year ago
Unless someone is actually aiming to do a time coordinated task with others, you really need to know the time, or even the day-of-week to do stuff, because most human things in world aren’t actually occurring in small time sensitive windows.
hcfman|1 year ago
jlubawy|1 year ago
I could reflash the OS, but I also "don't have the time". I'll just use my laptop until I do have the time someday to resurrect it.
hcfman|1 year ago
cyounkins|1 year ago
the-kenny|1 year ago
billsmithaustin|1 year ago
koala_man|1 year ago
jandrese|1 year ago
kakkoko|1 year ago
While systemd-timesyncd contains a workaround for this, chrony does not seem to.
unknown|1 year ago
[deleted]
iforgotpassword|1 year ago
M95D|1 year ago
michaelt|1 year ago
Plus the time to learn a few modprobe commands. Perhaps run a few commands echoing numbers to random sysfs files. Perhaps learn WTF a 'Device Tree Overlay' is - a concept that's completely absent on x86 systems. You see, this is an 'I2C' device, so it doesn't plug and play like USB and PCIe devices do. It is much simpler, and hence for some reason more difficult. Perhaps your device tree overlay needs to be complied? Or maybe not. Perhaps you'll edit your modules file and your kernel command line. Of course that won't involve update-grub, why would you think that?
Don't forget to periodically check on your settings, just in case an OS update or something has disabled one of those settings.
Oh and of course you won't be able to fully test this over SSH, if the network is up NTP will have set your clock whether the RTC is set up right or not. As we've learned from the article, if you run into clock problems you might not be able to connect remotely.
Then simply invent a time machine so you can do all of this in the past, before the pi SD card fails and it drops off the network.
icehawk|1 year ago
(And you can do the same thing were you write the time to the RTC before you have the correct time)