For what it's worth, I loathe this sort of thing (damning by vague questioning): "But, if a composite is full of a measure that’s biased, how accurate is it going to be?"
Well, I don't know, and you haven't told me. Is it a consistent high bias? Low? Variable? How biased? How accurate?
What you're trying to say, without accepting responsibility for actually saying it, is "It's an inaccurate composite because it's based on biased measurements." But you don't know that, or you'd say it without weasel words.
From the study linked [https://link.springer.com/article/10.1007/s40279-024-02066-5] (with acronym meanings added inline for readability):
"The lowest pulse arrival time standard deviation (PAT σ) at 2.0 times the standard deviation corresponded to 88.4% of the Region of Practical Equivalence (ROPE) for SDNN (Standard Deviation of Normal-to-Normal intervals), and 21.4% for RMSSD (Root Mean Square of Successive Differences). As the standard deviation of PAT increases, the equivalence between photoplethysmography-derived "heart rate variability" (PRV) and electrocardiography-derived heart rate variability (HRV) decreases for both SDNN and RMSSD. The width of the highest density interval (HDI), which encompasses 95% of the posterior distribution, increases with increasing PAT σ. This increase occurs at a higher rate for RMSSD than for SDNN."
So for "how accurate?", grossly and irresponsibly oversimplifying, the Apple method is roughly 90% accurate, and devices using the inferior method are roughly 20% accurate.
I would be really interested to know of any devices not locked down to Apple ecosystem that also use this approach, if anyone has any insight.
> Researchers found the method Apple uses, standard deviation of normal-to-normal, or SDNN, is the most accurate. Other brands implement a system called root mean square standard deviation, RMSSD, which Tenan said produces a wider range of error in the measurement
This is interesting for a couple of reasons. Firstly, I have one so it’s cool to know Apple designed it well in that respect, and that I shouldn’t lose confidence in the readings I get.
The second reason it’s interesting is that this kind of decision, which time-domain measure of heart rate variability metric to use, is the kind of super small design decision that really makes a system special.
Having really good developers and designers really shines through for moments like these.
I would love to know from somebody with an Apple watch how you use HRV.
I wear a Garmin, which I mostly like. The HRV measurement is predictive enough to be useful. Two occasions from the past 18 months are worth noting:
1) The measure dropped from green through yellow and into red. It was in yellow 7 days. On the first day it was in red, I was hit with covid symptoms. It stayed in red 2 1/2 weeks before starting to climb again; my symptoms had mostly cleared by the end of the first week, but I still felt sluggish for some time after. The kicker is that I hadn't looked at the HRV at all until sometime during the week of symptoms.
2) I went on a trip to a location +8 hours off my usual timezone, and jet lag symptoms were brutal. HRV dropped into yellow on day one of the trip, and stayed there during the 2 weeks of travel. A couple days after I returned it dropped into red, exactly coinciding with the onset of flu/covid symptoms. Again, it stayed in red throughout the illness, and then as I started to get fit again on the other side it ramped up through yellow and into green.
So even with 20% accuracy (if that is the case), it's a useful data point! What does 90% accuracy give you?
This is incorrect. RMSSD means root mean square successive differences. Interestingly, if you actually compute what you wrote ie root mean square standard deviation, you will get the same result as sdnn.
Just got the latest Apple Watch, so also glad to hear they perform better on HRV. It seems they get a lot of the little details better than competitors in the wearables space. I read previously about how they likely had the most accurate sleep tracking and also had some special sauce in their GPS handling code that made it outperform Garmin.
Apple had goals to make their these health monitoring features in their watch medically approved, to open up additional markets. For other watch makers, it was a checkbox on a feature list.
This is why I use a chest strap heart rate monitor as a cyclist. I see the difference between when my hands are down on the handlebar. It varies around 10-15 bpm (the watch is lower than the chest strap in this scenario). Also, while hiking I see the difference after 2-3 hours. Hikers know the hands swell after 2-3 hours of hiking (you will see people raising both hands while running, that's normal) and this affects my watch heart rate calculations. There is always a discrepancy between the strap and wristwatch.
Watch tracks if my heart rate is high or not but for intense sports like mountain biking I don't expect it to be accurate at all.
I am using a Garmin Instinct 2X Solar Tactical (watch) and a Garmin heart rate chest strap.
When I read the article, this exact question came out as borne out of genuine curiosity justifying the need for measuring and comparing stuff, not a rhetorical question aiming to dismiss the thing.
And indeed, down the road, the article says:
> “There’s a lot going on between when the heart beats and when the blood gets all the way to the arm or wrist, what’s called pulse arrival time,” Tenan explained. “By simulating that and knowing what’s happening at the heart, we determined what’s being measured with the consumer wearable is different. It’s not to say the consumer wearable is bad or not useful, it’s just not the same.”
As in, electric signal ECG/EKG at chest level and blood flow PPG at finger/wrist level factually are not the same measurement, and asking "do they somehow relate in any way?"
Then the question is about the difference between RMSSD and SDNN/SDRR, and here this article (2022) explains it well:
> “I don’t fundamentally see any reason why any of the wearable companies should be using the RMSSD measure
with:
> RMSSD is considered the best measure for short-term variations of HRV, but still a robust measure for longer-term analyses, with typical use cases for tracking stress, sickness, training, and recovery.
> SDNN is more useful for longer term cardiac health trends and analysis.
I've been tracking HRV for about 8 years and can conclude that (for me) any HRV measurement overnight using a wearable or related wearable readiness metrics are largely useless. Now, what I do - as recommended by Marco Altini, an HRV researcher - is measure in the morning, steated with a chest strap. I use his app, HRV4Training which I would recommend [1]. It's the only way to get decent data in my experience. HRV is so senstive to various stimulii you need to measure it under exacting conditions, which you don't really get when you are sleeping. Sleep stages matter, how you are breathing, body temperature, etc.
I think my Oura ring is great but... it consistently records average nighttime HRV of around 15ms, which is implausable. However, when I wake up and measure with my Polar chest strap I'll get a more sensible reading. Fully recovered it will by about 150ms. Hurting after a hard workout it will be around 40ms. Some other interesting experiences with oura/garmin/apple are:
1. Higher nightime HRV after drinking alcohol than abstaining for weeks but then I measure with a strap and it will be low as expected
2. HRV spikes when I get out of bed in the middle to the night to do childcare related things, when it should be lower when walking around
Another thing to note is that I think I have parasympathetic saturation which means due to physiology and environmental factors you actually end up with a super low HRV when you are lying down/resting. Endurance athletes often get it - I run about 100km a week and my resting HR when fully recovered is about 40 bpm, so I guess that's the reason. When I de-train then nighttime HRV goes back to normal.
If anyone is interested in HRV I definitely recommend reading Marco's blog [1], you will learn alot.
There’s nothing new here, most athletes are aware of these issues, and there are plenty of discussions on Reddit. For instance, you can check out the Garmin subreddit. I've even had problems with fitness bands like Polar.
My recommendation is to be cautious with extreme inaccuracies and interpretations, especially for health-related reasons. It's also helpful to develop greater awareness of your body as part of your training.
I agree, and IIRC, Garmin has a warning and disclaimer in their documentation stating the HR isn't guaranteed to be accurate, and it's easy to see in a graph that it's using a moving average.
I still use mine a lot, because I'm not too worried about the exact numbers while excercising, and I feel like it's probably okay for comparing against previous activities. That's to say if the watch shows ~180 bpm for 10 minutes on a run, it's probably similar exertion to the last time I hit ~180 bpm for 10 minutes on a run, but I don't trust that it's exactly 180 bpm, and I definitely don't trust the "ramp up" and "cool down" measurements from 180 bpm.
As you said, everybody I know who cares about accuracy already uses a chest strap.
This is a weird post. I'm by no means an expert but I've read some papers on HRV and this is extremely well known in HRV research. So much so that these facts (and others) are basically copypasta at the beginning of a lot of papers. People have definitely quantified this before, like a lot. Also, the general advice is that it isn't expected to be "accurate" and the measurements are relative anyway (both to the individual and to the standard of measurement.)
There weren't a bunch of researchers (or even biohackers) walking around who thought SDNN or RMSSD could be substituted for EKG.
Sure, the statistical method matters, but I think it's missing the forest for the tree.
Any metric will differ: bpm, sleep phases, body temp, etc.
Because those devices are not super accurate, and the context you wear them in doesn't allow them to be anyway.
If you were to do a respiratory polygraph with proper medical equipment you'll find different values from your Apple Watch or Oura ring for the times you wake up. But you'll also notice a gap between getting the values at home and in a hospital, because in the hospital the nurses will check your equipment setup.
Those devices are great to give you a big picture, follow general trends and understand your own body. That's it.
I find it incredibly puzzling why Polar is left out of these conversations when it comes to wearable health tech when Polar pioneered a lot of the technology and releases white papers for all their functions.
For example, Garmin has "body battery" which is a pseudo science measure of who knows what and Polar does orthostatic tests (based on HRV). Polar won't even let you do that test with a watch, you need a chest strap because of a lot of similar reasons outlined in this paper
Seems odd to compare a machine like a 12 lead EKG to a smartwatch with a single lead EKG support.
Of course variability will differ, that’s why they invest to much into improving the technology and ml algorithms to make up for the difference to personalize the data.
The watch could encourage you to go to a doctor and the doctor can use a more accurate instrument to diagnose.
Every owner of Garmin I know complains of the absolute useless POS it is to rely on the optical HR for HRV. A HRM on a strap will get you good/very good readings for HRV. Finally there is a good podcast from Attia which he talks with an HRV expert behind an HRV dedicated product.
I eventually stopped wearing my Coros running watch because I started to not trust the heart rate thing. Whenever it got cold, the readings shot up! Now I'm just trying to train myself to remember the right tempo from touch.
we had patients come in complaining their smart watch had high readings. after testing they usually had something unusual or indicative. but they were also large... or had other signs.
running an EKG is cheap and easy (might not be cheap to pay for, but it's not an MRI!). i still like patients having more access to health data - legibly - for them. if they were doctors they'd see signs by skin health, etc. the watchs are overall a net positive imo.
.mris are cheap too but a headache to schedule cause everyone needs the machine and there might only be 1 or 2 in a hospital.
But when it's inaccurate then it's usualy blatantly obvious. For example if you are out on an easy jog and it shows 188 bpm then that's your cadence, not your heart, unless something is going down really badly and then you don't need the watch to tell you that.
It's not like it will show 147 when the actual value is 134.
Everyone has a different baseline. For athletic training and recovery they usually look at variations from the baseline rather than the absolute value.
[+] [-] Zanni|1 year ago|reply
Well, I don't know, and you haven't told me. Is it a consistent high bias? Low? Variable? How biased? How accurate?
What you're trying to say, without accepting responsibility for actually saying it, is "It's an inaccurate composite because it's based on biased measurements." But you don't know that, or you'd say it without weasel words.
[+] [-] drilbo|1 year ago|reply
So for "how accurate?", grossly and irresponsibly oversimplifying, the Apple method is roughly 90% accurate, and devices using the inferior method are roughly 20% accurate.
I would be really interested to know of any devices not locked down to Apple ecosystem that also use this approach, if anyone has any insight.
[+] [-] l33t7332273|1 year ago|reply
This is interesting for a couple of reasons. Firstly, I have one so it’s cool to know Apple designed it well in that respect, and that I shouldn’t lose confidence in the readings I get.
The second reason it’s interesting is that this kind of decision, which time-domain measure of heart rate variability metric to use, is the kind of super small design decision that really makes a system special.
Having really good developers and designers really shines through for moments like these.
[+] [-] smallerfish|1 year ago|reply
I wear a Garmin, which I mostly like. The HRV measurement is predictive enough to be useful. Two occasions from the past 18 months are worth noting:
1) The measure dropped from green through yellow and into red. It was in yellow 7 days. On the first day it was in red, I was hit with covid symptoms. It stayed in red 2 1/2 weeks before starting to climb again; my symptoms had mostly cleared by the end of the first week, but I still felt sluggish for some time after. The kicker is that I hadn't looked at the HRV at all until sometime during the week of symptoms.
2) I went on a trip to a location +8 hours off my usual timezone, and jet lag symptoms were brutal. HRV dropped into yellow on day one of the trip, and stayed there during the 2 weeks of travel. A couple days after I returned it dropped into red, exactly coinciding with the onset of flu/covid symptoms. Again, it stayed in red throughout the illness, and then as I started to get fit again on the other side it ramped up through yellow and into green.
So even with 20% accuracy (if that is the case), it's a useful data point! What does 90% accuracy give you?
[+] [-] dxbydt|1 year ago|reply
This is incorrect. RMSSD means root mean square successive differences. Interestingly, if you actually compute what you wrote ie root mean square standard deviation, you will get the same result as sdnn.
[+] [-] raverbashing|1 year ago|reply
I was curious about this and curious if the measurements made by my smart watch were "good" or "bad", and it does calculate RMSSD (allegedly)
Though it seems while they have some correlation they're all over the place https://www.researchgate.net/figure/Scatterplot-for-SDNN-A-B...
SDNN also might apply to a different time range than RMSSD
[+] [-] _9omd|1 year ago|reply
[+] [-] SoftTalker|1 year ago|reply
[+] [-] system2|1 year ago|reply
Watch tracks if my heart rate is high or not but for intense sports like mountain biking I don't expect it to be accurate at all.
I am using a Garmin Instinct 2X Solar Tactical (watch) and a Garmin heart rate chest strap.
[+] [-] Mistletoe|1 year ago|reply
[+] [-] ID1452319|1 year ago|reply
Does it matter if the measure is biased, if you're measuring over time and looking for comparative differences?
[+] [-] lloeki|1 year ago|reply
And indeed, down the road, the article says:
> “There’s a lot going on between when the heart beats and when the blood gets all the way to the arm or wrist, what’s called pulse arrival time,” Tenan explained. “By simulating that and knowing what’s happening at the heart, we determined what’s being measured with the consumer wearable is different. It’s not to say the consumer wearable is bad or not useful, it’s just not the same.”
As in, electric signal ECG/EKG at chest level and blood flow PPG at finger/wrist level factually are not the same measurement, and asking "do they somehow relate in any way?"
Then the question is about the difference between RMSSD and SDNN/SDRR, and here this article (2022) explains it well:
https://tryterra.co/blog/measuring-hrv-sdnn-and-rmssd-3a9b96...
And answers TFA's question:
> “I don’t fundamentally see any reason why any of the wearable companies should be using the RMSSD measure
with:
> RMSSD is considered the best measure for short-term variations of HRV, but still a robust measure for longer-term analyses, with typical use cases for tracking stress, sickness, training, and recovery.
> SDNN is more useful for longer term cardiac health trends and analysis.
[+] [-] rojeee|1 year ago|reply
I think my Oura ring is great but... it consistently records average nighttime HRV of around 15ms, which is implausable. However, when I wake up and measure with my Polar chest strap I'll get a more sensible reading. Fully recovered it will by about 150ms. Hurting after a hard workout it will be around 40ms. Some other interesting experiences with oura/garmin/apple are:
1. Higher nightime HRV after drinking alcohol than abstaining for weeks but then I measure with a strap and it will be low as expected
2. HRV spikes when I get out of bed in the middle to the night to do childcare related things, when it should be lower when walking around
Another thing to note is that I think I have parasympathetic saturation which means due to physiology and environmental factors you actually end up with a super low HRV when you are lying down/resting. Endurance athletes often get it - I run about 100km a week and my resting HR when fully recovered is about 40 bpm, so I guess that's the reason. When I de-train then nighttime HRV goes back to normal.
If anyone is interested in HRV I definitely recommend reading Marco's blog [1], you will learn alot.
[1] https://www.hrv4training.com
[2] https://substack.com/@marcoaltini
[+] [-] wslh|1 year ago|reply
My recommendation is to be cautious with extreme inaccuracies and interpretations, especially for health-related reasons. It's also helpful to develop greater awareness of your body as part of your training.
[+] [-] jlarocco|1 year ago|reply
I still use mine a lot, because I'm not too worried about the exact numbers while excercising, and I feel like it's probably okay for comparing against previous activities. That's to say if the watch shows ~180 bpm for 10 minutes on a run, it's probably similar exertion to the last time I hit ~180 bpm for 10 minutes on a run, but I don't trust that it's exactly 180 bpm, and I definitely don't trust the "ramp up" and "cool down" measurements from 180 bpm.
As you said, everybody I know who cares about accuracy already uses a chest strap.
[+] [-] mannyv|1 year ago|reply
[+] [-] guerrilla|1 year ago|reply
There weren't a bunch of researchers (or even biohackers) walking around who thought SDNN or RMSSD could be substituted for EKG.
[+] [-] BiteCode_dev|1 year ago|reply
Any metric will differ: bpm, sleep phases, body temp, etc.
Because those devices are not super accurate, and the context you wear them in doesn't allow them to be anyway.
If you were to do a respiratory polygraph with proper medical equipment you'll find different values from your Apple Watch or Oura ring for the times you wake up. But you'll also notice a gap between getting the values at home and in a hospital, because in the hospital the nurses will check your equipment setup.
Those devices are great to give you a big picture, follow general trends and understand your own body. That's it.
[+] [-] varsock|1 year ago|reply
For example, Garmin has "body battery" which is a pseudo science measure of who knows what and Polar does orthostatic tests (based on HRV). Polar won't even let you do that test with a watch, you need a chest strap because of a lot of similar reasons outlined in this paper
[+] [-] ck2|1 year ago|reply
Not even at idle, but definitely useless in motion.
And definitely not at the wrist.
Optical at the wrist is all manipulated data via the watch software that is guessing what it is seeing or not.
The only real way is via a heartrate strap that is electrical.
And the next generation of sensors is going to be even more amazing with EKG
https://www.movesense.com/movesense-flash/
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] thenerdhead|1 year ago|reply
Of course variability will differ, that’s why they invest to much into improving the technology and ml algorithms to make up for the difference to personalize the data.
The watch could encourage you to go to a doctor and the doctor can use a more accurate instrument to diagnose.
[+] [-] dyauspitr|1 year ago|reply
[+] [-] gloflo|1 year ago|reply
[+] [-] lencastre|1 year ago|reply
Every owner of Garmin I know complains of the absolute useless POS it is to rely on the optical HR for HRV. A HRM on a strap will get you good/very good readings for HRV. Finally there is a good podcast from Attia which he talks with an HRV expert behind an HRV dedicated product.
[+] [-] beepbooptheory|1 year ago|reply
[+] [-] ineedaj0b|1 year ago|reply
running an EKG is cheap and easy (might not be cheap to pay for, but it's not an MRI!). i still like patients having more access to health data - legibly - for them. if they were doctors they'd see signs by skin health, etc. the watchs are overall a net positive imo.
.mris are cheap too but a headache to schedule cause everyone needs the machine and there might only be 1 or 2 in a hospital.
[+] [-] rubatuga|1 year ago|reply
[+] [-] sadcherry|1 year ago|reply
It's not like it will show 147 when the actual value is 134.
[+] [-] sieabahlpark|1 year ago|reply
[deleted]
[+] [-] byyoung3|1 year ago|reply
[+] [-] byyoung3|1 year ago|reply
[+] [-] Madmallard|1 year ago|reply
815 ms average HRV when normal is like 50-200 ms
[+] [-] nradov|1 year ago|reply
[+] [-] eps|1 year ago|reply