While I agree, it's been hard filling up the 2TB drive in my laptop.
My home server has a couple dozen terabytes (on spinning metal) and, with current fill rate, it's predicted it'll need an increase in space only after two of the drives reach retirement according to SMART. It hosts multiple development VMs and stores backups for all computers in the house.
Another aspect is that the total write lifetime is a multiple of the drive capacity. You can treat a 256TB drive as a very durable 16TB drive, able to last 16 times more writes than the 16TB one.
I vividly remember seeing a 5TB drive at Fry's Electronics sometime around 2010-2013 and thinking to my self "Who in gods name would ever need that much space"
But practically don’t you reach a threshold where storing that much data on one drive makes it a bottleneck and safety risk until the speed of the surrounding systems catch up?
I do digital histology, and our (research) lab currently has 204TB of image files. They live in a data center, of course, but if my institution decided to spin us off as a company or something and we needed to move the data, it'd be way faster to download it to disk and upload it in the destination center. I'm not really sure we'd do it with just one giant drive instead of a whole lot of 1TB ones, but who knows.
(I'm currently working on sending 100TB of images to some colleagues at the NIH for a study, we're doing it about 500GB a night for the next year or however long it'll take just because there's no hurry on the data, so it's not just some academic thought exercise!)
Exactly, we just gobble up all the storage there is. In diagnostics it's easily 250GB per patient just for HEs. And if stuff like CODEX or light-sheet (or some other 3D) microscopy become common place even these drives won't be enough.
That's just 5 20tb drives. Some cloud providers will copy it on to them if you send them in, and usually cheaper than the bandwidth cost. Same for ingress
A drive these days is a CPU, memory, and some flash chips. If the CPU and memory are swappable (isn't in consumer SSDs, no idea about enterprise), then one drive today is really many independent pieces of storage media. Thus, you'd imagine the failure case to be more like the failure case for one entire storage server (pipe drips on it, tornado sucks it up) rather than worrying about the failure case for mechanically-linked hermetically-sealed platters spinning at high speed.
There is always the chance that all the flash chips fail at the same time because of a manufacturing defect. That has always been the gamble over multiple drives as well; many documented cases of all the drives in a RAID array failing at the same time. (This happened to me once! Terrible shipping from NewEgg damaged all 3 drives in my 3x RAID-1 array. I manually repaired them by RMA-ing the disks one at a time; fortunately different blocks failed on different drives and with 3 drives you could do a "best of 2".)
No matter how many independent drives you have, you will always need your data stored in multiple data centers to survive natural disasters. So I don't see 256TB in one device any differently than putting 32 8TB SSDs in one server. If you need that much storage, you spend less of your day plugging it into your computer. Savings!
I bought my first HDD in the mid-90s: 850MB. There was a 1.2GB model, but I thought, "Why would I ever need that much space?" This was before videos, before mp3s, and images were all low quality jpegs.
Back in the summer of 1989, I did an internship at Imprimis (later bought by Seagate). The big thing they launched that summer was the Wren VII hard drive, which was the first consumer hard drive with 1GB of storage. It was massive!
I learned a lot about Statistical Quality Control that summer, and built them some tools for improving their SQC across all their models.
Yes, there are big data applications where direct-attached storage density has a large impact on the economics of working with and analyzing that data. This is mostly due to bandwidth constraints and the fact that many analytical workloads can run efficiently with limited compute/memory relative to storage. Using a vast number of servers when they are not actually required introduces its own challenges. Sensing and telemetry data models sometimes have this property.
Yeah, when you're trying to pack as much storage into a small space, such as data centers with expensive real estate. Or where distance is constrained due to latency, such as high end machine learning workloads.
If I have a century’s worth of point-in-time data, and I want to quickly run various tests against it: data marshaling will kill my soul if I’m using an HDD or various incarnations of hot-cold disk schemes.
Granted, I’ll have to read it in 1TB “pages” since motherboard and RAM engineering haven’t gotten us very far.
they build now with many smaller SSD attached, and maybe with this large one all maintenance process will be encapsulated in one component, so you can think about it as NAS but placed directly inside the server.
Datacenters where you are paying a huge premium on every square inch of physical space used. Also currently every other common way to store this much data eats a lot more power and generates more heat.
I think any data center would want some of these as long as the speeds are enough to reconstruct a disk after a failure in a reasonable time in a RAID configuration.
I’d argue they already are, it’s not infeasible to get ~320tb into an overstuffed NAS (two 10 drive arrays with two drives as parity) currently with a few drives being overly hot and we’re seeing HAMR hit the market now and should probably easily see density double before too long. We’re at the point you can home movie library the size of a streaming site’s library for less than 10k without needing a rack. If you drop density a bit and use used enterprise SAS drives you could probably get things done for 5k + a decent power bill. Still inaccessible to most but plenty accessible to an enthusiast with some disposable income.
QLC would be nice for such a home application over noise/space/power usage concerns but the cost is still extremely high.
nmcela|2 years ago
jaggederest|2 years ago
rbanffy|2 years ago
My home server has a couple dozen terabytes (on spinning metal) and, with current fill rate, it's predicted it'll need an increase in space only after two of the drives reach retirement according to SMART. It hosts multiple development VMs and stores backups for all computers in the house.
Another aspect is that the total write lifetime is a multiple of the drive capacity. You can treat a 256TB drive as a very durable 16TB drive, able to last 16 times more writes than the 16TB one.
KnobbleMcKnees|2 years ago
Ah yes, Yagnibytes.
bguebert|2 years ago
amelius|2 years ago
dabluecaboose|2 years ago
I now have 24 terabytes in my NAS
twerkmonsta|2 years ago
onychomys|2 years ago
(I'm currently working on sending 100TB of images to some colleagues at the NIH for a study, we're doing it about 500GB a night for the next year or however long it'll take just because there's no hurry on the data, so it's not just some academic thought exercise!)
yread|2 years ago
fortysixdegrees|2 years ago
jrockway|2 years ago
There is always the chance that all the flash chips fail at the same time because of a manufacturing defect. That has always been the gamble over multiple drives as well; many documented cases of all the drives in a RAID array failing at the same time. (This happened to me once! Terrible shipping from NewEgg damaged all 3 drives in my 3x RAID-1 array. I manually repaired them by RMA-ing the disks one at a time; fortunately different blocks failed on different drives and with 3 drives you could do a "best of 2".)
No matter how many independent drives you have, you will always need your data stored in multiple data centers to survive natural disasters. So I don't see 256TB in one device any differently than putting 32 8TB SSDs in one server. If you need that much storage, you spend less of your day plugging it into your computer. Savings!
Vt71fcAqt7|2 years ago
>Compared to stacking eight 32TB SSDs, one 256TB SSD consumes approximately seven times less power, despite storing the same amount of data.
This is aimed for servers, where the electricity costs are higher because they are running 24/7 and space is limited. Also less power means less heat.
unknown|2 years ago
[deleted]
jimt1234|2 years ago
Iulioh|2 years ago
Imagine how much data could you generate daily!
Thos aside, a FHD movie in good quality still takes 10-30gb
We are still limited by size and internet speed and need to compress data
bradknowles|2 years ago
I learned a lot about Statistical Quality Control that summer, and built them some tools for improving their SQC across all their models.
jandrewrogers|2 years ago
parl_match|2 years ago
chrisco255|2 years ago
TeMPOraL|2 years ago
unknown|2 years ago
[deleted]
janemydarling|2 years ago
If I have a century’s worth of point-in-time data, and I want to quickly run various tests against it: data marshaling will kill my soul if I’m using an HDD or various incarnations of hot-cold disk schemes.
Granted, I’ll have to read it in 1TB “pages” since motherboard and RAM engineering haven’t gotten us very far.
riku_iki|2 years ago
they build now with many smaller SSD attached, and maybe with this large one all maintenance process will be encapsulated in one component, so you can think about it as NAS but placed directly inside the server.
faeriechangling|2 years ago
Sirikon|2 years ago
Data density is a hell of a drug.
dharmab|2 years ago
dmonitor|2 years ago
faeriechangling|2 years ago
QLC would be nice for such a home application over noise/space/power usage concerns but the cost is still extremely high.
yieldcrv|2 years ago
I have 4k screens and an okay sound system, who are these whatever-philes that think these 60gb rips change the experience meaningfully
like, how many audio tracks you need and the other 50gb so you can stare at black pixels and say “wow these blacks are so impressive”
justsomehnguy|2 years ago
Other cases?
It can be a 1T in some use-cases, but DPWD is probably low.
klysm|2 years ago