I like the idea. Making it backwards compatible with FAT means that, in principle, regular FAT filesystem implementations could be transparently changed to support big fat files (hehe) transparently.
However, reading the spec, it doesn't look fully backwards compatible? It seems like there are file structures which are possible to represent in FAT which aren't possible to represent in BigFAT. In FAT, I could have a 4GB-128kB size file called "hello.txt", and next to it, a file called "hello.txt.000.BigFAT". A FAT filesystem will show this as intended, but a BigFAT implementation will show it as one file' "hello.txt". That makes this a breaking change.
I would kind of have hoped that they had found an unused but always-zero bit in some header which could be repurposed to identify whether a file has a continuation or not, or some other clever way of ensuring that you can represent all legal FAT32 file structures.
There are so many good filesystems out there. Is it really necessary to keep dragging FAT along?
ReactOS is using btrfs, which has so many useful options that FAT will never see (zstd, xxhash, flash-aware options, snapshots, send/receive, etc.). This is positioned both for Linux and Windows.
Microsoft itself restrains ReFS to enterprise use, and btrfs offers so much more functionality. We should stop using a file system from the '80s.
I imagine the premise is that if you mount this disk in an implementation that doesn't understand these structures it works and you don't corrupt it, making the format backwards compatible with old implementations. This is similar to the trick used to add long filenames: putting a special 8.3 file with a ~ that includes the full file name.
>Unfortunately, exFAT has been adopted by the SD Association as the default file system for SDXC cards larger than 32 GB. In our view, this should never have happened, as it forces anyone who wants to access SDXC cards to get a license from Microsoft, basically making this a field owned by Microsoft.
So, this is a bit of a cultural/perception gap between FOSS developers and standards bodies. Most standards bodies have a patent policy of "as long as all the standards-essential patents are licensable for a uniform fee, we're good". Convincing patent holders to not extract royalties from their patents for the sake of easing the lives of FOSS implementers is much, much harder[0].
Microsoft isn't even the only SEP holder for SD, and the standard makes no attempt at being a royalty-free standard. In fact, early SD standards were NDA'd[1] and prohibited FOSS implementation at all.
[0] In fact, so hard that the EU has a conspiracy theory that Google/AOM bullied a patent holder into doing this
[1] Remember, SD cards were basically MMC with primitive DRM
> We also support the eventual inclusion of a Linux kernel with exFAT support in a future revision of the Open Invention Network’s Linux System Definition, where, once accepted, the code will benefit from the defensive patent commitments of OIN’s 3040+ members and licensees.
I don’t know exactly what that means. But it sounds like something different from “we hereby grant everybody a license to any and all exFAT patents.”
> Why not exFAT... Microsoft owns several patents, and anyone who implements or uses exFAT technology needs Microsoft's permission, which typically also includes paying fees to Microsoft.
While BigFAT not being encumbered by any patents is a good thing, the camera industry have pretty much standardized on exFAT for their removable file storage format. Something I'm curious about is how a 5GB video file (quite common and actually on the smaller size for 4K and 8K recording sessions) is written and accessed between the two file systems. BigFAT says that the file would be written in 4GB chunks; is there something similar happening with exFAT or is the file "one chunk?" (Apologies if I have the terms wrong -- I'm not a filesystem expert.) The author laments that the exFAT format has been adopted for SDXC cards but given who all is in this group and what their use cases are I can discount "because Microsoft strong-armed them" as a reason for them selecting it.
The industry could have used UDF. Derived from ISO 9660, but it supports read-write random access storage.
I'm guessing they didn't if FAT12/16/32->exFAT driver changes are comparatively simple, and/or results in a smaller code base to support FAT32 and exFAT on the same device (e.g. a camera).
I actually am disappointed that Microsoft has a chance to fix some inherent problems with FAT but didn't, even considering the main use case of a simple FS. Notably, it still has the notorious year 2100 bug (or 2108 bug, depending on the implementation), the metadata is weird and not at all straightforward, it's basically just extending FAT32 and some minor updates since Unicode and UTC are now here.
The question I have is, why Segger? When I saw this I was like "the debugger company?!?!" Clearly this wouldn't fall under their business, so it makes sense for them to open it up, but why did they do it in the first place?
They offer their own file system implementation (emFile) which supports either their own storage format (EFS) or FAT. The BigFAT article is posted in the emFile section of their website.
My suspicion is that customers are bugging them to support large files in emFile and they don't want to pay the license fee for exFAT. I think they even can't do that with their current licensing model, which is one-time per product (not item) or product-family payment.
EDIT: I tried to find out if Microsoft's exFAT is licensed per product or per unit and I found that it used to be a 300000 USD flat fee in 2009 but seems to be free since 2019. So my theory from above has no basis and I wonder why Segger does not simply implement exFAT?
I'm a bit puzzled as to how split files with name standardization is an "extension." It seems to me that SEGGER is simply proposing a de facto file naming convention, and offering a few free tools (including a few abstraction drivers) to encourage adoption.
Can somebody fill me in, here- where's the value in what SEGGER is proposing, as opposed to what the entire IT community has already been doing for decades?
Well, if we view FAT32 + this name convention as a new filesystem, then filesystem drivers could let you transparently operate on files bigger than 4GB (GiB?) and take care of the splitting for you. FAT32 + this convention would essentially become a filesystem which supports files up to around 4TB. You wouldn't have to make the choice between the patent-encumbered exFAT and the open but limited FAT32.
I think many programmers have used file splitting technology at the application level for decades. I know I wrote one for a backup utility (Drive Image) back in the 90s that would split the output files into smaller pieces for transfer to removable media (floppies, Zip drives, CDs, DVDs, etc.).
It sounds like BigFat is an extension that takes away the need to do this at the application level. The code does all the splitting and merging for you so you can write a program that acts like the file is on a file system that supports files > 4GB.
I had the same impression, I cannot see how it is different from the tools that allow file splitting (for later archiving on floppies. CD's, DVD's, etc.) that exist since forever, when/if implemented in OS filesystem drivers then it will be more "transparent", but until then it seems to me not different from multi-part archives, such as rar or similar.
If I may it would make more sense (to me at least) to use a directory and have a descriptor file, not entirely unlike multi-part vmdk's are implemented.
Edit: I guess my first look is wrong, on second look, it appears it needs their own filesystem driver. If the hack as I wrote would've worked, it would've been very clever, and dangerous...
> de facto file naming convention
From a first look, like it's using Microsoft's own hack of long file names[1] to create file entries that look like they belong to 1 file. A file that has a long file name (more than the 8+3 character limit) is actually several file entries, but they're empty files. Seems like the tool is creating non-empty files instead, that Windows is chaining together as one.
Is this only compatible with FAT32, or is it also compatible with FAT12/16? It would be very cool if this would support floppy disks.
Regarding the format, once you convert it, does the target device need to have a driver to support the format? It mentions that this would allow for > 4GB files for TVs, but these are typically non-updated very out of date OSes.
I think MS missed a trick by not making the boot sector also contain a simplistic driver, although it would have been a push to keep it all down at 512 bytes.
It would be great to have a non-patent-encumbered simple file format that's supported everywhere. The fact that this is based on FAT32 might help adoption, everyone's computers can already at least read a BigFAT drive, and BigFAT support could be added at the application level for systems which don't support it at an OS level.
Straight from their FAQ:
We see emFile customers asking for solutions for bigger files. Implementing exFAT is not an option for us, as it is patent encumbered. SEGGER would need Microsoft's permission to implement and offer it, and our customers need to deal with Microsoft again to be able to use it in their products. This can be time-consuming and also expensive. We feel there should be a free alternative. The more popular BigFAT becomes, the better.
I guess using anything but FAT would make it hard for their developer base.
I already use FAT32 for some USB sticks when I need to be able to use them on various OSes without having to give it any thought, or for long term archiving.
This would be extremely niche but would have its audience. Heck, I wouldn't be surprised if HN readers adopted it just to taste the thrill of unlimited and obscene power.
Would it not be possible to create a filesystem with modern capabilities but with backwards compatibility with FAT? Why can't we just have "legacy" commands built into the ReFS filesystem that process any FAT filesystem access?
I'm very ignorant to this but I'd love some insight from someone vastly more knowledgeable than me.
mort96|3 years ago
However, reading the spec, it doesn't look fully backwards compatible? It seems like there are file structures which are possible to represent in FAT which aren't possible to represent in BigFAT. In FAT, I could have a 4GB-128kB size file called "hello.txt", and next to it, a file called "hello.txt.000.BigFAT". A FAT filesystem will show this as intended, but a BigFAT implementation will show it as one file' "hello.txt". That makes this a breaking change.
I would kind of have hoped that they had found an unused but always-zero bit in some header which could be repurposed to identify whether a file has a continuation or not, or some other clever way of ensuring that you can represent all legal FAT32 file structures.
chasil|3 years ago
ReactOS is using btrfs, which has so many useful options that FAT will never see (zstd, xxhash, flash-aware options, snapshots, send/receive, etc.). This is positioned both for Linux and Windows.
Microsoft itself restrains ReFS to enterprise use, and btrfs offers so much more functionality. We should stop using a file system from the '80s.
NelsonMinar|3 years ago
DannyBee|3 years ago
For example, LFN fails if you create too many files with the same first 6 letters :)
I'm actually honestly not sure why representing all legal FAT32 file structures is a particularly useful goal?
FAT in particular, in all of it's forms, has always had limitations and weirdness in filenames, etc.
saurik|3 years ago
kmeisthax|3 years ago
So, this is a bit of a cultural/perception gap between FOSS developers and standards bodies. Most standards bodies have a patent policy of "as long as all the standards-essential patents are licensable for a uniform fee, we're good". Convincing patent holders to not extract royalties from their patents for the sake of easing the lives of FOSS implementers is much, much harder[0].
Microsoft isn't even the only SEP holder for SD, and the standard makes no attempt at being a royalty-free standard. In fact, early SD standards were NDA'd[1] and prohibited FOSS implementation at all.
[0] In fact, so hard that the EU has a conspiracy theory that Google/AOM bullied a patent holder into doing this
[1] Remember, SD cards were basically MMC with primitive DRM
CodesInChaos|3 years ago
> exFAT was a proprietary file system until 2019, when Microsoft released the specification and allowed OIN members to use their patents.
https://en.wikipedia.org/wiki/ExFAT#Legal_status
slavik81|3 years ago
https://patents.google.com/patent/US20090164440?oq=US2009164...
NotYourLawyer|3 years ago
I don’t know exactly what that means. But it sounds like something different from “we hereby grant everybody a license to any and all exFAT patents.”
loeg|3 years ago
mikece|3 years ago
While BigFAT not being encumbered by any patents is a good thing, the camera industry have pretty much standardized on exFAT for their removable file storage format. Something I'm curious about is how a 5GB video file (quite common and actually on the smaller size for 4K and 8K recording sessions) is written and accessed between the two file systems. BigFAT says that the file would be written in 4GB chunks; is there something similar happening with exFAT or is the file "one chunk?" (Apologies if I have the terms wrong -- I'm not a filesystem expert.) The author laments that the exFAT format has been adopted for SDXC cards but given who all is in this group and what their use cases are I can discount "because Microsoft strong-armed them" as a reason for them selecting it.
cmurf|3 years ago
I'm guessing they didn't if FAT12/16/32->exFAT driver changes are comparatively simple, and/or results in a smaller code base to support FAT32 and exFAT on the same device (e.g. a camera).
lathiat|3 years ago
I guess 4GB seemed like a reasonable limit when FAT32 was designed.
Most likely FAT32 has a 32bit number for file size and ExFAT presumably has either a 64bit one or stores file size in some format other than bytes.
zinekeller|3 years ago
phkahler|3 years ago
weinzierl|3 years ago
My suspicion is that customers are bugging them to support large files in emFile and they don't want to pay the license fee for exFAT. I think they even can't do that with their current licensing model, which is one-time per product (not item) or product-family payment.
EDIT: I tried to find out if Microsoft's exFAT is licensed per product or per unit and I found that it used to be a 300000 USD flat fee in 2009 but seems to be free since 2019. So my theory from above has no basis and I wonder why Segger does not simply implement exFAT?
matja|3 years ago
ninefathom|3 years ago
Can somebody fill me in, here- where's the value in what SEGGER is proposing, as opposed to what the entire IT community has already been doing for decades?
mort96|3 years ago
didgetmaster|3 years ago
It sounds like BigFat is an extension that takes away the need to do this at the application level. The code does all the splitting and merging for you so you can write a program that acts like the file is on a file system that supports files > 4GB.
jaclaz|3 years ago
If I may it would make more sense (to me at least) to use a directory and have a descriptor file, not entirely unlike multi-part vmdk's are implemented.
netsharc|3 years ago
> de facto file naming convention
From a first look, like it's using Microsoft's own hack of long file names[1] to create file entries that look like they belong to 1 file. A file that has a long file name (more than the 8+3 character limit) is actually several file entries, but they're empty files. Seems like the tool is creating non-empty files instead, that Windows is chaining together as one.
[1] https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#...
sampa|3 years ago
bArray|3 years ago
Regarding the format, once you convert it, does the target device need to have a driver to support the format? It mentions that this would allow for > 4GB files for TVs, but these are typically non-updated very out of date OSes.
I think MS missed a trick by not making the boot sector also contain a simplistic driver, although it would have been a push to keep it all down at 512 bytes.
Dylan16807|3 years ago
It's simple enough to work on basically anything, but for what purpose?
The max file size on FAT12/16 is the same as the max drive size.
And FAT32 is very easy to implement for any system dealing with multiple megabytes.
pantalaimon|3 years ago
JAA1337|3 years ago
I love seeing this, don't get me wrong. I am just curious is there are any real world applications for this?
mort96|3 years ago
noAnswer|3 years ago
Straight from their FAQ: We see emFile customers asking for solutions for bigger files. Implementing exFAT is not an option for us, as it is patent encumbered. SEGGER would need Microsoft's permission to implement and offer it, and our customers need to deal with Microsoft again to be able to use it in their products. This can be time-consuming and also expensive. We feel there should be a free alternative. The more popular BigFAT becomes, the better.
I guess using anything but FAT would make it hard for their developer base.
Bakary|3 years ago
This would be extremely niche but would have its audience. Heck, I wouldn't be surprised if HN readers adopted it just to taste the thrill of unlimited and obscene power.
unknown|3 years ago
[deleted]
scohesc|3 years ago
I'm very ignorant to this but I'd love some insight from someone vastly more knowledgeable than me.
tenebrisalietum|3 years ago
The I/O API or "commands" are the same; different filesystems will implement it differently.
steeleduncan|3 years ago
stuaxo|3 years ago
tumetab1|3 years ago
quickthrower2|3 years ago
And what about converting FAT32 to a linux partition? Or buy a new disk and move data over to that.
Edit: it is a genuine question. downvote implies not but honestly it is.
tekchip|3 years ago
tzahifadida|3 years ago