[zeotroph]

That only seems to work on btrfs, XFS, (and maybe now or very soon) ZFS and bcachefs: "[duperemove] simply finds candidates for dedupe and submits them to the Linux kernel FIDEDUPERANGE ioctl." [1] (aka BTRFS_IOC_FILE_EXTENT_SAME), and this ioctl "performs the 'compare and share if identical'" (and locking etc.) work [2]. But on those filesystems, that is a nice feature, plus it lets the tool get away with a weak hash like murmur3.

1: http://markfasheh.github.io/duperemove/duperemove.html 2: https://manpages.debian.org/bookworm/manpages-dev/ioctl_fide...

[Arnavion]

A fallback to hardlinking would work for a lot more filesystems, though of course it would be limited to files that are completely identical rather than any identical extents / blocks within files.

[gavinsyancey]

More importantly, hardlinking means writes to one file would affect both, while FIDUPERANGE creates copy-on-write semantics.

[JackSlateur]

Quick note : I removed murmur3 support, and replaced it with xxhash