This seems to store the zfs send stream. That's a bad idea.
> Incremental ZFS send streams do not have any of these properties and full ZFS send streams only have a few of them. Neither full nor incremental streams have any resilience against damage to the stream; a stream is either entirely intact or it's useless. Neither has selective restores or readily available indexes. Incremental streams are completely useless without everything they're based on. All of these issues will sooner or later cause you pain if you use ZFS streams as a backup format.
If you’re looking to ZFS send streams for many of ZFS’s guarantees, absolutely.
However ZFS replication was originally designed with the assumption and use case in mind that organizations would store the send streams as opaque blobs on tape. This is, in part, why storing send streams as blobs is still a thing people do.
There are some use cases where this makes sense. I’ve stored full send streams of archived ZFS file systems in S3(-compatible services) where integrity is handled at the platform level. In that use case I didn’t benefit from having every copy of the filesystem in question running on live storage media, and incremental sends/snapshots weren’t on the table. (I also SHA checksummed the resulting files, and did restore tests.)
There is also a misconception that frequently gets brought up in the ZFS community that the send stream format isn’t stable between versions and cannot be relied upon in future, but it absolutely is stable. In fact the ZoL manpage for send explicitly states that it is. As with anything in ZFS though, you want to move versions forward or not at all, rather than backward.
Finally! Been looking for this a long time. File-based backups for large Pg databases are not very scalable (even incremental), having this in my toolkit would be amazing.
I'm not sure I follow. Wouldn't this be file-based (zfs-dataset-based) incremental backups? I don't think this has anything to do with postgresql other than copying the style of pgBackRest.
ylyn|6 months ago
> Incremental ZFS send streams do not have any of these properties and full ZFS send streams only have a few of them. Neither full nor incremental streams have any resilience against damage to the stream; a stream is either entirely intact or it's useless. Neither has selective restores or readily available indexes. Incremental streams are completely useless without everything they're based on. All of these issues will sooner or later cause you pain if you use ZFS streams as a backup format.
https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSSendNotA...
TheNewsIsHere|6 months ago
However ZFS replication was originally designed with the assumption and use case in mind that organizations would store the send streams as opaque blobs on tape. This is, in part, why storing send streams as blobs is still a thing people do.
There are some use cases where this makes sense. I’ve stored full send streams of archived ZFS file systems in S3(-compatible services) where integrity is handled at the platform level. In that use case I didn’t benefit from having every copy of the filesystem in question running on live storage media, and incremental sends/snapshots weren’t on the table. (I also SHA checksummed the resulting files, and did restore tests.)
There is also a misconception that frequently gets brought up in the ZFS community that the send stream format isn’t stable between versions and cannot be relied upon in future, but it absolutely is stable. In fact the ZoL manpage for send explicitly states that it is. As with anything in ZFS though, you want to move versions forward or not at all, rather than backward.
levkk|6 months ago
craftkiller|6 months ago