top | item 46771835

(no title)

arseniibr | 1 month ago

I agree that fixing the pipeline is indeed the correct decision, but I've created this tool to provide the detection.

In a complex environment, you often don't control the upstream ingestion methods used by every team. They might use git lfs, wget, huggingface-cli, or custom caching layers.

Relying solely on the hope that every downstream consumer correctly handles Git LFS is dangerous. This tool acts as a detector to catch those inevitable human or tooling errors before they crash the production.

discuss

order

embedding-shape|1 month ago

> This tool acts as a detector to catch those inevitable human or tooling errors before they crash the production.

Again, that sounds like a bigger issue, that a repository using Git LFS can somehow "crash the production", that's where I'd add resilience first. But as mentioned in another comment, I don't have the full view of your infrastructure, maybe it has to work like that for whatever reason, so YMMV.