[nexo-v1]

I really like this idea too. Generators are one of my favorite parts of Python — super memory efficient, and great for chaining transformations. But in practice, I’ve found they can get hard to reason about, especially when you defer evaluation too much. Debugging gets tricky because you can’t easily inspect intermediate states.

When working with other engineers, I’ve learned to be careful: sometimes it’s better to just materialize things into a list for clarity, even if it’s less “elegant” on paper.

There’s a real balance between cleverness and maintainability here.

[pletnes]

There’s some stuff in itertools to cut sequences into batches. Could be a useful intermediate step - grab 100 things at a time and write functions that receive and emit lists, rather than generators all the way down.

[UltraSane]

I often will intentionally store intermediate values in variables rather than just doing a clever one liner in Python specifically because I know it will make debugging easier.

[ankitml]

It is possible to test the chaining though, if you know your data well. If not, those edge cases in the data quality can throw things off balance very easily.