Our intern worked on a similar DSPy project last year for contracts, though the classifier was much simpler: binary classification of whether the line was the start of a new section or not. We saw some success, but it wasn’t a revelation, and in the end we ended up keeping our giant regex classifier since it was much faster and much cheaper (free). I do really like DSPy though - maybe looking again with a more well defined set of classes would work better.
binarymax|6 months ago