[diftraku]

While I have not been this deep into the inner workins, I personally have skimmed that rabbit hole when I started a side-project that was essentially grep but for (mainly) MKV files.

Idea was that you could "grep" by specific text in the subtitles and automatically create a clip of every occurrence of the text (by looking at the subtitle timing and padding that in both directions).

The biggest source of my frustration was that I was unable to get the clipping to work exactly as I wanted, where the start or end of the clip would seemingly drift back and forth. That was until I realized it boiled down to how the different seek modes in ffmpeg handled keyframes.

I still haven't gotten the clipping to work exactly as I want but I figured doing two passes might be the way to go: first pass would do a fuzzy match and ensure there is enough extra on both ends of the desired clip and the second pass could re-encode the fuzzy-matched clip to shuffle the keyframes around, allowing more accurate clipping.

[IanCal]

It's tricky. Subs aren't always brilliantly timed either.

I did something similar many moons ago to auto create summary videos based on changing sentiment in the subtitles for the BBC. Worked... interestingly.

If you want to nail scene changes, one thing you can do is look for sudden changes to the histogram frame by frame. It'll change pretty smoothly as people move, cameras move etc but there's a discontinuity when there's a cut. One issue though then is that there are a lot of camera cuts! Surprising how many there are that you don't really notice.