[HowardStark]

Curious if anyone has done anything meaningful with SAM2 and streaming. SAM3 has built-in streaming support which is very exciting.

I’ve seen versions where people use an in-memory FS to write frames of stream with SAM2. Maybe that is good enough?

[tom-in-july]

The native support for streaming in SAM3 is awesome. Especially since it should also remove some of the memory accumulation for long sequences.

I used SAM2 for tracking tumors in real-time MRI images. With the default SAM2 and loading images from the da, we could only process videos with 10^2 - 10^3 frames before running out of memory.

By developing/adapting a custom version (1) based on a modified implementation with real (almost) stateless streaming (2) we were able to increase that to 10^5 frames. While this was enough for our purposes, I spend way too much time debugging/investigating tiny differences between SAM2 versions. So it’s great that the canonical version now supports streaming as well.

(Side note: I also know of people using SAM2 for real-time ultrasound imaging.)

1 https://github.com/LMUK-RADONC-PHYS-RES/mrgrt-target-localiz...

2 https://github.com/Gy920/segment-anything-2-real-time