I created a basic similar app for my own use because I wanted to have an idea what people are conversing in russian in online lobbies. WPF, Tesseract OCR and Microsoft's translation API.
A fun side project that I do end up using a bit. Gonna bind the capture to some hotkey so I can use it without changing app focus. Most annoying problem though is that Tesseract OCR often gets confused when you make it read combined latin+cyrillic letters and the font isn't something Tesseract prefers. Especially when there's something behind the text. Kind of disappointed that the most popular API often has a lot worse results than a human would just transcribing the letters.
Wouldn't be surprised if OCR software would leap soon due to a product similar to Whisper.
Comes to mind that the best possible app that does this would be kind of like the old "word lens" iPhone application but on all screens, meaning it would replace text from the raw screen input with text of another language, while keeping the appearance/color/scale/rotation of the original text. This would free it from needing to be built-in to whatever UI library is producing the text, and would work on recorded video too. Immediate latency/performance problems come to mind though but could be a fun thing to try.
It is very nice to have free alternatives to this kind of software. I wanted to ask because I already use terassect, the most basic feature I want in this type of software is that I want to be able to edit the text on the image I screenshot in order to use it while guiding designers, especially in design. I think teras supports this feature, but I have not seen it actively in any project other than Project Naptha, which is not an actively developed project in this regard. I would like to hear if there is a project you know about this and want to share.
Tesseract might be "not very good" but it is still state-of-the-art, often available, with many languages supported.
The special sauce - what you need to get a better result - is good, adaptive thresholding (something more advanced that raw naive binary thresholding you get feeding naive color/grayscale images to OCR).
As far as I know, once you get that nailed it doesn't matter that much what OCR you use - as long as it's available and supports your target language.
As others mentioned, Tesseract is SOTA in FOSS OCR. It also still is being developed, improving slow but constantly.
The main issue for a use-case like NormCap are the trained models: they are optimized for images of _printed_ text and layouts, which is different from on-screen-text in many aspects. Unfortunately, I don't have the resources to train my own models.
Cuneiform was a long time competitor, but afaik development there is stalled.
[+] [-] maxlin|2 years ago|reply
https://streamable.com/ykng5u
A fun side project that I do end up using a bit. Gonna bind the capture to some hotkey so I can use it without changing app focus. Most annoying problem though is that Tesseract OCR often gets confused when you make it read combined latin+cyrillic letters and the font isn't something Tesseract prefers. Especially when there's something behind the text. Kind of disappointed that the most popular API often has a lot worse results than a human would just transcribing the letters.
Wouldn't be surprised if OCR software would leap soon due to a product similar to Whisper.
Comes to mind that the best possible app that does this would be kind of like the old "word lens" iPhone application but on all screens, meaning it would replace text from the raw screen input with text of another language, while keeping the appearance/color/scale/rotation of the original text. This would free it from needing to be built-in to whatever UI library is producing the text, and would work on recorded video too. Immediate latency/performance problems come to mind though but could be a fun thing to try.
[+] [-] sitkack|2 years ago|reply
Visual Universal Translator.
[+] [-] bjoli|2 years ago|reply
I will definitely use this.
[+] [-] bigmattystyles|2 years ago|reply
https://learn.microsoft.com/en-us/windows/powertoys/text-ext...
[+] [-] stavros|2 years ago|reply
[+] [-] quelltext|2 years ago|reply
[+] [-] yigitkonur35|2 years ago|reply
[+] [-] csdvrx|2 years ago|reply
It's not very good. I miss being able to copy/paste from blurry or deformed screenshots of youtube on Windows.
[+] [-] ElectricalUnion|2 years ago|reply
The special sauce - what you need to get a better result - is good, adaptive thresholding (something more advanced that raw naive binary thresholding you get feeding naive color/grayscale images to OCR).
As far as I know, once you get that nailed it doesn't matter that much what OCR you use - as long as it's available and supports your target language.
[+] [-] holbue|2 years ago|reply
The main issue for a use-case like NormCap are the trained models: they are optimized for images of _printed_ text and layouts, which is different from on-screen-text in many aspects. Unfortunately, I don't have the resources to train my own models.
Cuneiform was a long time competitor, but afaik development there is stalled.
[+] [-] m-p-3|2 years ago|reply
[+] [-] nicodjimenez|2 years ago|reply
[+] [-] denimboy|2 years ago|reply
[+] [-] holbue|2 years ago|reply
PS: People looking for (FOSS) alternatives, look here: https://github.com/dynobo/normcap#similar-open-source-tools
[+] [-] tough|2 years ago|reply
https://github.com/schappim/macOCR
Just rediscovered the Shortcuts a couple days ago while installing it on a friend's mac.
[+] [-] villgax|2 years ago|reply