Show HN: Convert screenshots of equations to LaTeX

[brownbat]

Immediately made me think of a question on the state of handwriting to LaTeX from a few years ago, and all the massive challenges involved:

https://tex.stackexchange.com/questions/1443/what-is-the-sta...

Under API... you're already doing handwriting? This is uh, nontrivial work to say the least. Really impressive.

The endorsements are a nice touch. :)

Made me really curious how far the system goes, what cases break it.

Oh... nevermind. You have a PDF of examples here: https://docs.mathpix.com

It's honing in on equations without getting distracted by nearby Hanzi or Cyrillic, or even pictures of dogs. Wow.

I keep going back to dig through your resources and getting more impressed.

EDIT: I guess my only constructive criticism is that you should brag more. I like a simple landing page, but I think you've earned a short list of examples of corner cases you tackle well, if the whole API is packed into that free app, because they're really impressive.

[yorwba]

The perfection in those examples makes me suspect that they are cherry-picked or part of the training data. Especially the handwritten text is not always clear and could reasonably be interpreted differently. I'd expect a machine-learning model to get at least some things wrong some of the time.

If I wanted to use this in an application, I'd definitely want to see some accuracy figures on validation data as well as a few failure cases to see whether the output remains reasonable even when it is wrong.

[sinab]

This is really awesome OP! Thank you for sharing :)

One note I should make: it was not entirely clear (to me) upon a cursory view of the website, that the purpose of mathpix was to convert handwritten text into LaTeX. For some reason (maybe my coffee hasn't kicked in yet) I thought this was strictly intended to take screenshots of equations on an existing pdf document or a website etc and that will be converted to LaTeX.

My thought at that point was "I wonder if they could do this for handwritten text" and then I looked at the docs and facepalmed..

[cup-of-tea]

"Screenshot" is a very odd word to use in this context.

[CJefferson]

Stupid question, how well would this work on a PDF of a latex document?

This would be great for blind people, as pdfed latex is extremely non-accessable, and I have to email authors of papers to get the original latex from them, which is often lost.

[tgb]

As someone who writes with Latex, thanks for pointing out this problem, I hadn't considered it.

[froindt]

> This would be great for blind people, as pdfed latex is extremely non-accessable, and I have to email authors of papers to get the original latex from them, which is often lost.

Many years ago I "translated" course materials into a form which was accessible to a blind grad student. It was a really interesting job and taught me a lot about accessibility.

I was effectively doing latex, but without all the leading \ characters. It made learning latex comparatively easy.

What interface do you use to read equations? Screen reader speaking the straight latex, or do you have some Middleware to make it more digestible when listened to?

[JBorrow]

Isn't that the whole point of this utility? Or am I missing something?

[ocrcustomserver]

It's not a stupid question. You need equation detection + equation OCR.

Mathpix only does the equation OCR part.

I've worked on this (for a PDF to HTML application), mail is in profile if you're interested.

[jimnotgym]

Im curious if the developers are fans of The Big Bang Theory TV series? They were using a smart phone app...and of course was less useful due to it being fiction...

https://www.springfieldspringfield.co.uk/view_episode_script...

[saganus]

Wow!

What kind of sorcery is this!?

Is this using deep learning or "regular" OpenCV or similar?

I would assume it's a highly tuned deep learning algo, but I'm not knowledgeable enough to distinguish a deep learning algo from a pile of rocks...

Edit: Aha, someone already asked this and got an answer.

https://news.ycombinator.com/item?id=16535467

[typon]

Suggestion: instead of making me download a pdf to see examples of what the results look like, maybe put them on the page directly. You can have a couple. Then put the details in the pdf.

Great software otherwise

[bagrow]

This is fantastic!!

Bug report: it appears that multiline summation subscripts are not recognized correctly. For example, Eq. 8 of [1]. These are often created using \substack as part of amsmath.

Awesome tool!

[1]: https://arxiv.org/pdf/1802.01194.pdf

[nicodjimenez]

Good catch!!! We're working on it, should be fixed by April 1st

[sitkack]

I assume you just got a lot of installs from India, because the large publishing houses contract out many re-typesetting jobs that are basically to take scans of technical texts and convert them back into LaTeX.

I strongly suggest you talk to the publishers about integrating your tech into their TeX.

[RhysU]

Want a math-ish PDF and some LaTeX source for training on possible edge cases? Think I might get someone (or something) to read my dissertation this way...

[skiman10]

What's your dissertation about?

[nicodjimenez]

yes please! [email protected]

[lowglow]

This is awesome work. What's the process of vision -> latex conversion?

[srush]

We published some research on one approach:

http://lstm.seas.harvard.edu/latex/

Here's how to do it with OpenNMT/PyTorch:

http://opennmt.net/OpenNMT-py/im2text.html

[vinni2]

I am curious to know as well.

[aashu_dwivedi]

Yup! Please!!

[Uninen]

I really appeciate the testimonials. All software deserves personality and a soul like this!

[lliiffee]

Any way you could make this available outside the Mac App store? Apple seems to have decided I did something horrible and unforgivable by moving to a different country after creating an account, thus making it impossible for me to use the store.

[nicodjimenez]

Good to know, thanks for posting this. We'll post the dmg file on our landing page this week or next.

[Ninn]

Awesome! Is the plan for it to be free forever, or what might the pricing look like? Maybe you would consider open sourcing the model?

Also it would be nice of some info on the process. Does work entirely locally, or is images uploaded to the cloud?

[howToLearnSpark]

from the site: $0.005 per request, first 1000 requests are free

[simonramstedt]

We used their API to make a simple screenshot2latex tool (select screen region -> puts latex formula in clipboard). From my experience it still fails on a couple of fairly common things like:

- \mathcal letters (recognized as non-mathcal)

- long equations (not recognized at all)

- multi-line equations (not recognized at all)

The screenshot2latex tool: https://github.com/rmst/screenshot2latex/blob/master/scripts...

[nicodjimenez]

All three points have seen big improvements in the last week, especially the first too, check again

[screye]

That's aweson OP. I can't imagine the number of times I've wished for something like this.

Coming from a grad student who hates writing equations in latex. I will probably try this out.

[_emacsomancer_]

Is there any chance of a Linux client?

[xfer]

They have an api. You can write your own.

[shafyy]

I was looking for an API that provides math OCR. Great, going to integrate it into our app soon :-) Let me know if you want to add us to your "trusted by" section.

[nicodjimenez]

sure! send the link to [email protected]

[andreareina]

Impressive. Any particular reason that you're using

    \left\{ \begin{array} ... \end{array} \right

instead of

    \begin{cases} ... \end{cases}

?

[nicodjimenez]

No particular reason, we were thinking to start returning cases syntax soon!

[amelius]

Mathematicians use operator overloading all the time. It would be nice to have a tool that explains to me what an equation actually means in a given context.

[ktpsns]

You are talking about the semantics of an equation while this tool is already satisfying when understanding correctly the syntax (in LaTeX).

There are actually a number of ongoing research projects to establish standards of semantical mathematical representations. Probably one of the best funded running projects (budget ~10MEUR) which has a work package on this topic is http://opendreamkit.org/ . Work is going on at https://mathhub.info/ from my knowledge. I would like to provide a deep link but the site seems to be in a broken state. Apparently people are working on it right in the moment.

[taeric]

Isn't that usually what the document is for? :)

The verbalization if most LaTeX commands can help learn to read the equations. Sometimes.

[dcchambers]

This is insanely impressive. Great work. Wish tools like this existed when I was still in school...almost makes me want to go back and do some more math :)