Show HN: I built an AI that recognizes food

[AJRF]

Something like this would be a fantastic addition to MyFitnessPal.

I find myself sometimes not logging food on that app because I couldn't be bothered to search for it.

On the other hand if the product has a barcode I always scan it in because that is convenient.

[subcosmos]

Same - its one of the reasons I wanted to make my own. That and data export functionality, which is requisite for research.

[jcoffland]

If you're eatting a lot of food with barcodes on it then your doing it wrong.

[nathan_f77]

I would pay something like $5 per day for this if it was a real person who looks at my photos, categorizes them accurately, and enters them into MyFitnessPal.

I won't trust an AI until it has a massive database and a million users. Until then, it's just not going to work well enough. You also need to convince users why they should use your app, instead of MyFitnessPal (which already has a massive database, including barcodes.)

[HamSession]

I agree with your suggestions for human assessment because of problem posed by food calories varying significantly due to different preparations.

Take for example soda, if I gave you a picture of soda in a glass could you tell it was diet or regular? You might scoff at such an edge case but it quickly becomes more common when looking into food perpetration techniques. This is why caloric estimation is a really difficult and causes restaurants to not list their calories as the calories of a meal do not equal the sum of it's parts.

All these solutions are common as people want a signal to tell them to stop eating, but these are insufficient as people will simply ignore it due to hunger cravings (as happens on diets). Any nutritionist service in addition to detailing calories would need to incentive the patient to recognize the need to lose weight or setup a helpline.

[logn]

It would be difficult to account properly for ingredients such as butter and oil which are hard to see in a finished dish and drastically affect the calories.

But people would probably still pay as long as it's a good faith attempt, since they can't necessarily do better themselves, so it would be just as accurate and save time.

And with geolocation, you could start to figure out where people are and then you'd know exactly the calories if they're at a chain restaurant.

[subcosmos]

I have considered something like this as a business model, although I was thinking of nutrition coaching instead of mere logging itself, but thats actually a really great idea.

[a_small_island]

Impossible. How can someone tell whether you cooked with butter and oil, for instance?

[subcosmos]

Built using those newfangled convolutional neural network things. Trained caffe on a set of about a million food related images. Can't believe how accurate the classifier is...

UI needs work. Would love feedback on how to make food logging more intuitive.

[swah]

Your app kinda sucks now, but the detection is great. You could also do food identification as a service...

I'm trying to use fatsecret/myfitnesspal for counting calories. Something like this could make it much easier!

To me, a better workflow would be: take pictures of everything you eat and classify later (at night, maybe on the PC). Of course, this software would simplify the classification, the user would only fix any errors and adjust sizes.

This workflow would be well suited to your app, which operates on a server? (my understanding).

Congrats, I understand the relevance of AI for simple apps now ;)

[AJRF]

You could probably outsource enough of the UI work to Fiverr to make a really decent looking app.

Things like Main screen, App Icon and colour palette.

[zopf]

Where'd you get your images? Food-101 plus something else?

We're working on a similar component for our commercial behavioral-economics-driven app suite, targeted toward patients with chronic diseases.

Do you use location as a way to filter the set of possible foods, as in Google's im2calories project/paper last year?

They do some pretty awesome depth calculation stuff too: https://www.google.com/?ion=1&espv=2#q=im2calories+type:pdf

Edit: oops, should have read further. I see that you pulled appropriate terms from WordNet and used ImageNet to gather training images from Flickr. Cool!

[Keats]

I would love a blog post about that!

[teLeopardthy]

Any info on the network itself (number of layers, input resolution, pooling, dropout etc)?

[krashidov]

How much experience with machine learning did you have before you started this project?

[sandGorgon]

why caffe ? why not something like h2o or tensorflow ? just curious.

[imacomputer2]

Login via Facebook? Come on! See you next app.

[koolba]

Someone should combine this AI with something like Google glass (so you don't have to pull out your phone), a basic calorie/nutrition filter, and a shock bracelet[1].

That way when I look at a Krispy Kreme donut I get immediate feedback!

[1]: http://pavlok.com/hello.php

[internaut]

I'm dying here. Sign me up for your startup.

Let's call it: "American Dystopia"

[misiti3780]

google glass is dead though right ?

[subcosmos]

I absolutely LOVE this idea .... :)

[solaxun]

I've been thinking of this exact thing for a few years now, good to see others are interested too. I think there is HUGE potential for an app that can accurately (within a reasonable error range) count macro-nutrients through pictures of food, but I think that is WAY more difficult than you would imagine. How can you train a classifier to tell how much butter is in a dish? How can you account for food that is literally underneath other food?

Maybe a combination of spectrometer, weight, and photos would be a future solution. I certainly don't have an idea of how it would be implemented, but I think using data from all 3 of those sources could be a good step.

[subcosmos]

Another angle is post-consumption. Ive always wondered if precisely monitoring body temperature might tell you something about consumption, since there is this 'thermic effect of food'. Would be hard to get right though ...

In the lab, we can use doubly-labelled water to precisely measure calorie intake and burn, but its so damned expensive. If only it were commercially viable.

[holografix]

This has HUGE potential. Some of the comments here disregard the fact that AI can do a better job than humans. An expert chef looking at dishes will make an educated guess on the ingredients and quantities that went into a dish and will have a better idea of the caloric content. The AI could learn a similar skill, identifying whether a dish contains pastry, buttery sauces, fatty meat etc etc.

[nickpsecurity]

You have a business opportunity here if you train that on fresh vegetables. Many people have trouble with these issues:

1. Knowing what the hell a (example) rhubarb is or looks like.

2. Knowing if it looks fresh or about to go bad.

Being able to aim a camera slowly across an isle with it getting highlighted would solve first problem. Spinning vegetable around in front of the camera would could solve second. So, try that and make it a paid app.

[subcosmos]

You know, its funny. being a former fat guy, Im not entirely familiar with all of my vegetables. I took this app to a farmers market recently and it was actually helping me identify a few things ...

Had never seen a beet before un-cut. Ashamed of my ignorance ;)

[zeveb]

Pretty nifty, but please consider allowing creation of an account with a simple email address, or even anonymously. I really don't want to have to connect my Facebook or other social-networking information to your app in order to use it.

Thanks for allowing one to try it out without logging in via another service; it'd be great to be able to actually have an account.

[subcosmos]

Email accounts are definitely on the long todo list.

[aembleton]

This is not compatible with my device which is an LeTV X500, running Android 5.0.2. Is there any reason that it might not work? I'm in the UK, if that might be the reason.

[subcosmos]

Im new to cordova (the toolkit I used to build the app), and I did notice that the number of supported devices is kind of low for my build. I may have included a plugin that narrows down the pool. I need to dig into this.

Sorry about this!

[rymate1234]

This happening for me as well with a Moto X Play running Android 6.0.2

Looking forward to trying the app once I can install it though, looks neat

[lj3]

This is happening for me too. Samsung Galaxy S3 on US ATT running Android 4.4.4 Cyanogenmod 11.

:(

[gchristen]

Hey Subcomos, great work! We've been working on something pretty similar, an app called Logameal (only available in the UK). We have run into (are running into) a lot of those UI issues too and would be glad to help. Let me know if you'd like to chat.

[aantix]

I remember the latest Android camera app being able to extrapolate distances in the app (thus simulating photos with large apertures/shallow depth of field).

I wonder if you could use this visual distance algorithm to somehow judge portion sizes/plate sizes?

[micro_cam]

Nice work! Another big user group of stuff like this is diabetics who need carb estimates to adjust insulin doses.

[subcosmos]

I earned my PhD in diabetes research at the Mayo clinic. My whole goal for this project is to cure that damned illness.

Aim high ...

[franciscop]

Isn't it strange that in [1] the beginning of the "Normal" BMI has more risk of death than the beginning of the "Obese"? I'd even venture to say that the "Overweight" range has, in average, lower Mortality Risk than the "Normal".

PS, another plus for the Metric System. The Imperial System form includes Feet+Inches, while the metric one only needs Centimeters for the height.

[1] https://www.infino.me/join/

[virtualritz]

First thing I tried: logging /two/ Granny Smith apples manually.

I had do the entire process twice because there is no (obvious) way to change the serving size to 2.

[subcosmos]

If you click the 'no barcode' button to pull down a fatsecret nutrition record, there is a servings pulldown that is displayed. Its kinda hard to see. Working on the CSS

[zaroth]

Please do blog about the technical details, if you could post a walkthrough of setting up and training the net it would be great!

[mark_l_watson]

That is very cool subcosmos! I am also very interested in both cooking+nutrition and AI (see for example my web site cookingspace.com). My contact information is in my HN profile - shoot me an email if you want to talk about AI, food technology and business, etc.

[Zolomon]

This should be in use at those self-registry machines at food markets. I imagine that you place the food item on the scanner and it will automatically register it or ask for human input when something unrecognized is found.

Would decrease cheating perhaps?

[sandGorgon]

Where did you get your training data set? I'm trying to build a network to recognize faces from ID cards and match them to faces/selfies. Onenof the hard things has been to get a training data set.

[zodPod]

This sounds like it could be used for the wrong purpose for sure.. Are you working for some nosy employer?

[devstack]

That's very interesting indeed. Are you using any API?

[mrfusion]

I think something like this could get really good at estimating calories too. I'm saying within +\- 50 calories.