[qz_kb]

How the hell does the Seek by iNaturalist app work so well and also be small/performant enough to the job completely offline on a phone? You should really try it out for IDing animals and plants if you haven't, it's like a real life pokedex. Have they released any information (e.g. a whitepaper?) about how the model works or how it was trained? The ability to classify things incrementally and phylogenetically makes it helpful to narrow down your own search even when it doesn't know the exact species. I've been surprised by it even IDing the insects that made specific galls on random leaves or plants.

[fogleman]

I reverse engineered their stuff a bit. I downloaded their Android APK and found a tensorflow lite model inside. I found that it accepts 299x299px RGB input and spits out probabilities/scores for about 25,000 species. The phylogenetic ranking is performed separately (outside of the model) based on thresholds (if it isn't confident enough about any species, it seems to only provide genus, family, etc.) They just have a CSV file that defines the taxonomic ranks of each species.

I use it to automatically tag pictures that I take. I took up bird photography a few years ago and it's become a very serious hobby. I just run my Python script (which wraps their TF model) and it extracts JPG thumbnails from my RAW photos, automatically crops them based on EXIF data (regarding the focus point and the focus distance) and then feeds it into the model. This cropping was critical - I can't just throw the model a downsampled 45 megapixel image straight from the camera, usually the subject is too small in the frame. I store the results in a sqlite database. So now I can quickly pull up all photos of a given species, and even sort them by other EXIF values like focus distance. I pipe the results of arbitrary sqlite queries into my own custom RAW photo viewer and I can quickly browse the photos. (e.g. "Show me all Green Heron photos sorted by focus distance.") The species identification results aren't perfect, but they are very good. And I store the score in sql too, so I can know how confident the model was.

One cool thing was that it revealed that I had photographed a Blackpoll Warbler in 2020 when I was a new and budding birder. I didn't think I had ever seen one. But I saw it listed in the program results, and was able to confirm by revisiting the photo.

I don't know if they've changed anything recently. Judging by some of their code on GitHub, it looked like they were also working on considering location when determining species, but the model I found doesn't seem to do that.

I can't tell you anything about how the model was actually trained, but this information may still be useful in understanding how the app operates.

Of course, I haven't published any of this code because the model isn't my own work.

[murphyslab]

I don't use Seek, but the iNaturalist website filters computer vision matches using a "Seen Nearby" feature:

> The “Seen Nearby” label on the computer vision suggestions indicates that there is a Research Grade observation, or an observation that would be research grade if it wasn't captive, of that taxon that is:

> - within nine 1-degree grid cells in around the observation's coordinates and

> - observed around that time of year (in a three calendar month range, in any year).

https://www.inaturalist.org/pages/help#computer-vision

For how the model was trained, it's fairly well documented on the blog, including different platforms used as well as changes in training techniques. Previously the model was updated twice per year, as it required several months to train. For the past year they've been operating on a transfer learning method, so the model is trained on the images then updated, roughly once each month, to reflect changes in taxa. The v2.0 model was trained on 60,000 taxa and 30 million photos. There are far more taxa on iNaturalist, however there is a threshold of ~100 observations before a new species is included in the model.

https://www.inaturalist.org/blog/83370-a-new-computer-vision...

https://www.inaturalist.org/blog/75633-a-new-computer-vision...

[a_bonobo]

>It looked like they were also working on considering location when determining species, but the model I use doesn't do that.

I do this in fish for very different work and there's a good chance the model for your species does not exist yet. For fish we have 6,000 distribution models based on sightings (aquamaps.org) but there are at least 20,000 species. These models have levels of certainty from 'expert had a look and fixed it slightly manually' to 'automatically made based on just three sightings' to 'no model as we don't have great sightings data'. So it may be that the model uses location, just not for the species you have?

[xigency]

Sounds like a real-life Pokémon Snap. You should add a digital professor who gives you points based on how good your recent photos are. (Size of subject in photo, focus, in-frame, and if the animal is doing something interesting.)

[datadrivenangel]

That sounds like an awesome setup! Would you be willing to share your script with another bird photography enthusiast?

[jamesash]

Comments like these are why I lurk on HN. Genius solution.

As a birder I have thousands of bird photos and would pay for this service.

[bombcar]

This post fits the username perfectly.

[komali2]

You wrote your own custom RAW photo viewer? Like, including parsing? That's incredibly cool, do you share it anywhere?

Also why not just darktable / digikam?

[kooshball]

I would pay for this

[kubrickslair]

Thanks for sharing - I was curious too but didn’t delve in myself.

[kooshball]

if you're willing it's totally fine to share your work with the model itself removed

[xipho]

https://openaccess.thecvf.com/content_cvpr_2018/html/Van_Hor...

https://openaccess.thecvf.com/content/CVPR2021/html/Van_Horn...

Etc.

[rahimnathwani]

I'd never heard of this app, but your description made me want to install it. When I googled it I was surprised at the app ratings:

Apple: 4.8

Google Play: 3.4

The most common issue mentioned by negative Play store reviews is the camera not focusing on the right thing, and needing to try many different angles before something is recognized correctly. This is probably nothing to do with the underlying model, which I guess is the same on both platforms.

[Taek]

Camera zoom is definitely annoying, there's no way to control how zoomed in it is.

And yes, it often takes as much as a minute to identify a species, because you have to keep adjusting zoom and angle and trying to catch every important feature.

That said, once you are used to it, it becomes less noticeable and just feels like part of the game.

[techterrier]

our own app Birda has this issue too, most of the 1 star reviews are 'not the app I was looking for' or 'not a game'

[bandergirl]

I wish I had the same experience as you. The vast majority of time I point at tree leaves in South East Asia it tells me it’s Dicots and it stops there. Only rarely I get the actual full classification; the last time it happened was for a common almond tree.

[burkaman]

It's very bad at trees for some reason. Also mushrooms, but I thought that might be intentional so they don't get blamed for someone eating something poisonous that was misidentified.

PlantNet often works better for trees.

[contingencies]

For trees try to photograph the flowers, the seeds, the bark, the leaves (both sides), the trunk growth habit (especially bottom portion), and the upper branches growth habit. Often when asking it to suggest a species switching between these will create progress.

[EGreg]

Probably because many people around the world participated in classifying what was posted?

I am guessing. Please tell me if that is correct. How do they prevent false labels ?

[nemo]

Any observations can be submitted, but the observation has to be verified by a different observer. Most identifiers are folks with more experience identifying things locally, and the data quality is high. There's very little incentive to game the system and if something is misidentified other iNatters can also add identifications correcting mistakes which happens regularly - often various scientists/specialists tend to sweep observations in their taxa of note and correct issues. There's criteria for a "high quality" observation, including being verified. Only those observations that are high quality are used for training.

[xipho]

There are hundreds of thousands of "false' labels. Pictures can be classified many times.