top | item 8430412

Google Now vs. Siri vs. Cortana – The Great Knowledge Box Showdown

298 points| nreece | 11 years ago |stonetemple.com

168 comments

order
[+] escapologybb|11 years ago|reply
Having used both of them, I really couldn't live without Siri to be honest. I'm quadriplegic, and whilst obviously I wouldn't die without Siri my life is immeasurably better with her.

Some of the killer features of Siri for me are being able to to write emails, send messages and get quick answers to general questions throughout the day. She is much better at the first two than she is at the last one though, but even then she is not too bad.

And now that Apple has hooked her into much more of iOS in iOS 8, it means that more of the OS is open to me than before just by simply using my voice. And the "Hey Siri!" feature is one of the best accessibility features they've added so far.

[+] eddieroger|11 years ago|reply
I never considered using Siri this way, which feels strange to me since I've spent a lot of time thinking about the other Accessibility features of iOS (mostly VoiceOver). Have you ever written up your experiences with iPhone accessibility? I'd be interested to read it.
[+] angersock|11 years ago|reply
I mean this in a very honest way: How are you posting here? Dictation software or something similar?
[+] michaelq|11 years ago|reply
I completely agree. I commute by running, and it's unsafe to look at a screen. So I frequently issue commands into my headphone mic like "read my texts" and "text <name> I'll be there in five" or "shuffle playlist 180 BPM". Many people dismiss Siri on account of mediocre answers to open ended questions, but it's quite effective at carrying out precise instructions.
[+] toxican|11 years ago|reply
I really like the "Hey Siri!" feature, although why it's only usable while the phone is charging is beyond me. I thought maybe battery concerns since it has to be constantly listening, but I wouldn't think it would be that taxing on the battery.

How do you handle typos in iOS as a quadriplegic? They're a pain in the ass to fix by hand and I didn't even know you could do it by voice.

[+] mikeash|11 years ago|reply
We had a quadriplegic friend live with us when I was a kid for several years in the 80s. He spent a lot of time watching TV... there just weren't a lot of options. We had a clapper set up that he could activate by making sounds with his tongue, and a stick he could hold in his mouth to manipulate the remote control, and also do things like make phone calls.

He was a smart guy and was really interested in all sorts of things, but he was sadly limited. Posts like yours remind me of him and make me wonder how he'd get on with modern technology. I do recall investigating speech recognition back in the day, I think it was an early version of Dragon, but the cost was just too high and the uses, with 80s-era computers and no internet or even nearby BBSs to connect to, were just too little. Anyway, thanks for sharing your experience.

[+] nobody_nowhere|11 years ago|reply
Hey, thanks for pointing out the impact of this technology from the perspective of a quadriplegic. It's always eye opening when a technology I really like turns out to be a lifesaver (hopefully not putting words in your mouth) for people with disabilities. Had a similar revelation re amazon home delivery recently. Great to see these technologies making meaningful improvements in peoples' lives.
[+] b2themax|11 years ago|reply
FWIW, I just switched from Windows Phone to iOS -- from the 1020 to iPhone6+. Personally, I preferred Cortana over Siri. And it basically comes down to speed and responsiveness. Cortana is much quicker to launch, and much quicker to find results. I haven't noticed any discernable difference in their query results. And actually, for one of the my most used queries - "What is the weather today?" -- I dont even use it on iOS because the stock weather app is disappointing, I use Yahoo. Cortana telling me the weather was much more usable. Siri is just so freaking slow, that I rarely use her.
[+] at-fates-hands|11 years ago|reply
Agreed on Cortana.

There are some things I feel Cortana does that Siri and Google Now does.

Every morning Cortana gives me the weather, appointments on my calendar, news stories and how long my commute will take given the current traffic conditions. It also learns when I normally leave for work, so about 10 minutes before I usually leave, I get an update on how long it will take to get home given the current conditions.

To me, Cortana is more like an assistant. You can set stuff so its off limits and she will ignore, and likewise, tell her to remember certain reminders like you pointed out.

Do either Google Now or Siri have these features that actively learn stuff about your preferences?

[+] wutbrodo|11 years ago|reply
> Personally, I preferred Cortana over Siri. And it basically comes down to speed and responsiveness. Cortana is much quicker to launch, and much quicker to find results.

I haven't used Cortana, and have only had disappointing experiences with Siri (after coming from Google Voice Search); how does the performance on Google Voice Search Compare? I wonder if the performance rankings are just a complete reversal of the quality rankings; If so, I wonder who's closest to the sweet spot?

[+] bphogan|11 years ago|reply
That's really interesting to me, because my daughter (6yo) asks Siri that question about 10 times a day on the 5S and the answer comes back in about a second. Whether we're on LTE or wifi. I wonder if there's other factors at play.
[+] sytelus|11 years ago|reply
There were not random queries. In fact, they were picked because we felt they were likely to trigger a knowledge panel.

For a good believable test your sampling methodology is very important. In fact, you want to have a random sample that is free of biases instead of somebody cherry picking queries. Perhaps you may do random weighted sample of queries with weights = frequency which represents usage pattern more closely. In any case, it's very important to describe your sampling methodology or otherwise this kind of testing has little value.

[+] apendleton|11 years ago|reply
Out of curiosity, how would you propose taking a random sample of all possible questions?
[+] chetanahuja|11 years ago|reply
Even though the study (albeit informal) did not include personal assistant functionality, that's where Google Now (as manifested in an Android phone connected to a google account) really shines, in that like a real personal assistant, it brings you relevant information before you even ask. Around this time last year, I used to be startled by a timely reminder on the home screen widget about the next thing I was about to do (reminder to leave now for a meeting across town scheduled for a half hour from now, or automagic notification about a flight cancellation etc). But by now, I'm so used to this functionality that I feel slightly annoyed if I actually have to ask for such information (which is almost always a voice query to my phone at this point).
[+] walterbell|11 years ago|reply
Who owns the key patents for voice recognition - AT&T, Nuance, IBM? When will they be expire and be available for use in open-source speech recognition?

Edit: some data at http://www.quora.com/Is-anyone-working-on-an-open-source-ver... & https://news.ycombinator.com/item?id=4987875

[+] syllogism|11 years ago|reply
Actually I think most of the work has been done in academia --- certainly that's where the recent deep learning stuff has come from. So, I don't think the important stuff is patented. (In general it's very hard to lock down ML improvements under patent. Once we can do it one way, and understand a little bit about what's working, we can usually replicate that performance with another technique.)

The big problem for open source speech recognition is training data.

[+] msoad|11 years ago|reply
Some query patterns I use in Google that you might be interested in

    Translate {words} to {language}
    My flights
    My schedule
    My packages
    What time is at {city}?
    How tall/old/heavy is {important person}?
    {description of a photos} in my photos (like 'ocean in my photos')
    Note: if you have G+ photos. description doesn't need to be typed in the photo meta info
    compare {food} and {food} (nutritions)
[+] nolok|11 years ago|reply
time in {city|country}

123 {currency} in {currency}

[+] msoad|11 years ago|reply
alarm for 5 minutes
[+] tbrock|11 years ago|reply
It seemed to me that Cortana returned the most correct answers without any extra fluff. The google results seemed to have the same accuracy but the phone said so much extra information in that monotonic voice that the clarity of the result was lost in the noise.
[+] mmastrac|11 years ago|reply
As an Android user, I've found that Siri tends to be generally better than Google Now. I don't usually ask my phone for trivia, but rather to do something. Most of the time I'll get routed to a search query when I'm looking to command the phone instead.

This is somewhat frustrating if you don't know the magic incantation to make Google Now do what you are asking it to do. Siri's engineers have done a better job anticipating the various forms of the commands and handling nearly all of them.

[+] ark15|11 years ago|reply
I am an Android phone user, past iPhone user and current iPad user and my experience has been the exact opposite. Google Now always understands me better than Siri.

When Siri fails for me, I sometimes ask my 7 year old to talk the same thing to Siri and she gets better results. My daughter has a more 'American' accent than me so I have concluded that Android is better at hearing through accents than iOS.

(I haven't yet read the original article but wanted to quickly comment since our observations are completely opposite)

EDIT - iOS tablet user = iPad user.

[+] Houshalter|11 years ago|reply
A while back I created a toy IRC bot that answers questions like this just by searching reddit and taking the top comment. It works surprisingly well for questions that are likely to have been asked before. And when it does work, the result is much better than a dry wikipedia excerpt.

I then added some simple machine learning to filter search results for the most relevant threads which improved it quite a bit.

[+] steveeq1|11 years ago|reply
This is called the "Take the First" heuristic. If you want to learn more about "fast and frugal" heuristics (which work surprisingly well in a lot of cases), read "Simple Heuristics that Make Us Smart". It explains how to get programs to give good answers when time and processing power are limited (similar to your case).

http://www.amazon.com/Simple-Heuristics-That-Make-Smart/dp/0...

[+] RBerenguel|11 years ago|reply
Interesting! Care to share some sample question that is likely to be answered by reddit? I tried it manually and can't find answers to any question I can think of right now (looks like I could be failing a Turing test any time soon)
[+] mrfusion|11 years ago|reply
That might actually make a new kind of search engine. You should put it on the web?
[+] pinkyand|11 years ago|reply
Sounds interesting. Do you mind sharing how did you trained your machine learning algorithm ? It seems like an awfully hard task, since in a sense you are competing with the Google algorithm and the order it offered you.
[+] yhlasx|11 years ago|reply
All things considered, no surprises here. Extreme majority of Google's profits depend on how well their search works. Same is not true for Apple with siri, or microsoft with Bing.
[+] mark_l_watson|11 years ago|reply
I think these results were slightly biased because questions seemed to have been chosen to trigger Google search infobox results.

BTW, I worked with Knowledge Graph last year when I consulted at Google. It is an incredibly nice project. The team, who helped me when I needed help, was great - constantly improving the platform.

I am also using the IBM Watson APIs right now while helping another customer, so I feel like I am getting a broad view of what is available.

I expect that Knowledge Box, Cordova, Siri, IBM Watson, etc. are all going to get much, much better in the coming years and will change the way most people use computing devices. Exciting times!

[+] pinkyand|11 years ago|reply
How does Watson compares to the Google technology in your opinion ?
[+] guardian5x|11 years ago|reply
One of the killer features of Cortana is that she gets back to you if she wants additional information. You can almost have something like a dialogue. I don't think that test has taken that into account at all.
[+] tdicola|11 years ago|reply
When you're looking for the answer to something quickly why would you want to have a back and forth game vs. just getting the answer immediately?
[+] Tloewald|11 years ago|reply
Siri did that from day one.
[+] rayiner|11 years ago|reply
I think this analysis is kind of pointless. Who asks their phone trivia questions? You want directions, reminders, etc. Siri seems a lot smarter about following spoken context in that situation.
[+] HelloMcFly|11 years ago|reply
I can't speak extensively for Siri (infrequent use on my iPad) or Google Now (no Android), but I've never had to repeat an instruction to Cortana for a reminder or directions. Not repeating myself when giving voice instruction is essentially my holy grail.
[+] jordanpg|11 years ago|reply
Strongly agree. Isn't this testing the "knowledge box", voice recognition, and NLP at once? It seems like the bulk of this could have been done in a desktop web browser.
[+] BigChiefSmokem|11 years ago|reply
I would rank Cortana as the best and Siri as the worst.
[+] w-ll|11 years ago|reply
Slightly OT, but in school I remember a teach talking about how good voice to text has gotten but was stuck, stating it was roughly about a 70% accuracy, but that we were pretty much at a wall there and haven't moved much in the last decade.

Has there been any major advances in voice recognition other than just growing your speech corpus?

[+] bane|11 years ago|reply
I've actually taken to dictating most of my texts on Android. It's faster for me than typing or swype-style keyboards and it's really very accurate until I need to use a strange proper name it doesn't understand. I'd say it easily gets about 90% of what I say, and it figures out the correct context for words like "there" "their" "they're" and "to" and "too" so far completely correct.

edit thought I'd add this. I just had a conversation with a united agent and they probably understood less than 70% of what I was saying. it was beyond frustrating I wish that I was actually talking to my phone instead.

[+] pkulak|11 years ago|reply
I think it's been mostly throwing hardware at the problem. Server hardware, backed by terabytes of context data.

I remember back in the day, most of the errors were the exact same errors that a human would make. Even you and I only really hear 95% or so of the words someone says. But we can fill in the rest from the context. It always amazes me when I dictate some sentence to my phone, and at the end I see one of the words change to another that sounds almost identical but makes much more sense in the entire context of the sentence.

[+] seanmcdirmid|11 years ago|reply
Deep neural networks have helped a lot in improving accuracy (at least in the last few years).
[+] th3byrdm4n|11 years ago|reply
Just machine learning algorithms and more data.
[+] mrfusion|11 years ago|reply
Side note; It's frusterating how long it takes Siri to launch. sometimes I feel like I'm holding down the home button for 5 seconds before Siri comes up.

And usually I'm trying to decide between a one sentence typing task or asking Siri to do it. The five second wait really throws off my time "profit margin".

[+] joyeuse6701|11 years ago|reply
As a huge halo fan, it makes me happy that Cortana is slowly becoming a well known name =). Having said that, I'd really like to give her a try, I would hope that it would exceed google now on my moto x which for the most part has been very useful while driving.
[+] emersive|11 years ago|reply
I recently bought an iPhone 6, and have really started using Siri for the first time. And Siri has actually surprised me in a good way at how well she understands what I say. I don't do anything too crazy with her. Set alarm at x time, remind me to do x when I get home, Call so and so, send text message to x.

I work on a UX team of three. One of our guys has a MotoX, One guy has a Windows Phone, I have an iPhone 6, and based off of what happens at work it seems that Google Now and Siri are slightly more functional.

This is from merely observations, but it seems like Google Now is faster than Siri. And Siri is better at accurately hearing/understanding the words you say.

[+] boyaka|11 years ago|reply
In regards to understanding voice input, I have been thoroughly impressed with Google's ability to do this ever since I started using their 1-800-GOOG-411 service. I didn't get my first smartphone with data until 2009 (Droid), so the service was really handy for me before that. I'm sure it also gave Google a ton of data to help them improve their capabilities (even mentions that in the Wikipedia article [1]).

http://en.wikipedia.org/wiki/GOOG-411