AI Assistants: Who’s Setting the Pace?

Our initial research into AI assistants (e.g. Alexa) has produced a few interesting findings that could bring about competitive advantages to companies in the future. In summary:

  • AI assistants use a variety of different sources from high ranking websites and YouTube comments to Wikipedia, varying wildly in reliability to answer your questions – so make sure you control the source.
  • This leads to discrepancies in facts and timings, such as ranging from minutes to hours in how long films are.
  • Initial results show that companies which share a name with common nouns could be worse off, as AI assistants return definitions instead of company information.

The race is on!

With the constraints and rules ever changing in the digital marketing footrace, particularly at the whim of Google and other giants, it’s easy for those without proper guidance and knowledge to be wrong footed and left behind.

On occasion, a new advancement, ruling or opportunity may blow the field wide open and reset. Those that exploit it at an opportune moment get a head start out of the blocks and a significant competitive advantage.

As discussed in our introductory post on Artificial Intelligence Optimisation, the most recent gamechanger is already upon us in the form of artificial intelligence (AI).

Having been in the public domain for many years, Siri, Apple’s personal digital assistant was changing how people searched from as early as October 14, 2011. However, with the October 2016 launch of Amazon Echo in the UK, the outreach and possibilities of AI are gathering momentum.

Although not easy to get hold of outside of the US (at the time of writing), the Google Home is the clearest competitor of the Echo. Other notable AI assistants and devices include Microsoft’s Cortana and Apple’s Siri.

Here at HOME, we’ve done some experiments to compare:

  • Amazon Alexa – on the Echo
  • Microsoft Cortana – in app form on an Android phone
  • Apple’s Siri – on an iPhone
  • Google Assistant – on an Android phone

Some of the limitations we’ve noticed so far are somewhat peculiar, but all could give room to manoeuvre if you’re trying to gain competitive advantage.

How long is your favourite film?

The use and potential in voice search is undeniably impressive and appealing. When asked even in a busy office, with multiple conversations in the air, Alexa can isolate a question directed her way.

For instance, ask, “How long is Star Wars a New Hope?” and Alexa will efficiently inform you that, “Star War Episode IV: A New Hope’s duration is 2 hours and 6 minutes.”

Great! Or so we thought.

Amazon Echo with Star Wars speech bubble

Then we asked Siri, who claims its shorter – by 2 minutes – at 2 hours 4 minutes. Even more confusingly, Google disagrees again, believing it’s 2 hours 5 minutes.

Of course, this is trivial, as a minute or two won’t make a difference to whether we choose to watch this classic on a lazy Sunday evening.

But time discrepancies can be much more drastic – 1 hour 39 minutes, in fact, for the film Ben-Hur. Alexa references the 1959 classic, not the modern-day version, whereas Google Assistant does the opposite.

This mismatch was easy to spot, but what if it isn’t easy to notice?

What if we are trying to find out the journey time to an important meeting or baking time of a Victoria Sponge for the family? This variance for time sensitive questions can be the difference between showing up on time or being late, delicious success or burnt failure.

Did you know an apple is a member of the rose family? We didn’t!

Discrepancies between AI agents stem from the different search engine used. Cortana relies on Bing – as you would expect, with them both being Microsoft products.

Similarly, Google Assistant relies on, you guessed, it Google.

Both Alexa and Siri rely on contributions from Bing, but also make heavy use of Wikipedia and Wolfram Alpha.

Furthermore, both Google Assistant and Cortana prioritise responses primarily from the featured snippet (or equivalent) – the blocks at the beginning of search results. Failing that, they use Knowledge Graphs and only then the normal “10 blue links”-style search results.

This is true of all questions we’ve asked so far.

For instance, “What is an apple?” prompts the response, “The round fruit of a tree of the rose family, which typically has thin green or red skin and crisp flesh.”

This is taken from the first line of the definition in the featured snippet.

Consistently, the first definition in a featured snippet is almost always referenced. What’s interesting is that this causes some strange results, such as the question, “What is Apple?” to which Google tells us about the fruit, but not the technology company.

If we ask Alexa this, right now it tells us the same thing, but for a while it gave us details on Apple, taken from the Knowledge Graph/Wikipedia entry, usually the secondary source.

Over time this will give companies with noun-based names some grief. Not even Apple gets good treatment right now, but assuming this is fixed, will a company need to be as globally renowned as Apple for the search engines and AI agents to overlook the featured snippet in favour of the Knowledge Graph?

Most likely the responses from Google, Bing and all AI agents based on them will be manipulated in favour of specific companies, services and products, just as they can be in traditional SEO.

Sorry, I can’t find the answer to the question I heard.

Despite pulling from the combined resources of Bing, Wikipedia and Wolfram Alpha, a noticeable number of questions to Alexa go unanswered with an apologetic, “Sorry, I can’t find the answer to the question I heard.”

This includes questions where the building blocks are all there, such as, “What mountain range is Ben Nevis part of?” The Wikipedia entry, for instance, even says where it is, but we were left wondering.

Luckily, Google Assistant duly informed us that, “Ben Nevis is part of the Grampian Mountains”. Interestingly, this was worked out from the Knowledge Graph, as no featured snippet was available.

In contrast when, “Who is the next king of England?” was asked, again Alexa left us in the dark, but Google Assistant confidently explained “Prince William will be next king of England”. The source for which it went on to explain “Queen Elizabeth has decided. According to YouTube” taken from a YouTube comment.

(Sadly, this result no longer appears.)

Mobile phone with Prince Charles

By not having an answer, as is often the case with Alexa, she appears less impressive in comparison to Google Assistant. Google can typically give an answer to more obscure questions, but at the cost of drawing content from unreliable sources, leading to false information, like above.

As time goes on, we need to consider how much faith we put in the single answers we receive from these assistants. The industry needs to continue improving to ensure more obscure questions can be answered, but not through sacrificing the reliability of the source.

A noticeable limitation with Alexa means it doesn’t break away from the pack. Both Cortana and Google Assistant understand pronouns if they relate to someone I just asked about. This allows for a much superior continued search experience, rather than repeating the subject matter with Alexa.

Granted it isn’t flawless, Cortana can only retain pronouns if there is a Knowledge Graph to then pull information from.

For example, after being asked, “Who is Theresa May?” and then “What does she do?” Cortana was stumped, as this information wasn’t in the Knowledge Graph.

She also became increasingly confused, mirroring our frustrations when homonyms were used in voice search.

For all of you reading this without a AI assistant handy to ask, a homonym is one of a group of words that share the same pronunciation, but have different meanings, whether spelled the same or not, like ‘weight’ and ‘wait’.

Google Assistant is certainly striding ahead in this aspect, confident with what and who pronouns refer to even with no featured snippet or Knowledge Graph as well knowing a “male heir” from “mail hair”.

The danger of definitions

Not without its own Achilles Heel, however, we found Google Assistant relies too heavily on the first definition result in the featured snippet.

Even worse than the example of Apple, above, when we asked, “What is Mars?”, we expected to be told about a planet, a god or maybe a chocolate bar.
Alexa, referenced the planet from Wikipedia, and Cortana the god, from the first definition of a Bing featured snippet.

Siri, however, linked us to the Wikipedia page on Laird James McCullen Destro XXIV, founder of M.A.R.S. Industries in G.I. Joe.

Thinking this was just a language processing issue, we moved confidently on to Google Assistant. Not to be outdone, Google used the first definition from the Google featured snippet on the word ‘mar’, telling us that ‘mars’ is the third person present.

Interestingly, Mars the planet does have the Knowledge Graph entry for Google Assistant to call upon, but once again, Google didn’t use it.

Neither Mars the planet, nor Mars the Greek god of war have anything to gain from occupying the Knowledge Graph. Mars, the popular British chocolate bar and parent company of the same name that has a hand in a few industries, however, does.

What’s next?

We’ll continue our research into the different AI agents and assistants to understand how to exploit the situation. You too now know to exploit the selection of featured snippet over Knowledge Graph for AI assistants, but if you want to know any more, get in touch and we’d be happy to help.

Written by:

Daryl Lucas

Category:

What we think

Date:

27/03/2017

You may also like

What we think

  

/  13 Nov 2019

Facebook Pay: yay or nay?

Facebook’s bid for world domination levelled up this week with the announcement that Facebook Pay would shortly be rolling out to users in the US, with the intention of the service going global over time. Facebook Pay will be enabled within Faceboo

Read more

What we think

  

/  06 Nov 2019

Everyone has terrible passwords

Earlier this year, GCHQ’s National Cyber Security Centre division issued a report that stated the most commonly used passwords include words such as 'password', 'liverpool' and '123456'. This is a worrying headline, but depressingly, not a surpr

Read more