Quote from Matthew Whitewood on November 12, 2021, 2:39 pm

Brief Journey of Google's AI Developments for its Search Engine

I am rather interested in how Google changes its search engine algorithms.
Particularly the AI component of its search engine.

There have been some AI developments from Google that may have affected their search engine results.
This article seems to breakdown its key changes:
A roundup of Google updates from September 2021

• Google link spam update
• TFR-BERT Update
• Google search ranking algorithm update
• MUM technology update

I am more familiar with TFR-BERT and MUM so will talk about those 2 changes.

Here's Google's blog for its general AI developments:
Google AI Blog

Advances in TF-Ranking (July 27, 2021)

I suspect that this is what's causing the changes in search engine results.
Google is probably experimenting with new ranking models.
As such, the ranking results could have some major changes (or undesirable effects).

TF-Ranking stands for TensorFlow Ranking.
It's a library for Learning-to-Rank (LTR) techniques.
Google even coupled BERT (see below) and LTR to get a new model TFR-BERT.

That's how I found the first article link above:

MUM - Multitask Unified Model (May 18, 2021)

Google claims MUM to be 1000 times better than BERT although I'm not sure what that means.
This model is not open-sourced so I know much less about it.

The article says MUM is trained

• across 75 languages
• on many different tasks at once (tasks are classification, entity recognition, etc)
• on different kinds of data (text, images, etc) - multimodal in other words

I think the animation on the link best showcases its improvement:

BERT (2018 - 2020)

The biggest one was the introduction of BERT in 2018 and incorporating the model into their search engine in 2019.
It was introduced to solve query ambiguities.
In 2020, it seems that Google used the model for almost every English search query.
And it further developed BERT models for other languages like Spanish, Portuguese, etc.

BERT was one of the huge breakthroughs in natural language processing.
The novelty is that the model reads left to right and also right to left so it has better context.
For previous models, the model usually only takes into account the words on the left of each word.

Previous Models - unidirectional

This is an example of a sentence. Earlier models don't focus on the words on the right.

BERT - bidiretional

This is an example of a sentence. BERT also takes into account words on the right in addition to the left for each word.

Kind of like how people read books linearly and only take into account the previous context.
But, if you jump ahead and then re-read previous portions, you get better context.

The original, open-sourced version of BERT is pre-trained on a large English dataset originally.

• BooksCorpus, a collection of free novels (800M words) (Zhu et al.,2015) and
• English Wikipedia (2,500M words)

So it's like a kid who read a lot of English books and articles.

Then engineers or researches usually train this pre-trained model a second time on a more specific dataset (fine-tuning).
So that BERT performs better within a more specific domain like finance for example.

It can be very challenging to train BERT and generalise its effectiveness on a wide domain.
You need a lot of clean, training data and GPU/TPU power.
Google probably has found a way to achieve this to a certain extent.

BERT is open-sourced on GitHub.
Most easily tried out via Hugging Face's implementation of BERT in my opinion.

Fun Fact: BERT is named after the Sesame Street character.
There's also ELMo by AllenNLP and Ernie by Baidu, the Chinese search engine :).