One of the key advantages of Yext Search is its multi-algorithm strategy. Search deploys multiple algorithms in production that can either work in isolation or interact with each other, depending on what the user configures.

Our in-house data science and ML operations teams have created and fine-tuned advanced ML models that are regularly trained on large datasets of human-labeled data to ensure high-quality, highly-accurate outputs.

Advanced Search Tier Features: Document search, semantic search, and inferred filters (except on builtin.location) are only available on the Advanced Search tier.

Once a user runs a query, Search determines which algorithm(s) to apply based on which searchable field type(s) are configured. The searchable fields algorithms are:

• Keyword Search
• Phrase Match
• Semantic Search
• Inferred Filter
• Static Filter
• Document Search

doc.Score

The Search algorithm uses a concept of a document 'score', doc.score, which is simply the number of token matches in the result text.

Search uses doc.score in three (3) main junctures:

Keyword Search

The most basic algorithm offered today is Keyword Search, which relies on keyword matches between the query and the result. When you enable Keyword Search on a field, Search will return results whenever there is a match between a query and field token, unless they are stop words. When ranking results, Search considers the relative importance of the token with respect to all of the content in your platform.

Keyword Search uses an algorithm called BM-25, which works on the concept of TF-IDF, or term frequency-inverse document frequency. Put simply, it measures how important a word/term is to a result field, for each result in a result set. The result with the most occurrences of 'important' terms is deemed the most relevant.

• TF (term frequency) works by looking at the frequency of a specific term relative to the field value string. This is calculated by counting the number of times a term appears in a string and dividing it by the total number of words in that string.
  • Stop words are words that appear all the time and generally contain no meaning to the searcher, like 'a', 'the', and 'of'. These are only used as tiebreakers in scenarios where two results adjusted for stop words are the same. Search maintains a built-in list of these, and users can define additional stop words in their configuration specific to their business.
  • Since Keyword Search is entirely based on token matches, stop words are used for token matching (and thus may return results) when no non-stop words are present.
• IDF (inverse document frequency) applies weighting to words such that common terms have a minimized weight, and rarer terms have a greater impact.

The individual TF and IDF scores are multiplied together to arrive at a TF-IDF score. The higher the score, the more relevant that word is to the document. Scores are calculated uniquely for each experience, because terms that are rare in one experience may be common in another, and vice versa.

Phrase Match

Phrase Match is a variation of Keyword Search that only returns a given result for a query when the entire field value appears in the search query. Rather than calculating relevance by the weighted frequency of a term or set of terms amongst all the words in a document, Phrase Match will only consider a document relevant if all terms defined in the query appear in the document as a continuous phrase.

Query: When is the super bowl?

Phrase Match works well with keywords fields on entities, or in any other scenario where you'd like to use Keyword Search, but do not want noisy keyword matches on parts of a field value. Phrase Match applies a boost on the result's doc.score.

Semantic Search

Advanced Search Tier Feature: Semantic search is only available on the Advanced Search tier.

Semantic Search incorporates a result's similarity to the query's underlying meaning.

There are often many different ways to phrase the same idea. Take a FAQ result for 'How do I return an order?' and some examples of query phrasing:

FAQ: How do I return an order?

By relying on semantic similarity rather than simple keyword matches, semantic search is able to understand that the first three queries above have a similar meaning to the question being asked.

Embedding Model

Semantic Search measures similarity by using a process called embedding. Our Data Science team built our own embedding model on top of MPNet (similar to BERT).

When a user enters a query, the model embeds the query and a result. This transforms them into points in high-dimensional vector space. Here, a query and result that mean the same thing will be close together, even if they don't contain any of the same words.

Token Matching and Similarity Thresholds

The semantic return threshold for a result is 0.3. Any result above this is considered semantically similar enough to be returned organically by semantic search.

When semantic similarity is low, the algorithm falls back to token matches. When semantic similarity is high (at or above the similarity boost threshold of 0.65), the result gets boosted straight to the top regardless of the number of token matches.

Here is the logic in code:

{
  similarity = // the similarity function
  matchedTokens = doc.score // score per token match
  boostThreshold = 0.65 // the similarity boost threshold
  if (similarity > boostThreshold) {
    return 1000 + matchedTokens + similarity;
  }
  return matchedTokens + similarity;
}

Query: are bonita fish big?

In this example, "What's a skipjack tuna?" was bumped to #2 because its similarity (0.75) is above the boost threshold of 0.65, even though it has no matched tokens.

Inferred Filter

Advanced Search Tier Feature: Inferred filtering (except on builtin.location) is only available on the Advanced Search tier.

The idea behind inferred filters is to literally infer filters from the query, such as color == blue or location == New York.

The inferred Filter algorithm behaves differently depending on the type of field it is applied on. The three (3) main categories are:

• Normal Field Value (any field not builtin.location or builtin.entityType)
• Location (builtin.location)
• Entity Type (builtin.entityType)

Inferred Filters on Normal Fields

For a normal searchable field with inferred filtering configured, the algorithm looks for a field value's tokens in the query (e.g., 'blue hat' or 'doctors who accept blue cross blue shield') and applies those tokens as a strict filter.

"appliedQueryFilters": [
  {
    "displayKey": "Color",
    "displayValue": "Blue",
    "filter": {
      "color": {
        "$eq": "Blue"
      }
    },
    "type": "FIELD_VALUE"
  }
]

If the potential inferred filter field has two or more tokens, there needs to be a match with at least two tokens from the query for the inferred filter to be applied.

If there are multiple matches on an inferred filter field of two or more tokens, the best match (highest percentage of tokens matched) is used as the tiebreaker.

'Using Up' Tokens: When a token triggers an inferred filter, that token gets 'used up'. This means you can't apply inferred filter and another algorithm (like keyword search) on the same token. Inferred filters are always applied first.

If there are no additional matches from other algorithms, Search shows all results that match the inferred filter(s):

Inferred Filters on Location Fields

The special inferred Filter behavior for location fields only works on builtin.location. Address subfields such as address.city and address.region are treated like regular fields.

When a user inputs a location search with inferred filter enabled on builtin.location, our Named Entity Recognition (NER) model detects key location-specific tokens in the query, then sends those to Mapbox to find the central latitude and longitude.

Note: If NER is not supported for a particular language, an n-grams approach (up to 3-grams) is used instead. NER Location Detection is currently supported in English, French, German, Italian, and Spanish.

If a bounding box is configured, only places within that bounding box are considered. The system then applies logic to prioritize places that have entity matches in the Knowledge Graph.

Once a place and its central lat/long are identified, an approximate radius is calculated:

• 0-25 miles - search within a 25 mile radius
• 25-50 miles - search within a 50 mile radius
• Greater than 50 miles - use the approximate radius

This is configurable via the minLocationRadius property (in meters).

Example API response for an inferred filter on 'Brooklyn':

"appliedQueryFilters": [
  {
    "displayKey": "Location",
    "displayValue": "Brooklyn",
    "filter": {
      "builtin.location": {
        "$eq": "P-locality.66915052"
      }
    },
    "type": "PLACE",
    "details": {
      "latitude": 40.652601,
      "longitude": -73.949721,
      "placeName": "Brooklyn, New York, United States",
      "featureTypes": ["locality"],
      "boundingBox": {
        "minLatitude": 40.566161483,
        "minLongitude": -74.042411963,
        "maxLatitude": 40.739446,
        "maxLongitude": -73.833365
      }
    }
  }
]

Inferred Filters on Entity Type Fields

Each vertical has a builtin.entityType field enabled for inferred filtering by default. When you create a 'Location' entity, for instance, the system generates builtin.entityType fields for 'Location' and 'Locations' (singular and plural). If a user searches for 'Locations near me', builtin.entityType limits the results to Location entities.

Special Cases

"Open Now" - If someone searches 'open now' and builtin.hours is configured for inferred filtering, results are filtered to currently open locations.

"Near Me" - This only works for builtin.location. The system prompts the user for HTML5 location (browser popup), or falls back to IP-based geolocation. Mapbox is not involved.

Static Filter

Static Filters narrow down a result set based on filter conditions set by the user. They are generated on the client-side (pre-configured on the front-end and passed into the Search API) and apply independently of query content.

Document Search

Advanced Search Tier Feature: Document search is only available on the Advanced Search tier.

The purpose of Document Search is to take dense, long-form documents and extract relevant passages of text based on the query (snippeting).

A Featured Snippet is a type of Direct Answer that, given a question and a span of text that exactly answers it, serves the text as a direct answer:

Document Search returns results using the BM-25 score, then applies the Extractive QA (Question/Answering) model on the top five results. If there is an exact answer to the question, Search returns it as a Featured Snippet. If there isn't one, no Featured Snippet is returned.

An Inline Snippet is used when there isn't a direct question to answer but it still makes sense to show the most relevant span of text for each entity. There is one featured snippet per query, but one inline snippet per entity. Inline snippets show the ~250 character section of the body text containing the most token matches.

How Our Algorithms Interact

Note: Document Search is not configurable with other algorithms on the same field.

Phrase Match + Keyword Search

If both are configured on the same field, both matches get returned, but Phrase Match applies a boost.

Query: 'nike sportswear shorts' | Field: name (Phrase Match, Keyword Search)

Keyword Search + Semantic Search

If both are configured on the same field, the algorithm behaves as if plain Keyword Search wasn't even applied - it just performs Semantic Search. Be cautious when configuring these on separate fields, as keywords from one field can introduce noisy token matches into semantic search on another.

Inferred Filter + Others

Inferred Filter is always applied first and applies a strict filter on the results set. Any tokens used by the inferred Filter are ineligible for other algorithms. If inferred Filter couldn't find any filters to apply, it "falls back" to Keyword Search or whatever other algorithms are configured on that field.