How Wikipedia Searches 65 Million Articles in Milliseconds

Wikipedia hosts over 65 million articles.

When you type "Virat Kohli" into the search bar, the system has to scan that massive dataset, handle typos, rank results by relevance, and deliver the answer in milliseconds.

If you tried this with a standard SQL database query (LIKE %Virat%), it would trigger a Full Table Scan—reading every single row. At Wikipedia’s scale, that would take seconds or even timeout.

Hence, they use Elasticsearch.

Here is the engineering logic behind that speed, broken down into 4 steps.

1. The Data: Documents, Not Tables

Relational databases split data across multiple tables to reduce redundancy. Elasticsearch optimizes strictly for retrieval speed.

It stores data as denormalized JSON Documents. Here is what the document for "Virat Kohli" looks like in our index:

 {
  "title": "Virat Kohli",
  "description": "Indian cricketer (born 1988)",
  "content_urls": {
    "desktop": { "page": "https://en.wikipedia.org/wiki/Virat_Kohli" }
  },
  "extract": "Virat Kohli is an Indian international cricketer..."
}

By keeping the data together in one object, we avoid expensive JOIN operations during the search.

2. Analysis: Breaking Down the Input

When we save this document, Elasticsearch runs it through an Analysis Pipeline. The goal is to convert raw text into searchable tokens.

For example, if the input is "Virat Parvam", the engine performs two key steps:

1. Lowercase: "Virat Parvam" to "virat parvam"

2. Tokenize: Split by whitespace: ["virat", "parvam"]

The engine indexes these specific tokens, not the raw sentence. This is why a search for "virat" (lowercase) will still match "Virat" (uppercase).

3. The Index: The "Back of the Book"

This is the secret sauce. Instead of scanning every document (like SQL), Elasticsearch uses an Inverted Index.

Think of it like the index at the back of a textbook. It lists every unique word and the exact Document IDs where that word appears.

Here is a simplified view of the index for our dataset:

When you search for "Virat":

The engine jumps straight to "V" in the index. It finds "Virat".
It instantly retrieves the list: [doc1859, doc8834].

This changes the time complexity from O(N) (linear scan) to O(1) (instant lookup).

4. Ranking: It’s Not Just "Found", It’s "Best"

We found multiple documents containing "Virat". But which one is first?

• The article titled "Virat Kohli" and a list of centuries by Virat Kohli

• An author who wrote a book about him

• Cartoon roughly based on Virat Kohli

• Similar name

  

Elasticsearch assigns a Relevance Score to every result (using the BM25 algorithm). It calculates this based on three main factors:

1. Term Frequency (TF): How often does the word appear in the document? If a document mentions "Virat" 10 times, it is likely more relevant than a document that mentions him once.

2. Inverse Document Frequency (IDF): How rare is the word? Common words like "the" have a low score. Rare words like "Virat" have a high score.

3. Custom boosting logic: E.g. A match in the Title field is often weighted as more important than a match in the Body field.

Consider that the following criteria is used to assign points

• Virat: 0.25 (term match in title)

• Kohli: 0.5 (Rare term match in title)

• Virat in description: 0.20

• Kohli in description: 0.45

• Exact query match: 0.2

Here is how our results might be scored:

1. Virat Kohli (Exact Match + High Field Boost): Score 0.95

2. List of Centuries... (High Term Frequency): Score 0.75

3. Virat in description (Low Term Frequency): Score 0.20

Summary

The shift from SQL to Elasticsearch isn’t just about changing databases; it’s about changing your mindset. We move from asking "How do I save space?" to "How do I find this instantly?" This retrieval-first approach relies on four key architectural pillars:

1. ​ Ingestion: Storing data as JSON documents.

2. ​ Analysis: Tokenizing text into searchable terms.

3. ​ Indexing: Using an Inverted Index for O(1) lookups.

4. ​ Ranking: Scoring results by relevance.

This is a perfect example of how choosing the right data structure(Inverted Index) solves scalability problems that raw computing power cannot.

---

I use Elasticsearch in production daily. I wrote this article to solidify the fundamentals and visualize the mechanics. If you’re working on similar search scaling challenges, feel free to reach out!

Note: This article was drafted with the assistance of Google Gemini to refine the technical explanations and structure.