What is Search Engine?
Definition
A search engine is a software system that discovers web pages with automated programs, analyses and stores their content in an index, and ranks the results it judges most useful for each query a user types. Google, Bing and Yandex are general web search engines; vertical engines such as image, map or on-site search focus on a single type of content.
Also known as: web search engine, search provider, internet search engine
Three systems behind one search box
On the surface there is a query box and a list of results. Underneath sit three separate systems that constantly feed one another. Google names them crawling, indexing and serving; the internals of Bing, Yandex and other general engines differ, but the architecture follows the same logic.
- Discovery and crawling: programs called crawlers follow links on known pages and read sitemaps to find new URLs, download those pages and, where needed, render them by running their JavaScript.
- Indexing: the downloaded content is analysed, including text, headings, images, links, language and structured data. Near-duplicate pages are clustered and one representative (canonical) version is chosen per cluster.
- Ranking and serving: when a query arrives, candidate pages are pulled from the index, ordered, and assembled into a results page.
The separation has a practical consequence. A page can be crawled yet never indexed, or sit in the index without ranking for anything useful. Diagnosing a visibility problem starts with finding out which of the three stages it is stuck in.
The inverted index
A search engine does not browse the web when you press Enter; it looks things up in an index it built in advance. The classic data structure of information retrieval is the inverted index, which maps every term to the list of documents containing it, much like the index at the back of a textbook.
"espresso" → [doc-14, doc-90, doc-233]
"grinder" → [doc-90, doc-512]
"manual" → [doc-90, doc-233, doc-512]
Query: "manual espresso grinder"
Intersection → doc-90 (contains all three terms)Modern engines layer much more on top: language models that understand synonyms and what a query actually means, semantic search that represents passages as vectors, and knowledge graphs that connect people, organisations and places. The core idea survives, though: at query time the engine selects good candidates from an index prepared beforehand.
What puts one result above another
Google says the results a person sees depend on many factors, including their location, language and device. Broadly, ranking systems try to answer a handful of questions:
- What does the query mean, and does the searcher want information, a product or a particular website?
- Which pages are relevant and genuinely answer it?
- Are those pages reliable and carefully made?
- Is the page usable on a phone and reasonably fast?
- What is the context? A search for "pharmacy open now" in Leeds should not return the same list as one in Bristol.
Two points are routinely misunderstood. Google does not accept payment to crawl a site more often or rank it higher; ads are a separate system labelled "Sponsored". And even a page that follows every guideline is not guaranteed to be crawled, indexed or served. Google's own walkthrough is In-depth guide to how Google Search works.
Kinds of search engine
| Kind | Examples | What it indexes |
|---|---|---|
| General web search | Google, Bing, Yandex | Pages, images and videos on the open web |
| Vertical search | Google Images, Google Maps, shopping search | One content type or one data source |
| On-site search | The product search box of an online shop | Only that site's own records |
| AI-assisted search | Google's AI Overviews and AI Mode, web search inside chat assistants | A conventional index plus a generated answer with source links |
The last row places a generated answer next to, or instead of, the list of links. Google explains that AI Overviews and AI Mode may run several related searches across subtopics while composing a response, which means they still select sources from a search index. AI search adds a presentation layer on top of search infrastructure rather than replacing it.
What it means for site owners
Each stage asks something different of a website. Crawling needs reachable URLs and healthy server responses; indexing needs readable, non-duplicated content; ranking needs pages that answer the query better than the alternatives. The discipline that covers all three is SEO. The most direct view of how an engine sees your site comes from the tools it offers site owners: Search Console for Google and Bing Webmaster Tools for Bing.