I Built My Own Search Engine Because I Wanted to Know How Search Actually Works

I have a habit of building things that I probably don't need to build.

At some point I started wondering: how hard would it actually be to build a search engine?

Not a search bar over a database.

Not a wrapper around Google's API.

Not something where I send a query to Bing and redesign the results page.

I wanted the entire thing.

A crawler that discovers the web. A system that stores those pages. Something that understands links between websites. Something that decides which pages are important. An index that can search millions of words quickly. A ranking system. Image search. Autocomplete. Spelling correction. Semantic search. Feedback from clicks. Knowledge panels. And eventually even my own browser built around it.

That project became Boogle.

And Boogle became considerably larger than I originally intended.

Today it is a locally runnable search-engine stack with its own crawler, processing pipeline, link graph, ranking system, search APIs, web interface, mobile interface, embeddings, feedback system, experimental AI services and, more recently, a desktop browser called Surf.

This blog is the story of how I built it, why I rewrote major parts of it multiple times, how it works today, what I learned while building it, and also the parts that are still imperfect.

It started with JavaScript

When I started Boogle, I wrote it in JavaScript and Node.js.

There wasn't some sophisticated architectural reason behind that decision.

I simply knew JavaScript.

Most of the things I had built until then were around the JavaScript ecosystem, so when I wanted to experiment with a crawler, Node.js was the obvious place to start.

The first version was pretty straightforward:

seed URLs
↓
fetch page
↓
parse HTML
↓
extract text + links
↓
store page
↓
add discovered links to queue
↓
repeat

And it worked.

That was actually an important moment.

You don't need Google's infrastructure to understand the fundamental idea behind Google.

At its simplest, a search engine starts with a crawler repeatedly doing something like this:

const html = await fetch(url)

const links = extractLinks(html)
const text = extractText(html)

await save({
url,
text,
links
})

queue.push(...links)

Of course, the difference between this and a real search engine is enormous.

The difficult part isn't downloading one page.

The difficult part is doing it for thousands, millions or billions of pages while answering questions like:

• Have I already visited this URL?
• Are these two URLs actually the same page?
• Should I crawl this link?
• How aggressively can I crawl this domain?
• Is this page useful or garbage?
• Is this content duplicated somewhere else?
• How do I store the link relationships?
• How do I search the resulting dataset quickly?
• How do I decide which result deserves position #1?

Once I started asking those questions, Boogle stopped being a weekend crawler.

It started becoming a search engine.

JavaScript wasn't where I wanted to stay

As the crawler became larger, performance started bothering me.

I wanted something that could comfortably run a lot of concurrent network work, process large batches of pages and eventually become a collection of backend services.

So I did what developers occasionally do when they become obsessed with performance.

I rewrote it in Rust.

On paper, Rust made perfect sense.

Fast.

Memory efficient.

Excellent concurrency.

Great type system.

Perfect language for infrastructure.

There was only one problem.

I wasn't productive in it.

I would spend an unreasonable amount of time fighting the language, debugging ownership problems, understanding lifetimes and fixing things that had almost nothing to do with building a search engine.

Rust wasn't the problem.

My familiarity with Rust was.

I eventually realised something obvious:

The fastest language isn't particularly useful if I spend more time debugging the language than building the product.

So Boogle got rewritten again.

This time in Go.

There was another problem: I didn't know Go

This is probably my favourite part of the project.

I switched an increasingly complicated search engine to a programming language I didn't properly know.

I knew what I wanted architecturally.

I understood APIs, databases, queues, concurrency and backend systems from other languages.

I just didn't know how Go wanted me to express those things.

So I used GPT heavily.

Sometimes I would ask how a Go concept worked.

Sometimes I would translate a pattern I already understood from JavaScript into Go.

Sometimes GPT helped write portions of the implementation.

Then I would run it, break it, inspect it, change it and gradually understand what was happening.

My learning loop looked something like:

I know what I want
↓
I don't know how Go expresses it
↓
ask GPT
↓
read generated code
↓
run it
↓
break something
↓
debug it
↓
understand why
↓
rewrite/improve it

I don't pretend that every line of Boogle materialised from my head without assistance. AI helped me learn Go and helped build many parts of the system.

But that's also part of why the project is interesting to me.

A few years ago, deciding to build a distributed search engine in a language you didn't know would have required first spending weeks or months learning the language.

Now I could learn the language while building the thing.

GPT essentially became an interactive combination of documentation, Stack Overflow, rubber duck and pair programmer.

The important thing was that I still had to understand the system.

Because once something broke across the crawler, Redis, Kafka, MongoDB, Meilisearch and the embedding service, saying "GPT wrote it" wasn't going to fix anything.

What Boogle actually is

Boogle doesn't call Google or Bing to get its search results.

It owns its own corpus.

The current system can:

• crawl pages from seed URLs;
• discover new URLs;
• normalize and deduplicate them;
• extract the useful content from HTML;
• extract links, images and metadata;
• calculate content quality and spam signals;
• construct a web link graph;
• calculate PageRank;
• generate semantic embeddings;
• index pages and images;
• perform lexical and semantic retrieval;
• rerank candidates using multiple signals;
• provide image, news and video search;
• provide autocomplete and spelling correction;
• generate direct answers and auxiliary panels;
• track impressions and clicks;
• feed CTR information back into ranking;
• expose everything through a web UI and mobile app;
• provide experimental AI/RAG services;
• and act as the search engine inside Surf, the desktop browser I started building this month.

The checked-in implementation is split into dedicated crawler, processor, pipeline, search, embedding, AI, frontend, mobile and browser components.

At a high level, you can imagine Boogle like this:

INTERNET
│
▼
┌─────────────┐
│ Go Crawler │
└──────┬──────┘
│
raw pages + events
│
┌────────────┴────────────┐
▼ ▼
MongoDB Redpanda
│
▼
Go Pipeline
│
▼
Go Processor
│
clean / score / PageRank
│
┌─────────────────────┼──────────────────┐
▼ ▼ ▼
MongoDB Meilisearch Embeddings
│
▼
Go Search
│
┌────────────────────────┼──────────────────────┐
▼ ▼ ▼
Web UI Mobile App Surf

There are Redis caches and deduplication layers throughout the system as well.

The point isn't that every one of these services is necessary for a small search engine.

It absolutely isn't.

Part of Boogle is me deliberately experimenting with architecture.

Step 1: crawling the web

Everything starts with URLs.

Boogle loads URLs from a seed file and can additionally pull previously discovered URLs from MongoDB, meaning the crawler can gradually create its own frontier.

Before crawling something, the URL gets normalized.

Things like:

https://example.com/article#comments

https://example.com/article?utm_source=twitter

https://EXAMPLE.com/article

should generally not become three completely independent pages in the index.

Boogle therefore removes fragments and common tracking parameters, cleans paths, normalizes hosts and deduplicates URLs.

The processed identity is deterministic:

id = "u_" + SHA1(canonicalURL)

SHA-1 here isn't being used cryptographically. It's simply a compact deterministic identifier.

Crawling politely

One thing I didn't want was:

32 workers
↓
all discover example.com
↓
example.com gets murdered

So the crawler partitions work by domain.

Each host gets its own queue and domain worker while an overall semaphore limits global concurrency.

The current default global fetch concurrency is 32, with per-host buffered queues. The crawler also has a maximum crawl depth of two.

This was one of those moments where a "crawler" became a concurrency problem.

The system isn't simply asking:

What URL should I download next?

It is asking:

What URLs can I download concurrently without repeatedly fetching the same thing or disproportionately hammering one host?

That is much more interesting.

Not every HTML page deserves to become a search result

Downloading HTML is easy.

Extracting useful information from arbitrary HTML is much uglier.

Pages contain:

<nav>
<footer>
<script>
<style>
<form>
<aside>
cookie banners
navigation menus
advertisements
related articles

Boogle tries to remove much of that and prioritises content from:

main
article
[role="main"]
body

It extracts things such as:

title
main body
description
site name
author
published date
modified date
outbound links
anchor text
images
OpenGraph metadata
JSON-LD / schema.org metadata

It then applies quality gates.

For example, the crawler currently expects a sufficiently long body, a meaningful title and limited thin-content signals before treating a page as rich enough to expand further.

This immediately made me appreciate how ugly the real web is.

HTML is technically structured.

Actual websites are not.

Deduplication became a problem everywhere

The same URL can enter a search system repeatedly.

A crawler discovers it twice.

A different page links to it.

A process restarts.

An event gets replayed.

The same article appears under multiple URLs.

Another site republishes the article.

So Boogle ended up with several layers of deduplication.

Redis keeps keys such as:

visited:<sha1(url)>
processed:<sha1(canonicalUrl)>
content:<sha1(content)>
pipeline:dedupe:crawl:<sha1>
pipeline:dedupe:process:<sha1>
embed:processed:<sha1(url)>

Mongo and Meilisearch writes are also designed to be idempotent through deterministic IDs/upserts.

And search performs another deduplication pass before showing results.

The system therefore tries to defend against duplicates at the crawl frontier, event pipeline, processing layer, embedding layer, storage layer and finally retrieval layer.

This is another thing building Boogle taught me:

distributed systems create the same problem in multiple places.

"Don't crawl the same page twice" sounds like one feature.

It isn't.

MongoDB became the source of truth

I use MongoDB as the durable corpus.

The crawler first stores relatively raw page information.

The processor later produces enriched documents containing things such as:

URL
canonical URL
title
body
links
anchors
images
metadata
keywords
quality
spam score
freshness
backlinks
mentions
PageRank
composite rank
embedding
CTR
timestamps

The web graph is stored separately as edges.

That matters because eventually your representation of:

page -> every link from that page

becomes real data rather than an abstract graph from a DSA textbook.

Then I needed a search index

MongoDB stores the corpus, but I didn't want it to be the primary full-text retrieval engine.

So Boogle uses Meilisearch as the derived search index.

The basic philosophy is:

MongoDB
│
│ durable truth
▼
Processor
│
▼
Meilisearch
│
│ disposable / rebuildable search representation
▼
Search API

The processor can clear and rebuild the Meilisearch indices from the durable corpus. Embeddings can live inside the indexed documents for reranking, while Mongo remains authoritative.

I like this separation because losing an index shouldn't mean losing the crawled web.

You rebuild the index.

Building my own tiny web graph

This was one of the coolest parts.

Every crawled page contains outbound links.

Once those URLs are normalized into IDs, I effectively have a graph:

A → B
A → C
B → C
C → A
D → C

And suddenly all the graph theory I had seen academically became useful.

A page isn't important only because it contains the query.

A page may also be important because other important pages link to it.

That leads naturally to PageRank.

PageRank

Boogle calculates PageRank over its own crawled graph.

The processor currently performs 20 iterations with a damping factor of:

d = 0.85

The conceptual equation is:

PR(A) =
(1 - d) / N
+
d × Σ(PR(T) / outgoingLinks(T))

where T represents pages linking to A.

I also inject freshness into the computation and then combine PageRank with several other authority signals.

The resulting authority calculation roughly includes:

PageRank
backlinks
mentions
freshness
outbound links
content quality

One current composite formula is:

Rank =
PageRank * 0.45
+ normalizedBacklinks * 0.20
+ normalizedMentions * 0.10
+ freshness * 0.15
+ normalizedOutbound * 0.05
+ normalizedQuality * 0.05

This isn't Google's ranking algorithm.

Obviously.

But implementing even a simplified version completely changed how I thought about search.

Search isn't:

SELECT *
FROM pages
WHERE body LIKE '%query%'

Search is mostly a ranking problem.

Relevance is not one number

Suppose someone searches:

golang concurrency

Page A contains those words repeatedly but is garbage.

Page B contains an excellent explanation but phrases things differently.

Page C contains an exact title match but is ten years old.

Page D has fewer keyword matches but hundreds of relevant backlinks.

Which one should rank first?

There isn't a single correct signal.

So Boogle combines many of them.

The current reranking pipeline considers things including keyword relevance, semantic similarity, PageRank, backlinks, mentions, freshness, content quality, click-through rate, anchor text, domain authority, graph inlinks, token proximity, exact-title matches, phrase matches and spam penalties.

Conceptually:

┌── Keyword relevance
├── Semantic similarity
├── PageRank
├── Backlinks
├── Freshness
Search Candidate ────────┼── Content quality
├── CTR
├── Anchor relevance
├── Domain authority
├── Graph importance
├── Phrase/title boosts
└── Spam penalty
│
▼
Final Score

And then I sort.

That is effectively the heart of Boogle.

Lexical search wasn't enough

Normal keyword search works surprisingly well.

But it has an obvious limitation.

Humans search by meaning.

Documents contain words.

So I added semantic embeddings.

Boogle currently uses:

BAAI/bge-small-en-v1.5

to generate normalized 384-dimensional text vectors.

A page becomes something like:

[
0.021,
-0.038,
0.104,
...
384 dimensions
]

The query gets embedded too.

Then Boogle can calculate cosine similarity between the vectors.

cosine(A, B) =
(A · B) / (||A|| × ||B||)

Now a document doesn't necessarily need to contain exactly the same words to be considered relevant.

I don't always run embeddings

Embeddings are more expensive than simple lexical comparisons.

So Boogle doesn't blindly embed every query.

If lexical signals are already strong, semantic computation may not be necessary.

The search service checks whether useful document embeddings exist and whether the existing semantic/lexical signal is weak enough to justify embedding the query.

The query embedding request also has a very short timeout.

If the embedding service is slow or dead:

semantic search unavailable
↓
continue with lexical search

The search request doesn't need to fail.

That idea appears repeatedly throughout Boogle:

degrade the feature instead of killing the entire request.

Image search

Pages also contain images.

Boogle extracts them and builds a separate image index containing information such as:

image URL
page URL
page title
alt text
image title
keywords
PageRank
freshness
CTR
spam score
embedding

For semantic image similarity I experimented with CLIP.

When available, Boogle combines image and textual context roughly as:

0.7 × image embedding
+
0.3 × text embedding

and normalizes the result.

If CLIP isn't available or image fetching fails, it can fall back to text-based embeddings.

Again:

ideal path fails
↓
use weaker path
↓
still return something

What happens when you search something

This part became much more elaborate than I expected.

Suppose you type:

best golang concurrency patterns

The request hits go-search.

It doesn't simply forward that string to Meilisearch.

Roughly:

query
↓
spell check
↓
cache lookup
↓
direct-answer detection
↓
parse query intent
↓
retrieve many candidates
↓
filter garbage
↓
deduplicate
↓
semantic embedding if useful
↓
rerank
↓
domain diversity
↓
paginate
↓
related searches
↓
track impressions
↓
response

I deliberately over-fetch candidates before pagination.

If the user requests 10 results, ranking only 10 candidates would mean Meilisearch has effectively made the final decision.

Instead Boogle can retrieve a much larger candidate pool, apply its own ranking logic and only then return the requested page.

I also added spelling correction

If someone searches:

javscript promises

Boogle can inspect candidate vocabulary and use Levenshtein distance to propose a correction.

It doesn't automatically rewrite every unfamiliar word because that would destroy searches for names, acronyms and uncommon terminology.

A rewrite is only considered when the original search performs poorly and the corrected version actually produces better candidates.

That sounds obvious.

It wasn't obvious until I built it.

Domain diversity

Another surprisingly annoying problem:

1. example.com/article1
2. example.com/article2
3. example.com/article3
4. example.com/article4
5. example.com/article5

Even if all of those pages score well, the search page feels terrible.

So Boogle currently limits the number of results from the same host.

Search quality isn't purely:

sort(score)

The result set itself matters.

Search started becoming more than ten blue links

Eventually I added separate endpoints for:

/search
/search/images
/search/news
/search/videos
/autocomplete
/knowledge
/faq
/dictionary
/market/history
/images/similar

along with feedback and AI endpoints.

So the frontend can show things like:

• normal web results;
• images;
• news;
• videos;
• autocomplete;
• "Did you mean?";
• related searches;
• knowledge information;
• FAQs;
• dictionary definitions;
• market data;
• direct answers;
• similar images.

This is where I began appreciating just how much stuff exists around the actual search results on modern search engines.

Boogle learns from clicks

I also wanted ranking to change based on behaviour.

So when Boogle displays results, it records impressions.

When someone clicks a result, it records the click.

That gives me:

CTR = clicks / impressions

CTR then becomes another ranking signal.

There is also an immediate smoothed click boost:

newCTR = oldCTR × 0.9 + 0.1

and periodic backfilling updates the index.

So information flows backwards:

search
↓
results
↓
user clicks
↓
feedback
↓
MongoDB
↓
CTR calculation
↓
Meilisearch
↓
future ranking

Boogle therefore has a tiny learning loop without needing to retrain a neural network. The checked-in system writes click/impression data into MongoDB and feeds CTR updates back into the ranking index.

This is one of my favourite parts of the architecture.

Kafka entered the project too

At some point I wanted the crawler, processor and embedding pipeline to stop behaving like one giant program.

So I introduced an event pipeline.

Technically I'm using Redpanda, which exposes a Kafka-compatible interface.

The main topics include:

crawl-events
process-events
embed-events
feedback-events

The infrastructure around the project currently consists primarily of Redpanda, Redis, MongoDB and Meilisearch.

So the flow becomes something like:

Crawler
│
├──► MongoDB raw_pages
│
└──► crawl-events
│
▼
Pipeline
│
▼
process-events
│
▼
Pipeline
│
▼
embed-events
│
▼
Embedding Worker

Is Kafka necessary for my current corpus size?

No.

Could I build Boogle with dramatically less infrastructure?

Absolutely.

But Boogle stopped being purely about creating the smallest possible search engine.

It became a playground for learning systems.

Web UI

The main interface is built with Next.js.

It behaves roughly like a conventional search engine:

search box
autocomplete
tabs
results
images
news
videos
knowledge panels
related searches
pagination

Autocomplete is debounced by 300ms, and request IDs prevent stale responses from replacing newer search results when someone types quickly.

There is also a React Native mobile client consuming the same backend APIs.

One backend.

Multiple clients.

Then I started building a browser

This month I took the project one step further.

I thought:

If I already have my own search engine, why not build the thing people use to access it?

So I started Surf.

Surf is a desktop browser built with Wails and Go.

It isn't part of the crawler or ranking pipeline.

It is another client.

When you type something into the address bar, Surf has to decide:

is this a URL?
│
yes│ no
▼ ▼
navigate search Boogle

That sounds trivial until you start dealing with inputs like:

google.com
localhost:3000
192.168.1.1
golang concurrency
how does pagerank work

Surf currently has its own browser-session model covering tabs, navigation, back/forward history, loading state, zoom and session restoration. It also persists history and bookmarks, provides autocomplete based on browser data, supports request blocking and has built-in browser pages and screenshot functionality.

So Boogle is gradually becoming an ecosystem:

┌─────────────┐
│ Boogle │
│ Search │
└──────┬──────┘
│
┌─────────────────┼────────────────┐
▼ ▼ ▼
Web UI Mobile App Surf
Browser

I didn't start Boogle thinking I would eventually be writing browser tab/session logic.

Projects have a funny way of expanding.

I also experimented with my own AI layer

Because apparently building a search engine and browser wasn't enough.

There is a go-ai service which builds vocabulary and unigram/bigram/trigram statistics from documents in Boogle's own index and contains a tiny transformer experiment.

There is also a Perplexity-like retrieval service.

It retrieves relevant pages from Boogle and produces a source-grounded answer structure.

I would not call either of these a replacement for an actual modern LLM.

That would be dishonest.

They're experiments in understanding the pieces.

And that's really the philosophy of Boogle.

I'm not trying to beat Google.

I'm trying to understand what happens underneath things I normally take for granted.

Boogle isn't pretending to be production Google

There are plenty of limitations.

For example, the current public search API doesn't have proper authentication or rate limiting.

The Kafka pipeline uses duplicate suppression, but it does not provide mathematically perfect exactly-once delivery.

There are edge cases where an event can be lost between claiming a dedupe key and successfully forwarding the message.

The asynchronous embedding path also currently updates MongoDB without automatically patching the already-created Meilisearch document, meaning those embeddings aren't immediately useful to retrieval until the index is updated/rebuilt.

Some services are experiments rather than production implementations.

The thing named stable-diffusion, for example, isn't actually running Stable Diffusion right now. It procedurally generates deterministic images using Pillow.

And that's okay.

I'd rather describe what the system actually does than turn the README into marketing.

The current codebase is kind of ridiculous

The project now contains things like:

go-crawler
go-processor
go-pipeline
go-search
embed_server.py
embed_worker.py
go-ai
go-perplexity
go-image-similar
stable-diffusion
search-ui
search-mobile
surf-browser
go-debug

The backend has distinct data-building jobs and serving processes: the crawler collects data, the processor publishes it into the searchable representation, and the search/UI services serve that processed corpus.

The funny thing is that this all started with basically:

fetch(url)
parse(html)
save(page)

What building Boogle taught me

The biggest lesson wasn't PageRank.

It wasn't Go.

It wasn't Kafka.

It wasn't embeddings.

It was that the simple version of almost every system hides an enormous number of decisions.

Before Boogle, "search engine" mentally looked something like:

crawler → database → search

Now I see:

┌── URL normalization
├── crawl scheduling
├── robots/politeness
├── deduplication
├── HTML extraction
├── metadata extraction
├── content quality
├── spam detection
├── canonicalization
├── link graph
├── PageRank
THE WEB ─────────────┼── indexing
├── embeddings
├── candidate retrieval
├── query understanding
├── spelling correction
├── ranking
├── domain diversity
├── caching
├── snippets
├── feedback
├── CTR
├── images
├── knowledge
├── APIs
└── clients

And my implementation is still microscopic compared with Google.

That gives you some idea of the engineering hiding behind a search box.

Go also changed how I think about learning languages

I used to think the normal sequence was:

learn language
↓
practice language
↓
become comfortable
↓
build complicated project

Boogle was closer to:

pick complicated project
↓
hit problem
↓
learn enough Go to solve it
↓
hit next problem
↓
learn more Go
↓
repeat

GPT made that workflow much more practical.

I wasn't learning goroutines because Chapter 7 of a course told me to.

I was learning them because my crawler needed concurrency.

I wasn't learning channels because they were on a roadmap.

I needed workers to communicate.

I wasn't learning mutexes academically.

I had shared state.

The project created the curriculum.

That is probably how I want to learn more things going forward.

JavaScript → Rust → Go

Looking back, even the rewrites were useful.

JavaScript let me prototype the idea quickly.

Rust showed me that raw technical suitability isn't the only thing that matters.

Go gave me a balance I liked:

fast enough
simple enough
great concurrency
quick compilation
small binaries
good networking ecosystem
easy deployment

And most importantly:

I actually enjoyed building with it.

I started Boogle in the language I knew.

I moved to the language I thought I should use.

Then I ended up with the language that let me keep building.

There is probably a lesson in there.

AI wrote parts of Boogle. I still consider it my project.

I think we're entering a weird period in software where developers will have to become comfortable saying this.

Yes, GPT helped me.

A lot.

It taught me Go concepts.

It generated code.

It helped me debug.

It helped me think through architecture.

Sometimes it produced something wrong and I had to figure out why.

Sometimes I accepted an implementation before understanding it properly and paid for that later when something broke.

But the interesting skill is slowly shifting from:

Can you type every implementation from memory?

towards:

Can you understand a system well enough to design it, interrogate it, debug it, improve it and know when the machine is wrong?

Boogle made that distinction very obvious to me.

AI made it possible for me to attempt something much larger than I probably would have attempted otherwise.

But complexity didn't disappear.

It just moved.

Where Boogle goes from here

I don't really have some startup pitch for Boogle.

I don't expect it to replace Google.

That isn't why I built it.

I built it because search engines are fascinating.

Browsers are fascinating.

Distributed systems are fascinating.

Ranking systems are fascinating.

And there is something deeply satisfying about typing a query into a browser I built, sending it to a search API I built, searching an index created from pages crawled by my crawler, ranking them using a graph my processor constructed and seeing an actual result appear.

The entire path is mine:

URL
↓
crawler
↓
raw document
↓
event pipeline
↓
processor
↓
link graph
↓
PageRank
↓
search index
↓
embedding
↓
retrieval
↓
reranking
↓
API
↓
Boogle
↓
Surf
↓
me

And somehow all of this started because I thought:

"How hard can building a search engine be?"

Turns out, pretty hard.

Which is exactly why it became fun.