Boogle search engine

• Boogle Search

A Distributed Search Engine Built from Scratch

Boogle Search is a distributed, Google-style search engine built from the ground up using Go, Python, Next.js, Kafka-compatible event streaming, Redis, MongoDB, and Meilisearch.

The platform crawls websites, extracts and cleans page content, removes duplicates, analyzes link relationships, generates semantic embeddings, ranks documents, indexes pages and images, and serves results through a modern search interface.

Beyond traditional web search, it supports image search, semantic ranking, knowledge panels, FAQs, dictionary results, market information, AI-assisted answers, and image similarity search.

The Problem

Building a search engine involves much more than matching a query against stored text.

The system must:

• Discover and crawl pages efficiently
• Avoid repeatedly processing the same URLs
• Extract meaningful content from noisy HTML
• Understand relationships between pages
• Rank relevant and trustworthy results
• Support both keyword and semantic search
• Process large workloads asynchronously
• Serve results with low latency
• Collect feedback to improve ranking
• Present multiple search experiences through one interface

I built Boogle Search to understand and implement the complete lifecycle of a search result, from crawling a URL to displaying it on a search engine results page.

The Solution

I designed the platform as a collection of independently running services connected through an event-driven data pipeline.

The main flow is:

Crawler → Event Pipeline → Processor → Embedding Service → Search Index → Search API → Web Interface

Each service is responsible for a specific part of the system and can be developed, scaled, and deployed independently.

Web Crawler

The crawler is written in Go and uses a breadth-first crawling strategy.

It supports:

• Concurrent crawling across multiple domains
• Per-domain queues
• Global fetch limits
• URL normalization
• Tracking-parameter removal
• Content-type validation
• Retry handling with backoff
• Main-content extraction
• Metadata and structured-data extraction
• Link and image discovery
• Redis-based URL deduplication
• Batched MongoDB persistence
• Kafka event publishing

The crawler extracts page titles, body content, links, anchor text, images, Open Graph metadata, Twitter metadata, JSON-LD schema information, and keywords.

It also filters thin pages, blocked domains, irrelevant paths, unsupported file types, and low-quality content before sending pages further into the pipeline.

Event-Driven Processing

The system uses Redpanda, a Kafka-compatible message broker, to connect services asynchronously.

The pipeline includes topics for:

• Crawled pages
• Pages ready for processing
• Pages ready for embedding
• Search feedback events

A dedicated Go pipeline service routes events between stages and uses Redis to prevent duplicate URLs from being processed repeatedly.

This architecture keeps the crawler independent from slower operations such as ranking, indexing, and embedding generation.

Page Processing and Ranking

The processor reads raw pages from MongoDB and transforms them into searchable documents.

It performs:

• Text cleanup and normalization
• Boilerplate removal
• Keyword extraction
• Spam detection
• Content-quality scoring
• Freshness scoring
• Canonical URL generation
• Link-graph construction
• Backlink calculation
• Mention analysis
• PageRank computation
• Composite authority scoring
• Search-index generation

The ranking pipeline combines PageRank, backlinks, mentions, freshness, content quality, keyword relevance, semantic similarity, click-through rate, domain trust, and spam penalties.

This creates a more realistic search-ranking model than simply sorting results by keyword matches.

Semantic Search

A Python embedding service generates vector representations for pages using a sentence-transformer model.

The system supports:

• Text embeddings
• Image embeddings through CLIP
• Batched embedding generation
• LRU embedding caching
• Asynchronous Kafka-based embedding workers
• Semantic similarity scoring
• Hybrid keyword and vector ranking

Search results combine traditional lexical relevance with semantic similarity, allowing the engine to return pages that are conceptually relevant even when they do not contain the exact search terms.

Search API

The search gateway is written in Go and exposes APIs for:

• Web search
• Image search
• News search
• Video search
• Autocomplete
• Knowledge panels
• FAQs
• Dictionary definitions
• Market history
• Similar-image discovery
• AI chat
• Perplexity-style answers
• Search click and impression feedback

The API performs query parsing, stemming, stop-word removal, phrase detection, keyword scoring, semantic scoring, domain-trust analysis, spam filtering, snippet generation, pagination, and query correction.

It also supports direct answers for queries such as conversions, market prices, and time-based questions.

Search Ranking

The final search score combines multiple relevance and authority signals:

• Keyword relevance
• Semantic similarity
• PageRank
• Backlinks
• Mentions
• TF-IDF-style relevance
• Query-token coverage
• Freshness
• Content quality
• Click-through rate
• Domain trust
• Phrase matches
• Title matches
• Term proximity
• Spam penalties

Separate ranking models are used for regular pages, images, and visually similar images.

Feedback and CTR Learning

The platform records result impressions and clicks.

User feedback is:

• Sent to Kafka
• Stored in MongoDB
• Used to calculate click-through rates
• Written back into the search index
• Included as a ranking signal

A background process periodically recalculates result CTR values, allowing ranking to adapt based on actual user interaction.

Image Search and Similarity

The processor extracts images from crawled pages and stores them as independent searchable documents.

Image search uses:

• Alt text
• Page titles
• Keywords
• Image embeddings
• Page authority
• Freshness
• CTR
• Spam filtering

The image-similarity feature compares vector similarity, surrounding text, page authority, and source-domain relationships to identify visually or contextually related images.

Search Interface

The frontend is built using Next.js, React, TypeScript, and Tailwind CSS.

It recreates a Google-style search experience with:

• Web, image, news, video, and AI tabs
• Autocomplete suggestions
• Highlighted result snippets
• Knowledge panels
• Dictionary cards
• Market charts
• “People also ask” sections
• AI-generated answers
• Image-generation tools
• Image preview modals
• Similar-image results
• Result pagination
• Click and impression tracking

The application includes request-race protection, debounced autocomplete, responsive layouts, and multiple search verticals connected to the same backend gateway.

AI Features

The platform includes several experimental AI services.

Local Language Model

I built a standalone Go service using n-gram generation and a small transformer implementation created from scratch.

It trains on indexed search documents and combines generated text with retrieved evidence from the search index.

Retrieval-Based Answer Engine

A separate Perplexity-style service retrieves relevant documents and returns a concise answer with cited source links.

Image Generation

The platform includes a local procedural image-generation service with a Stable-Diffusion-compatible API structure. It creates deterministic images from prompt-derived palettes, gradients, noise, shapes, and keyword-based scene composition.

Infrastructure and Deployment

The full platform can run locally through Docker Compose and includes deployment configurations for Kubernetes and Helm.

The infrastructure includes:

• Redis
• Redpanda
• MongoDB
• Meilisearch
• Go microservices
• Python embedding services
• Next.js frontend
• Kubernetes deployments
• Horizontal pod autoscaling
• Helm deployment templates
• Persistent local development volumes
• Cross-platform orchestration scripts

The services can be started together through Windows PowerShell or Bash orchestration scripts.

Technical Highlights

• Distributed crawler and search architecture
• Go-based concurrent services
• Kafka-compatible event streaming
• Redis-based deduplication and caching
• MongoDB document storage
• Meilisearch full-text indexing
• PageRank and link-graph analysis
• Hybrid lexical and semantic ranking
• Sentence-transformer embeddings
• CLIP-based image embeddings
• Real-time feedback and CTR updates
• Multiple search verticals
• Local AI experimentation
• Docker, Kubernetes, and Helm deployment support

Tech Stack

Frontend: Next.js, React, TypeScript, Tailwind CSS

Backend: Go, Python, FastAPI

Search and Storage: Meilisearch, MongoDB, Redis

Messaging: Redpanda, Kafka APIs

AI and Machine Learning: Sentence Transformers, PyTorch, CLIP, custom n-gram model, custom transformer

Infrastructure: Docker Compose, Kubernetes, Helm

My Contributions

I designed and developed the system across the complete search-engine lifecycle.

My work included:

• Building the concurrent web crawler
• Implementing URL normalization and crawl deduplication
• Creating the event-driven Kafka pipeline
• Developing content cleaning and enrichment logic
• Implementing PageRank and authority scoring
• Building hybrid keyword and semantic ranking
• Developing the embedding services
• Indexing pages and images in Meilisearch
• Building the Go search API and service gateway
• Implementing search feedback and CTR processing
• Creating image-search and image-similarity features
• Developing AI-assisted search experiences
• Building the Next.js search interface
• Creating Docker, Kubernetes, and Helm configurations
• Building debugging and graph-visualization tools

Outcome

Boogle Search demonstrates my ability to design and build a distributed system involving crawling, asynchronous processing, information retrieval, semantic search, ranking algorithms, machine learning, search analytics, and cloud-native deployment.

Rather than relying on an existing search API, the project implements the complete pipeline from discovering web pages to ranking and displaying search results.