Personal Knowledge Base Search
A search tool built to demonstrate relevance ranking from first principles — no LLM in the loop. It indexes real Wikipedia articles via the MediaWiki API and lets you compare two ranking strategies side by side: TF-IDF keyword matching against LSA semantic/concept matching.
📊 Project Summary
- Ingests a topically diverse corpus of real Wikipedia articles — science, technology, history, arts, geography — via the MediaWiki API, then builds two independent search indexes over it.
- TF-IDF index: a sparse, keyword-weighted vector per document (scikit-learn
TfidfVectorizer), ranked by cosine similarity to the query — rewards exact vocabulary overlap. - LSA index: the same TF-IDF matrix projected into a lower-dimensional topic space with
TruncatedSVD, so documents can match a query on shared concept even without shared exact wording. - FastAPI backend exposes
GET /search?q=...&method=tfidf|semantic, plus/statsand interactive Swagger docs at/docs. - Streamlit frontend with a "Compare both" view that runs the same query through both rankers at once, so the difference in results is visible directly, not just described.
🧰 Tools Used
- Python, FastAPI, Uvicorn — backend API
- scikit-learn — TF-IDF vectorization, TruncatedSVD (LSA), cosine similarity
- Streamlit — frontend
- Wikipedia (MediaWiki) API — data source
📷 Screenshots
⚙️ How the Ranking Works
- TF-IDF: each document and query becomes a sparse vector weighted by term frequency and inverse document frequency; relevance is the cosine similarity between them. Rewards exact vocabulary overlap.
- LSA: the TF-IDF matrix is projected into a lower-dimensional topic space with
TruncatedSVD. Cosine similarity in that reduced space captures documents that are conceptually related even when they don't share exact terms — e.g. a query for "digital money without banks" surfacing an article about decentralized currency without either phrase appearing verbatim.