Internal Knowledge Base Semantic Search and Q&A Engine

A real semantic search system over genuine technical documentation, built entirely from embeddings and similarity ranking — proving retrieval works before any generative layer ever gets involved.

6-8 hours end to end

·NLP

Problem Statement

A company has hundreds of internal documents — policies, FAQs, technical documentation — and employees waste real time searching through them with exact keyword search that misses relevant results whenever their query wording doesn't match a document's exact phrasing. This is precisely the gap Module 14 measured directly with a hand-built example; this project builds the complete, real system at genuine scale: indexing a real document collection, embedding real queries, ranking by semantic similarity, and extracting the actual answer span from the best-matching document — a complete search and answer system built entirely from this course's own techniques, with no external orchestration framework involved.

Dataset

Technical Q&A and Documentation Corpus

A real collection of technical questions and answers, structured similarly to how internal company documentation and FAQ systems are organized — genuine questions paired with genuine, sometimes lengthy explanatory answers, with real variation in how a question can be phrased differently from the document that actually answers it.

~20,000 question-answer pairs, ~50 MBStack Exchange Data Dump, released publicly under Creative Commons license

Architecture Decisions

This project is built entirely from techniques already proven inside this NLP course, deliberately without any external agent or chaining framework, since that orchestration layer belongs to a separate course. The core engine is Module 14's exact embed-documents-once, embed-query-at-search-time, rank-by-similarity mechanism, scaled from a 5-document illustration to a genuine multi-thousand-document index. Module 12's topic modeling is applied first to automatically group the document collection into browsable categories, since a real knowledge base benefits from both search AND browsing. Once a search ranks the best-matching document, this project adds one further, genuinely new capability beyond Module 14: extractive answer highlighting, using sentence-level embedding similarity to identify and return the SPECIFIC sentence within the best-matching document that most directly answers the query, rather than returning the entire document and leaving the user to find the answer themselves.

Built On

•Module 12 — Topic Modeling, used to automatically categorize the document collection for browsing alongside search
•Module 14 — Semantic Search and Sentence Embeddings, the exact embed-rank-retrieve mechanism this project scales to a real document collection
•Module 1 Lesson 2 — Sentence tokenization, used to split documents into individual sentences for extractive answer highlighting
•Module 18 — Building a Production NLP Pipeline, the exact FastAPI serving pattern this project's final system follows

Step 1 — Indexing a Real Document Collection Once

Following Module 14 Lesson 2's exact principle that document embeddings should be computed once and reused for every future query, this step builds the real indexing pipeline: loading a genuine multi-thousand-document collection, embedding every document exactly once, and saving the resulting embeddings to disk so they never need recomputing at actual search time.

Index Once, Search Many Times

Every document is embedded exactly once and saved. At search time, only the incoming query needs embedding — the expensive indexing step never repeats, exactly Module 14's core efficiency principle at real scale.

01_indexing_document_collection.py

python

1import pandas as pd
2import numpy as np
3from sentence_transformers import SentenceTransformer
4import warnings
5warnings.filterwarnings("ignore")
6
7print("=== INDEXING A REAL, MULTI-THOUSAND-DOCUMENT COLLECTION ===\n")
8
9qa_data = pd.read_json("./technical_qa_corpus.jsonl", lines=True)
10print(f"Total question-answer documents: {len(qa_data):,}\n")
11
12# Each "document" is the ANSWER text -- what a search should actually
13# retrieve and let the user read
14documents = qa_data["answer_text"].tolist()
15
16model = SentenceTransformer("all-MiniLM-L6-v2")
17
18print("Embedding every document ONCE -- this is the expensive step,")
19print("paid exactly one time, following Module 14 Lesson 2's exact principle.\n")
20
21document_embeddings = model.encode(documents, show_progress_bar=True, batch_size=64)
22
23print(f"\nEmbedding matrix shape: {document_embeddings.shape}")
24print(f"Each of {len(documents):,} documents compressed into a "
25      f"{document_embeddings.shape[1]}-dimensional vector.\n")
26
27# Saving to disk -- these embeddings are reused for EVERY future query,
28# never recomputed at actual search time
29np.save("document_embeddings.npy", document_embeddings)
30qa_data.to_json("indexed_documents.jsonl", orient="records", lines=True)
31
32print("""
33Saved document_embeddings.npy and indexed_documents.jsonl -- the
34complete search index this project's remaining steps build on. A
35real search request will only ever need to embed the incoming
36QUERY and compare it against this already-computed matrix, exactly
37Module 14 Lesson 2's core efficiency principle, now proven at
38genuine multi-thousand-document scale rather than a 5-document
39illustration.
40""")

Gotchas

⚠batch_size=64 processes documents in batches during embedding rather than one at a time — a real, practical efficiency detail at this scale, since embedding 20,000 documents individually would be measurably slower than batching them, though the resulting embeddings themselves are identical either way.
⚠This index must be REBUILT whenever documents are added, removed, or edited — it is not automatically kept in sync with the underlying document collection, a real operational consideration Step 5's serving system needs to account for.
⚠Saving embeddings as a plain NumPy array works cleanly at this project's scale (tens of thousands of documents) — a collection with millions of documents would need a dedicated vector database or approximate nearest-neighbor index instead, following the exact same scaling caution Module 14 Lesson 2 raised directly.

Step 2 — Auto-Categorizing the Collection for Browsing

This step applies Module 12 Lesson 3's exact BERTopic technique to automatically group the indexed document collection into browsable categories, since a real knowledge base benefits from both search AND the ability to browse by topic without typing a query at all.

02_auto_categorizing_with_bertopic.py

python

1import pandas as pd
2from bertopic import BERTopic
3import warnings
4warnings.filterwarnings("ignore")
5
6print("=== AUTO-CATEGORIZING THE COLLECTION, MODULE 12 LESSON 3's EXACT TECHNIQUE ===\n")
7
8indexed_documents = pd.read_json("indexed_documents.jsonl", lines=True)
9documents = indexed_documents["answer_text"].tolist()
10
11# BERTopic determines a reasonable topic count automatically from the
12# embeddings' actual structure, exactly Module 12 Lesson 3's advantage
13# over LDA's required topic count
14topic_model = BERTopic(min_topic_size=30, verbose=False)
15topics, probabilities = topic_model.fit_transform(documents)
16
17indexed_documents["topic_id"] = topics
18
19topic_info = topic_model.get_topic_info()
20print("=== AUTOMATICALLY DISCOVERED CATEGORIES ===\n")
21print(topic_info[["Topic", "Count", "Name"]].head(10).to_string(index=False))
22
23indexed_documents.to_json("indexed_documents.jsonl", orient="records", lines=True)
24topic_model.save("topic_model")
25
26print(f"""
27=== WHY THIS MATTERS DIRECTLY ===
28
29Every document now has a topic_id assigned, letting the final
30system offer BOTH a search bar (Step 1's embeddings) AND a
31browsable category list (this step's automatic topics) -- two
32genuinely different, complementary ways to reach the same
33underlying documents, exactly what a real internal knowledge base
34tool needs to be genuinely useful for different user habits.
35""")

Gotchas

⚠min_topic_size=30 is set higher here than Module 12 Lesson 3's original example, since this project's document collection is meaningfully larger — a real production tuning pass would test several values directly, following the exact same measured tradeoff Module 12 Lesson 2 established for topic count selection.
⚠BERTopic can assign topic -1 to documents it treats as noise/outliers, exactly as Module 12 Lesson 3 noted — these documents remain fully searchable through Step 1's embedding search even without a clean topic category, so nothing is lost, only left uncategorized for the browsing view specifically.
⚠This categorization step is entirely independent of Step 1's search index — a document's embedding vector (used for search) and its assigned topic (used for browsing) serve two different, complementary user needs and neither depends on the other.

Step 3 — Extractive Answer Highlighting Within the Best-Matching Document

This step goes one step beyond Module 14's original search-and-retrieve mechanism: once the best-matching document is found, this step splits that document into individual sentences using Module 1 Lesson 2's proper tokenization, embeds each sentence, and returns the single sentence with the highest similarity to the query — directly highlighting the actual answer rather than making the user read the entire document to find it.

03_extractive_answer_highlighting.py

python

1from sentence_transformers import SentenceTransformer
2from sklearn.metrics.pairwise import cosine_similarity
3import nltk
4import numpy as np
5import pandas as pd
6import warnings
7warnings.filterwarnings("ignore")
8
9nltk.download("punkt_tab", quiet=True)
10from nltk.tokenize import sent_tokenize
11
12print("=== EXTRACTIVE ANSWER HIGHLIGHTING WITHIN A MATCHED DOCUMENT ===\n")
13
14model = SentenceTransformer("all-MiniLM-L6-v2")
15
16def highlight_best_sentence(query: str, document_text: str) -> dict:
17    """Splits a matched document into sentences, Module 1 Lesson 2's
18    exact proper tokenization, then finds the SINGLE sentence most
19    similar to the query -- extending Module 14's document-level
20    search down to the sentence level."""
21    sentences = sent_tokenize(document_text)
22    if not sentences:
23        return {"sentence": None, "similarity": 0.0}
24
25    query_embedding = model.encode([query])
26    sentence_embeddings = model.encode(sentences)
27
28    similarities = cosine_similarity(query_embedding, sentence_embeddings)[0]
29    best_index = int(np.argmax(similarities))
30
31    return {
32        "sentence": sentences[best_index],
33        "similarity": round(float(similarities[best_index]), 4),
34        "sentence_position": best_index,
35        "total_sentences": len(sentences),
36    }
37
38# A realistic, longer document -- exactly the kind where reading the
39# WHOLE thing to find the answer wastes real time
40sample_document = """
41Setting up a virtual environment in Python is considered a best practice for
42managing project dependencies. There are several tools available for this
43purpose, including venv, virtualenv, and conda. The built-in venv module,
44available since Python 3.3, is sufficient for most standard projects. To
45create a new virtual environment, run python -m venv myenv in your terminal.
46Activate it using source myenv/bin/activate on Mac or Linux, or
47myenv\Scripts\activate on Windows. Once activated, any packages installed
48with pip will be isolated to that specific environment, preventing conflicts
49between different projects' dependencies.
50"""
51
52query = "how do I activate a virtual environment on windows"
53
54result = highlight_best_sentence(query, sample_document)
55
56print(f"Query: '{query}'\n")
57print(f"Full matched document ({result['total_sentences']} sentences total):")
58print(sample_document.strip())
59print(f"\n=== HIGHLIGHTED ANSWER (sentence {result['sentence_position']+1} of {result['total_sentences']}) ===\n")
60print(f"'{result['sentence']}'")
61print(f"Similarity to query: {result['similarity']}")
62
63print("""
64=== WHY THIS MATTERS DIRECTLY ===
65
66Rather than returning the entire multi-sentence document and making
67the user read all of it to find the Windows-specific activation
68command, this step correctly surfaces the ONE sentence that
69actually answers the question -- a genuine, measurable improvement
70in how quickly a real user gets their actual answer.
71""")

Gotchas

⚠This technique returns the single highest-similarity sentence, which works well when an answer is genuinely self-contained in one sentence — a question whose answer spans two or three connected sentences would need this technique extended to return a small window of sentences rather than just one, a real, honest limitation worth checking directly on your own document collection.
⚠Embedding every sentence in a document at query time, as done here, adds real computational cost compared to Step 1's pure document-level search — for a production system handling high query volume, pre-computing sentence-level embeddings for every document during indexing (following Step 1's exact once-only principle) would be the more efficient approach.
⚠This is extractive highlighting, not generative answering — the returned sentence is guaranteed to be an exact quote from the source document, following the same factual-faithfulness guarantee Module 13 Lesson 1's extractive summarization provides, deliberately avoiding the hallucination risk Module 13 Lesson 2 and Module 21 measured directly for generative approaches.

Step 4 — Measuring Real Search Quality on Vocabulary-Mismatched Queries

Following Module 14 Lesson 2's exact honest-comparison methodology, this step tests the complete search system directly against a set of real queries deliberately phrased using different vocabulary than the actual matching documents, measuring whether this project's embedding-based approach genuinely outperforms simple keyword search at real scale, not just on one hand-picked example.

04_measuring_real_search_quality.py

python

1import pandas as pd
2import numpy as np
3from sentence_transformers import SentenceTransformer
4from sklearn.feature_extraction.text import TfidfVectorizer
5from sklearn.metrics.pairwise import cosine_similarity
6import warnings
7warnings.filterwarnings("ignore")
8
9print("=== MEASURING SEARCH QUALITY: EMBEDDINGS vs TF-IDF, AT REAL SCALE ===\n")
10
11indexed_documents = pd.read_json("indexed_documents.jsonl", lines=True)
12document_embeddings = np.load("document_embeddings.npy")
13documents = indexed_documents["answer_text"].tolist()
14
15model = SentenceTransformer("all-MiniLM-L6-v2")
16
17# TF-IDF search, following Module 3's exact technique, for a genuine comparison
18tfidf_vectorizer = TfidfVectorizer(stop_words="english", max_features=20000)
19document_tfidf_vectors = tfidf_vectorizer.fit_transform(documents)
20
21# Real test queries, deliberately phrased using DIFFERENT vocabulary than
22# the actual correct answer's likely wording -- following Module 14
23# Lesson 2's exact vocabulary-mismatch testing principle
24test_queries_with_known_answers = [
25    ("how do I make my python code run without errors when there's a bug", 42),
26    ("what is the fastest way to search through a sorted list", 118),
27    ("my program crashes with a memory issue, what's happening", 205),
28]
29
30def tfidf_search_top_result(query: str) -> int:
31    query_vector = tfidf_vectorizer.transform([query])
32    similarities = cosine_similarity(query_vector, document_tfidf_vectors)[0]
33    return int(np.argmax(similarities))
34
35def semantic_search_top_result(query: str) -> int:
36    query_embedding = model.encode([query])
37    similarities = cosine_similarity(query_embedding, document_embeddings)[0]
38    return int(np.argmax(similarities))
39
40print(f"{'Query (vocabulary-mismatched)':>60} | {'TF-IDF top match':>16} | {'Semantic top match':>18} | {'Correct doc':>12}")
41print("-" * 115)
42
43tfidf_correct = 0
44semantic_correct = 0
45
46for query, true_document_index in test_queries_with_known_answers:
47    tfidf_result = tfidf_search_top_result(query)
48    semantic_result = semantic_search_top_result(query)
49
50    tfidf_correct += (tfidf_result == true_document_index)
51    semantic_correct += (semantic_result == true_document_index)
52
53    print(f"{query[:57]:>60} | {tfidf_result:>16} | {semantic_result:>18} | {true_document_index:>12}")
54
55print(f"\n=== THE MEASURED RESULT ===\n")
56print(f"TF-IDF correctly retrieved the right document: {tfidf_correct}/{len(test_queries_with_known_answers)}")
57print(f"Semantic search correctly retrieved the right document: {semantic_correct}/{len(test_queries_with_known_answers)}")
58
59print(f"""
60=== THE HONEST CONCLUSION ===
61
62Report your own measured numbers on your real document collection.
63This directly extends Module 14 Lesson 2's small, illustrative
645-document test to genuine scale -- confirming (or, honestly,
65potentially disconfirming on some specific queries) whether semantic
66search's advantage holds up when there are thousands of candidate
67documents to rank, not just five, exactly the kind of real,
68measured verification this course applies throughout rather than
69assuming a small-scale finding automatically generalizes.
70""")

Gotchas

⚠This test uses only 3 queries with manually identified correct answers for illustration — a genuinely rigorous evaluation of a real search system needs many more labeled query-document pairs, ideally using a proper information retrieval metric like Mean Reciprocal Rank rather than simple top-1 accuracy, to give a statistically robust measurement.
⚠Identifying the 'true' correct document index for a real query requires manual verification against the actual document collection — this lesson's example indices are illustrative placeholders standing in for genuine, manually-confirmed correct answers a real evaluation would establish first.
⚠At genuine scale (20,000 documents versus Module 14's original 5), computing cosine similarity against every single document on every query becomes noticeably slower — this is precisely the scaling limitation Module 14 Lesson 2 flagged directly, and a real production system beyond this project's scope would use an approximate nearest-neighbor index to keep search fast at much larger scale.

Step 5 — Serving the Complete Search and Q&A Engine

This closing step wraps the complete system — semantic search, topic browsing, and extractive answer highlighting — in a FastAPI server following Module 18's exact load-once pattern, giving a real, callable search and question-answering API built entirely from this course's own techniques.

05_serving_search_and_qa_engine.py

python

1from fastapi import FastAPI
2from pydantic import BaseModel
3from sentence_transformers import SentenceTransformer
4from sklearn.metrics.pairwise import cosine_similarity
5import nltk
6import numpy as np
7import pandas as pd
8
9app = FastAPI(title="Internal Knowledge Base Search API")
10
11model = None
12indexed_documents = None
13document_embeddings = None
14
15@app.on_event("startup")
16def load_search_index():
17    global model, indexed_documents, document_embeddings
18    model = SentenceTransformer("all-MiniLM-L6-v2")
19    indexed_documents = pd.read_json("indexed_documents.jsonl", lines=True)
20    document_embeddings = np.load("document_embeddings.npy")
21
22    nltk.download("punkt_tab", quiet=True)
23
24    print(f"Search index loaded: {len(indexed_documents):,} documents ready.")
25
26class SearchRequest(BaseModel):
27    query: str
28    top_k: int = 3
29
30@app.post("/search")
31def search(request: SearchRequest):
32    from nltk.tokenize import sent_tokenize
33
34    query_embedding = model.encode([request.query])
35    similarities = cosine_similarity(query_embedding, document_embeddings)[0]
36    ranked_indices = similarities.argsort()[::-1][:request.top_k]
37
38    results = []
39    for index in ranked_indices:
40        document_text = indexed_documents.iloc[index]["answer_text"]
41        topic_id = int(indexed_documents.iloc[index]["topic_id"])
42
43        # Extractive highlighting, Step 3's exact technique, applied
44        # to each returned result
45        sentences = sent_tokenize(document_text)
46        sentence_embeddings = model.encode(sentences)
47        sentence_similarities = cosine_similarity(query_embedding, sentence_embeddings)[0]
48        best_sentence_index = int(np.argmax(sentence_similarities))
49
50        results.append({
51            "document_similarity": round(float(similarities[index]), 4),
52            "topic_id": topic_id,
53            "highlighted_answer": sentences[best_sentence_index],
54            "full_document": document_text,
55        })
56
57    return {"query": request.query, "results": results}
58
59@app.get("/browse/topics")
60def browse_topics():
61    """The complementary browsing path, Step 2's automatic categories,
62    for users who want to explore rather than search directly."""
63    topic_counts = indexed_documents["topic_id"].value_counts().to_dict()
64    return {"topics": [{"topic_id": int(k), "document_count": int(v)} for k, v in topic_counts.items()]}
65
66# Run with: uvicorn 05_serving_search_and_qa_engine:app --host 0.0.0.0 --port 8000

Gotchas

⚠This endpoint recomputes sentence-level embeddings for each of the top_k results on every single search request — following Step 3's exact honest gotcha, a real high-traffic production system should pre-compute sentence-level embeddings during indexing rather than at query time, a genuine efficiency improvement beyond this project's illustrative serving example.
⚠The /browse/topics endpoint is deliberately independent of the /search endpoint, giving users two genuinely separate ways to reach the same document collection — this reflects Step 2's exact reasoning that search and browsing serve different real user habits, not one replacing the other.
⚠This system returns the full_document alongside the highlighted_answer specifically so a user can verify the highlighted sentence in its real context — following the same transparency principle as Module 13's extractive summarization, where every claim can be traced back to its exact source.