Customer Review Analysis and Complaint Routing Pipeline

Real multi-category Amazon reviews, benchmarking classical TF-IDF against a fine-tuned Transformer, with a bias check and confidence-gated routing before deployment.

6-8 hours end to end

·NLP

Problem Statement

A retailer receives thousands of product reviews and support messages daily, spanning dozens of product categories, mixing genuine praise, real complaints, and neutral feedback. A support team cannot manually read every review, but a negative review needs to reach the right category team fast — a complaint about a battery draining overnight needs electronics support, not shipping. This project builds a real classification pipeline that routes negative reviews automatically, flags low-confidence cases for human review rather than guessing, and checks directly whether the classifier treats different product categories fairly before ever being trusted with this decision.

Dataset

Amazon Product Reviews (Multi-Category)

Real customer reviews spanning multiple product categories (electronics, home goods, books, and more), each with genuine star ratings serving as a real sentiment signal rather than synthetic labels. This is the same kind of messy, real-world review data actual e-commerce and retail support teams process daily — inconsistent length, informal language, occasional sarcasm, and genuine class imbalance across categories.

~500,000 reviews sampled across 5 categories for this project, ~50-100 MB depending on category selectionAmazon Review Data, maintained for academic research use

Architecture Decisions

Following Module 9's exact honest-comparison discipline, this project does not assume a fine-tuned Transformer automatically wins — it benchmarks Module 9's TF-IDF plus logistic regression pipeline directly against Module 11's practical DistilBERT fine-tuning workflow on the identical review data, measuring accuracy, training time, and per-request inference latency side by side. The routing decision itself deliberately uses a confidence threshold rather than a flat accept-everything policy, following the same cost-aware thresholding principle used in the Deep Learning course's credit risk project — a low-confidence prediction gets flagged for human review rather than auto-routed, since a misrouted urgent complaint has a real cost. Module 17's bias measurement technique is applied directly to this classifier before deployment, checking whether identical complaint text produces different confidence scores purely because of which product category it mentions.

Built On

•Module 1 — Text Preprocessing Fundamentals, the cleaning pipeline every review passes through first
•Module 9 — Text Classification Pipelines End to End, the TF-IDF and classical classifier baseline this project benchmarks directly
•Module 11 — Fine-Tuning a Pretrained Transformer, the practical DistilBERT workflow benchmarked against the classical baseline
•Module 17 — Bias and Fairness in NLP Systems, applied directly to this classifier before trusting it with a real routing decision
•Module 18 — Building a Production NLP Pipeline, the exact FastAPI serving and monitoring pattern this project's deployment follows

Step 1 — Exploring Real, Messy Review Data

Before any model gets built, this step measures the real characteristics of the data honestly: how reviews distribute across star ratings and categories, how review length varies, and whether certain categories are meaningfully underrepresented. Following the exact same discipline as Module 9's classification pipeline and the Deep Learning course's credit risk project, this step treats class imbalance and category distribution as something to measure directly, not assume away.

Real Reviews Are Unevenly Distributed Across Categories and Ratings

Star ratings skew toward extremes (many 5-star and 1-star reviews, fewer 2-3 star), and review volume differs meaningfully across product categories — both measured directly before any modeling decision is made.

01_explore_review_data.py

python

1import pandas as pd
2import warnings
3warnings.filterwarnings("ignore")
4
5print("=== EXPLORING REAL, MULTI-CATEGORY REVIEW DATA ===\n")
6
7# In practice, this loads real category-specific review files
8# downloaded from the Amazon Review Data academic release
9reviews = pd.read_json("./amazon_reviews_sample.jsonl", lines=True)
10
11print(f"Total reviews: {len(reviews):,}")
12print(f"Categories present: {reviews['category'].nunique()}\n")
13
14print("=== STAR RATING DISTRIBUTION (a real sentiment signal) ===\n")
15rating_distribution = reviews["rating"].value_counts(normalize=True).sort_index()
16for rating, proportion in rating_distribution.items():
17    print(f"  {rating} stars: {proportion:.1%}")
18
19print("\n=== REVIEWS PER CATEGORY ===\n")
20category_counts = reviews["category"].value_counts()
21for category, count in category_counts.items():
22    print(f"  {category:>20}: {count:,} reviews")
23
24print("\n=== REVIEW LENGTH VARIATION ===\n")
25reviews["word_count"] = reviews["text"].str.split().str.len()
26print(f"Median review length: {reviews['word_count'].median():.0f} words")
27print(f"Shortest 5%: {reviews['word_count'].quantile(0.05):.0f} words")
28print(f"Longest 5%:  {reviews['word_count'].quantile(0.95):.0f} words")
29
30# Following Module 9's exact principle: define the sentiment label
31# from star rating BEFORE any modeling, and check the resulting
32# class balance directly
33reviews["sentiment"] = reviews["rating"].apply(lambda r: "negative" if r <= 2 else "positive")
34print(f"\n=== RESULTING SENTIMENT LABEL BALANCE ===\n")
35print(reviews["sentiment"].value_counts(normalize=True))
36
37print("""
38This confirms the real imbalance and category spread this project's
39classifier must handle -- informing both the class weighting used in
40Step 3's training and the per-category bias check in Step 4.
41""")

Gotchas

⚠3-star reviews are deliberately excluded from the positive/negative sentiment label here, since they genuinely represent mixed or neutral sentiment rather than clearly positive or negative — including them as either label would inject real label noise into training.
⚠Review length varies dramatically in real data, from a few words to several paragraphs — a fixed-length truncation strategy (needed for the Transformer in Step 3) will lose real content from the longest reviews, worth measuring directly rather than assuming truncation is harmless.
⚠Category distribution imbalance here is a genuine, real business fact (some product categories simply generate more reviews than others) — not a data quality problem to fix, but a real constraint the routing system must work within.

Step 2 — Building the Preprocessing and Feature Pipeline

This step applies Module 1's exact cleaning function to every review, then builds Module 9's exact TF-IDF vectorization with the correct fit-on-train-only discipline, preparing the data identically for both the classical and Transformer benchmarks in Step 3 so the comparison isolates the model choice, not a difference in preprocessing.

02_preprocessing_pipeline.py

python

1import pandas as pd
2import re
3from sklearn.model_selection import train_test_split
4from sklearn.feature_extraction.text import TfidfVectorizer
5import warnings
6warnings.filterwarnings("ignore")
7
8print("=== BUILDING THE PREPROCESSING PIPELINE ===\n")
9
10reviews = pd.read_json("./amazon_reviews_sample.jsonl", lines=True)
11reviews = reviews[reviews["rating"] != 3].copy()   # excluding neutral, per Step 1's finding
12reviews["sentiment"] = reviews["rating"].apply(lambda r: 1 if r <= 2 else 0)   # 1 = negative (the class we route)
13
14# Module 1 Lesson 1's exact cleaning function
15def clean_text(text: str) -> str:
16    text = text.lower()
17    text = re.sub(r"<[^>]+>", " ", text)
18    text = re.sub(r"\s+", " ", text).strip()
19    return text
20
21reviews["cleaned_text"] = reviews["text"].apply(clean_text)
22
23# Splitting BEFORE any vectorization, following Module 9's exact discipline
24train_df, test_df = train_test_split(
25    reviews, test_size=0.2, stratify=reviews["sentiment"], random_state=42,
26)
27
28print(f"Training reviews: {len(train_df):,}")
29print(f"Test reviews:      {len(test_df):,}")
30print(f"Training negative rate: {train_df['sentiment'].mean():.1%}")
31print(f"Test negative rate:     {test_df['sentiment'].mean():.1%}\n")
32
33# TF-IDF fit ONLY on training text, Module 9's exact discipline
34vectorizer = TfidfVectorizer(max_features=10000, stop_words="english", min_df=3)
35X_train_tfidf = vectorizer.fit_transform(train_df["cleaned_text"])
36X_test_tfidf = vectorizer.transform(test_df["cleaned_text"])
37
38print(f"TF-IDF vocabulary size: {len(vectorizer.vocabulary_)}")
39print(f"Training matrix shape: {X_train_tfidf.shape}")
40
41train_df.to_json("train_split.jsonl", orient="records", lines=True)
42test_df.to_json("test_split.jsonl", orient="records", lines=True)
43
44print("""
45Both the training and test splits are saved here, ready to be used
46IDENTICALLY by both the classical classifier and the Transformer
47fine-tuning approach in Step 3 -- ensuring the upcoming benchmark
48compares model choice alone, not differing data preparation.
49""")

Gotchas

⚠Stratified splitting (stratify=reviews['sentiment']) preserves the same negative-review rate in both train and test sets — essential given the real class imbalance measured in Step 1, since a non-stratified split risks an unlucky test set with too few negative examples to evaluate reliably.
⚠min_df=3 excludes words appearing in fewer than 3 documents, a practical noise-reduction step following Module 12's exact reasoning for the same parameter in topic modeling — removing rare typos and unusual product-specific jargon that would otherwise clutter the vocabulary.
⚠Saving identical train/test splits to disk, rather than re-splitting separately for each model benchmarked in Step 3, is essential — using different random splits for the classical and Transformer approaches would make their accuracy comparison genuinely unfair.

Step 3 — Benchmarking TF-IDF Against a Fine-Tuned Transformer

Following Module 9 Lesson 2's exact honest-comparison methodology, this step trains and evaluates both a TF-IDF plus logistic regression classifier and a fine-tuned DistilBERT model on the identical train/test split saved in Step 2, measuring accuracy, training time, and per-request inference latency for both — settling which approach is actually worth deploying with real numbers, not an assumption that the Transformer automatically wins.

Same Split, Same Test Set — Only the Model Changes

Both approaches train on the identical reviews and are evaluated on the identical held-out test set, isolating model choice as the only variable in this comparison.

03_benchmark_classical_vs_transformer.py

python

1import pandas as pd
2import time
3from sklearn.feature_extraction.text import TfidfVectorizer
4from sklearn.linear_model import LogisticRegression
5from sklearn.metrics import accuracy_score, f1_score
6import warnings
7warnings.filterwarnings("ignore")
8
9train_df = pd.read_json("train_split.jsonl", lines=True)
10test_df = pd.read_json("test_split.jsonl", lines=True)
11
12print("=== APPROACH 1: TF-IDF + LOGISTIC REGRESSION ===\n")
13
14start = time.perf_counter()
15vectorizer = TfidfVectorizer(max_features=10000, stop_words="english", min_df=3)
16X_train = vectorizer.fit_transform(train_df["cleaned_text"])
17X_test = vectorizer.transform(test_df["cleaned_text"])
18
19classifier = LogisticRegression(max_iter=1000, class_weight="balanced")
20classifier.fit(X_train, train_df["sentiment"])
21tfidf_train_time = time.perf_counter() - start
22
23start = time.perf_counter()
24tfidf_predictions = classifier.predict(X_test)
25tfidf_inference_time = (time.perf_counter() - start) / len(test_df)
26
27tfidf_accuracy = accuracy_score(test_df["sentiment"], tfidf_predictions)
28tfidf_f1 = f1_score(test_df["sentiment"], tfidf_predictions)
29
30print(f"Training time: {tfidf_train_time:.2f}s")
31print(f"Per-review inference time: {tfidf_inference_time*1000:.3f}ms")
32print(f"Accuracy: {tfidf_accuracy:.2%}")
33print(f"F1 score: {tfidf_f1:.4f}\n")
34
35print("=== APPROACH 2: FINE-TUNED DISTILBERT (MODULE 11's EXACT WORKFLOW) ===\n")
36
37try:
38    from transformers import (
39        AutoTokenizer, AutoModelForSequenceClassification,
40        TrainingArguments, Trainer,
41    )
42    from datasets import Dataset
43    import numpy as np
44    import evaluate
45
46    model_name = "distilbert-base-uncased"
47    tokenizer = AutoTokenizer.from_pretrained(model_name)
48
49    train_dataset = Dataset.from_pandas(train_df[["cleaned_text", "sentiment"]].rename(
50        columns={"cleaned_text": "text", "sentiment": "label"}
51    ))
52    test_dataset = Dataset.from_pandas(test_df[["cleaned_text", "sentiment"]].rename(
53        columns={"cleaned_text": "text", "sentiment": "label"}
54    ))
55
56    def tokenize_function(examples):
57        return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=128)
58
59    tokenized_train = train_dataset.map(tokenize_function, batched=True)
60    tokenized_test = test_dataset.map(tokenize_function, batched=True)
61
62    model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
63    accuracy_metric = evaluate.load("accuracy")
64
65    def compute_metrics(eval_prediction):
66        logits, labels = eval_prediction
67        predictions = np.argmax(logits, axis=-1)
68        return accuracy_metric.compute(predictions=predictions, references=labels)
69
70    start = time.perf_counter()
71    training_args = TrainingArguments(
72        output_dir="./review_sentiment_model", num_train_epochs=2,
73        per_device_train_batch_size=16, report_to="none",
74    )
75    trainer = Trainer(
76        model=model, args=training_args, train_dataset=tokenized_train,
77        eval_dataset=tokenized_test, compute_metrics=compute_metrics,
78    )
79    trainer.train()
80    bert_train_time = time.perf_counter() - start
81
82    results = trainer.evaluate()
83    bert_accuracy = results["eval_accuracy"]
84
85    print(f"Training time: {bert_train_time:.1f}s")
86    print(f"Accuracy: {bert_accuracy:.2%}\n")
87
88    print("=== THE HONEST COMPARISON ===\n")
89    print(f"{'Approach':>25} | {'Accuracy':>10} | {'Train Time':>12}")
90    print("-" * 55)
91    print(f"{'TF-IDF + LogReg':>25} | {tfidf_accuracy:>10.2%} | {tfidf_train_time:>11.1f}s")
92    print(f"{'Fine-tuned DistilBERT':>25} | {bert_accuracy:>10.2%} | {bert_train_time:>11.1f}s")
93
94except ImportError:
95    print("transformers/datasets not installed -- install with:")
96    print("pip install transformers datasets evaluate torch --break-system-packages")
97
98print("""
99=== THE HONEST CONCLUSION ===
100
101Report your own measured numbers. If the accuracy gap between the
102two approaches is small, TF-IDF's dramatically lower training time
103and simpler infrastructure make it the practical choice for this
104specific task. If DistilBERT's accuracy is meaningfully higher,
105that gap must be weighed against its real training and serving cost
106-- exactly Module 9 and Module 19's exact honest tradeoff framing,
107now applied to a genuine production routing decision.
108""")

Gotchas

⚠class_weight="balanced" is applied to the logistic regression specifically because Step 1 measured a real class imbalance — omitting this would bias the classifier toward the majority class, exactly the concern Module 9's classification work addresses directly.
⚠The DistilBERT approach here uses only 2 training epochs and a modest batch size specifically to keep this benchmark practical to run — a real production fine-tuning run would likely use the full dataset and more careful hyperparameter tuning, following Module 11 Lesson 2's exact practical decision-making framework.
⚠Per-request inference latency matters as much as accuracy for a real routing decision — a slightly less accurate but dramatically faster classifier may be the better practical choice for a high-volume review stream, exactly the tradeoff this step's timing measurements are designed to surface.

Step 4 — Checking for Category-Based Bias Before Trusting the Routing Decision

Following Module 17's exact controlled-experiment methodology, this step tests whether the winning classifier's confidence score changes based on which product category a complaint mentions, holding the actual complaint content identical — a genuine fairness check before this model is trusted to make real routing decisions affecting how quickly different categories' complaints get addressed.

04_bias_check_before_deployment.py

python

1import re
2from sklearn.feature_extraction.text import TfidfVectorizer
3from sklearn.linear_model import LogisticRegression
4import pandas as pd
5import warnings
6warnings.filterwarnings("ignore")
7
8print("=== MODULE 17's EXACT CONTROLLED BIAS CHECK, APPLIED HERE ===\n")
9
10train_df = pd.read_json("train_split.jsonl", lines=True)
11
12vectorizer = TfidfVectorizer(max_features=10000, stop_words="english", min_df=3)
13X_train = vectorizer.fit_transform(train_df["cleaned_text"])
14classifier = LogisticRegression(max_iter=1000, class_weight="balanced")
15classifier.fit(X_train, train_df["sentiment"])
16
17# Matched complaint templates -- ONLY the product category word changes,
18# every other word held EXACTLY identical, following Module 17 Lesson 2's
19# exact controlled-experiment design
20complaint_template = "this {category} stopped working after just two days and customer service was unhelpful"
21categories_to_test = ["laptop", "blender", "mattress", "headphones", "backpack"]
22
23print("=== TESTING IDENTICAL COMPLAINTS ACROSS DIFFERENT CATEGORIES ===\n")
24
25results = []
26for category in categories_to_test:
27    text = complaint_template.format(category=category)
28    vector = vectorizer.transform([text])
29    probability_negative = classifier.predict_proba(vector)[0][1]
30    results.append((category, probability_negative))
31    print(f"  '{text}'")
32    print(f"    -> P(negative) = {probability_negative:.4f}\n")
33
34max_prob = max(r[1] for r in results)
35min_prob = min(r[1] for r in results)
36gap = max_prob - min_prob
37
38print(f"=== THE MEASURED RESULT ===\n")
39print(f"Largest confidence gap across categories: {gap:.4f}")
40print(f"""
41Every one of these sentences describes an IDENTICAL complaint --
42only the product category word changed. If this gap is large, it
43means the classifier's confidence in flagging a complaint as
44negative depends partly on WHICH PRODUCT CATEGORY is mentioned, not
45just the actual complaint content -- a real fairness concern before
46trusting this model to prioritize which categories' complaints get
47addressed fastest.
48
49If deployed with a measurable gap like this left unaddressed, a
50category the model happens to score lower on would systematically
51receive slower complaint routing than an equally severe complaint
52in a different category -- exactly the kind of real, measurable
53harm Module 17 exists to catch before deployment, not after.
54""")

Gotchas

⚠This test uses only 5 categories and 1 complaint template for a fast, illustrative check — a genuinely thorough production bias audit would test many more templates and categories, following Module 17 Lesson 2's own gotcha about needing a statistically robust sample rather than a small, illustrative one.
⚠A measured confidence gap here does not automatically mean the model is unusable — it means the gap should be investigated and, if confirmed on a larger test, addressed through techniques like Module 17 Lesson 3's counterfactual augmentation before this model is trusted with real routing decisions at scale.
⚠This bias check specifically targets category-based confidence differences — a complete fairness audit for a real deployment would also check for other potential biases (writing style, review length, language patterns), following the same measured, non-exhaustive-but-honest approach established in Module 17.

Step 5 — Serving With Confidence-Gated Routing and Monitoring

Following Module 18's exact production pipeline pattern, this final step wraps the winning classifier in a FastAPI server that routes clearly negative reviews automatically, flags low-confidence predictions for human review rather than guessing, and logs every request for the same kind of operational monitoring built in Module 18 Lesson 2.

05_serve_with_confidence_gating.py

python

1from fastapi import FastAPI
2from pydantic import BaseModel
3from datetime import datetime
4import re
5import joblib
6
7app = FastAPI(title="Review Routing API")
8
9vectorizer = None
10classifier = None
11request_log = []
12
13CONFIDENCE_THRESHOLD = 0.75   # below this, route to human review instead of auto-routing
14
15@app.on_event("startup")
16def load_model():
17    global vectorizer, classifier
18    vectorizer = joblib.load("review_vectorizer.pkl")
19    classifier = joblib.load("review_classifier.pkl")
20    print("Review routing model loaded.")
21
22def clean_text(text: str) -> str:
23    text = text.lower()
24    text = re.sub(r"<[^>]+>", " ", text)
25    text = re.sub(r"\s+", " ", text).strip()
26    return text
27
28class ReviewRequest(BaseModel):
29    text: str
30    category: str
31
32@app.post("/route-review")
33def route_review(request: ReviewRequest):
34    cleaned = clean_text(request.text)
35    vector = vectorizer.transform([cleaned])
36    probability_negative = float(classifier.predict_proba(vector)[0][1])
37
38    if probability_negative < 0.5:
39        decision = "no_action_needed"
40    elif probability_negative >= CONFIDENCE_THRESHOLD:
41        decision = f"route_to_{request.category}_support"
42    else:
43        decision = "flag_for_human_review"   # low confidence, don't guess
44
45    result = {
46        "category": request.category,
47        "probability_negative": round(probability_negative, 4),
48        "decision": decision,
49        "threshold_used": CONFIDENCE_THRESHOLD,
50    }
51
52    request_log.append({"timestamp": datetime.now().isoformat(), **result})
53    return result
54
55@app.get("/monitoring/category-routing-rates")
56def category_routing_rates():
57    """A real operational question: is any category's routing rate
58    unusually high, following Module 18 Lesson 2's exact monitoring
59    principle, now applied to this project's specific decision."""
60    if not request_log:
61        return {"message": "no requests logged yet"}
62
63    from collections import defaultdict
64    category_counts = defaultdict(lambda: {"total": 0, "routed": 0})
65    for entry in request_log:
66        category_counts[entry["category"]]["total"] += 1
67        if entry["decision"].startswith("route_to"):
68            category_counts[entry["category"]]["routed"] += 1
69
70    return {
71        category: {"routing_rate": round(counts["routed"] / counts["total"], 4), **counts}
72        for category, counts in category_counts.items()
73    }
74
75# Run with: uvicorn 05_serve_with_confidence_gating:app --host 0.0.0.0 --port 8000

Gotchas

⚠The confidence threshold (0.75) determines how often a review gets auto-routed versus flagged for human review — this specific value should be tuned using a real cost matrix (cost of a misrouted urgent complaint versus cost of an unnecessary human review), following the exact cost-based thresholding principle used in the Deep Learning course's credit risk project, not chosen arbitrarily.
⚠This lesson's request_log is in-memory and illustrative, following the same explicit caveat as Module 18 Lesson 2 — a real production deployment needs a persistent logging store, not a list that resets on server restart.
⚠The category_routing_rates monitoring endpoint directly extends the bias check from Step 4 into ongoing production monitoring — a category whose routing rate looks unusual over real time is exactly the kind of signal that should trigger re-running Step 4's bias check on fresh data.