Implementing Semantic Discussion Clustering Using TF-IDF Instead of Vector Embeddings

Semantic Discussion Clustering Without Embeddings

Mervin developed a system to monitor product trends and bugs across platforms like Reddit and GitHub. The architecture replaces expensive vector search with a TF-IDF and Cosine Similarity pipeline.

Why This Matters

Modern NLP often defaults to high-cost stacks involving OpenAI embeddings, Pinecone, or Weaviate. However, for discussion monitoring where specific keywords (e.g., ‘rate limits’, ‘api quota’) dominate the signal, traditional TF-IDF is often sufficient and can run on a cheap VPS, eliminating the financial overhead of vector databases.

Key Insights

• Similarity Thresholding: A threshold of 0.25 was identified as the optimal balance, whereas 0.35 was too strict and 0.15 too loose.
• Keyword Signal: Technical discussions often rely on repeated terms like ‘billing’ or ‘token cost’, making TF-IDF effective for grouping.
• Performance Stack: The system utilizes BullMQ for queueing, PostgreSQL for storage, and Groq for fast, low-cost summary generation.

Practical Applications

•  Use case: Product monitoring systems tracking API throttling complaints via automated clustering. Pitfall: Setting similarity thresholds too high (e.g., 0.35), resulting in every thread becoming its own cluster.
•  Use case: Trend detection across Reddit and Hacker News using Groq for rapid summarization. Pitfall: Over-reliance on expensive LLM embeddings for simple keyword-heavy datasets, leading to unnecessary infrastructure costs.

References:

• https://dev.to/mervindublin/how-i-built-semantic-discussion-clustering-without-embeddings-and-why-it-was-good-enough-21dl
• https://github.com/melyx-id/discussion-radar