Transforming RAG Search into an Answer Engine with Gemma 4

My Bookmark Engine Returned Chunks. I Added One Endpoint to Make It Answer.

Daniel Nwaneri and Alex She developed a search engine indexing 50,000 saved tweets. The system utilizes hybrid retrieval combining BM25 keyword search and vector search reranked by a cross-encoder.

Why This Matters

The transition from returning ranked chunks to generating answers reveals the critical dependency between retrieval precision and synthesis quality. While the model can synthesize effectively, low-dimensional embeddings (such as bge-small at 384 dimensions) create a performance ceiling, resulting in ‘thin’ answers when the retrieved context lacks substantive depth despite being technically grounded.

Key Insights

• Token Budget Constraints: Using max_tokens: 512 resulted in empty answers because Gemma 4 is a thinking model that exhausts small budgets on internal reasoning; increasing to 2048 fixed this.
• Compound Intelligence: The system surfaces ‘reflection’ type entries—previously generated insights stored back into the index—allowing the engine to synthesize both raw content and its own prior conclusions.
• Embedding Dimensionality: Current retrieval scores range from 0.006–0.013 due to the use of bge-small (384 dimensions), necessitating a migration to qwen3-0.6b (1024 dimensions) for higher precision.

Working Examples

The grounding prompt used to ensure the LLM only answers based on provided source chunks.

const prompt = 
`Answer the question below using only the sources provided. ` +
`If the sources don't contain the answer, say so directly.\n\n` +
`Question: "${query}"\n\n` +
`Sources:\n${context}\n\n` +
`Write a direct answer in 2–4 sentences. No preamble. No bullets.\n` +
`Answer:`;

The API endpoint added to trigger synthesis mode instead of raw chunk return.

POST /search?mode=answer
{ "query": "your question here" }

Practical Applications

• 。Use case: Personal Knowledge Management (PKM) systems utilizing @cf/google/gemma-4-26b-a4b-it via Cloudflare Workers for cost-effective ($5/month) grounded synthesis.
• 。Pitfall: Relying on low-dimension embedding models during initial ingestion leads to poor retrieval precision, which cannot be fixed via prompting and requires full re-ingestion of data.

References:

• https://dev.to/dannwaneri/my-bookmark-engine-returned-chunks-i-added-one-endpoint-to-make-it-answer-317j