Build a Dual-Model RAG System: Integrating Claude and ChatGPT for Smarter AI Responses

Build a RAG System with Claude & ChatGPT APIs

Gate of AI published a step-by-step tutorial on building a Retrieval-Augmented Generation (RAG) system that integrates both Claude and ChatGPT APIs. The implementation uses Node.js v18+ and requires both OpenAI and Anthropic API keys.

Why This Matters

While individual large language models like GPT-4o and Claude 3.5 Sonnet offer impressive general knowledge, they lack access to proprietary or real-time data, limiting their usefulness in enterprise contexts like customer support or internal research. This RAG architecture solves that gap by retrieving relevant documents from a local repository before generating responses, ensuring outputs are grounded in verifiable, up-to-date content rather than relying solely on the model’s training data.

Key Insights

• RAG architecture: Fetches relevant documents from a pre-indexed set using keyword matching, then sends combined context to language models for response generation—demonstrated in the 45-minute tutorial by Gate of AI in 2026.
• Multi-model approach: The system sends the same prompt to both Claude 3.5 Sonnet and GPT-4o, returning separate responses for comparison or combination—enabling users to leverage the strengths of each model.
• Modular integration: Uses separate SDKs (OpenAI and Anthropic) with environment variables for API keys, clearly separated functions for each model, and a unified query handler that combines retrieval and response generation.
• Common mistake warning: The tutorial explicitly warns that misconfigured environment variables will cause authentication errors, as both APIs require correctly set OPENAI_API_KEY and ANTHROPIC_API_KEY.

Working Examples

Installs required SDKs for accessing the OpenAI and Anthropic APIs, and the dotenv package for managing environment variables.

npm install openai anthropic dotenv

Sets up a local document repository by reading a JSON file and filtering documents based on the user query.

const fs = require('fs');

const documents = JSON.parse(fs.readFileSync('documents.json', 'utf8'));

function getRelevantDocuments(query) {
  // Simple keyword matching for relevance
  return documents.filter(doc => doc.text.includes(query));
}

module.exports = { getRelevantDocuments };

Integrates both the OpenAI and Anthropic clients, providing functions to send prompts to GPT-4o and Claude 3.5 Sonnet and retrieve responses.

require('dotenv').config();

const { OpenAI } = require('openai');
const { Anthropic } = require('anthropic');

const openAIClient = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const anthropicClient = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

async function generateResponseWithClaude(prompt) {
  const response = await anthropicClient.chat.completions.create({
    model: "claude-3-5-sonnet-20241022",
    messages: [{ role: "user", content: prompt }]
  });
  return response.data.choices[0].message.content;
}

async function generateResponseWithChatGPT(prompt) {
  const response = await openAIClient.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }]
  });
  return response.data.choices[0].message.content;
}

module.exports = { generateResponseWithClaude, generateResponseWithChatGPT };

Core query handling logic that retrieves relevant documents, builds a combined context, and sends the prompt to both Claude and ChatGPT for response generation.

const { getRelevantDocuments } = require('./documentRepository');
const { generateResponseWithClaude, generateResponseWithChatGPT } = require('./aiIntegrations');

async function handleUserQuery(query) {
  const relevantDocs = getRelevantDocuments(query);
  const combinedContext = relevantDocs.map(doc => doc.text).join('\n');
  
  const prompt = `Based on these documents:\n${combinedContext}\nAnswer the following question: ${query}`;
  
  const claudeResponse = await generateResponseWithClaude(prompt);
  const chatGPTResponse = await generateResponseWithChatGPT(prompt);
  
  return {
    claude: claudeResponse,
    chatGPT: chatGPTResponse
  };
}

module.exports = { handleUserQuery };

Test script that simulates a user query to verify the RAG system returns responses from both Claude and ChatGPT.

const { handleUserQuery } = require('./queryHandler');

(async () => {
  const query = "How does the RAG system work?";
  const responses = await handleUserQuery(query);
  console.log("Claude's response:", responses.claude);
  console.log("ChatGPT's response:", responses.chatGPT);
})();

Practical Applications

• Customer support systems can use the dual-model RAG to generate accurate, context-aware answers by querying internal knowledge bases through both Claude and ChatGPT, then selecting the best response—pitfall: simple keyword matching may miss relevant documents, leading to incomplete or off-topic answers if the query doesn’t exactly match document text.
• Research assistants: The system can combine retrieved document context with AI generation to produce comprehensive summaries—pitfall: sending the full context to both models without truncation can exceed token limits, causing errors or incomplete responses.

References:

• https://dev.to/gateofai/build-a-rag-system-with-claude-chatgpt-apis-nao