Most Node.js backends now have at least one LLM integration. The async, event-driven model fits this workload well: the bottleneck is network I/O, not CPU, which is exactly what Node handles without complaint. This post covers the OpenAI API basics, conversation state management, streaming, function calling, RAG with vector search, and the rate-limiting patterns that matter in production.

ai llm nodejs

Why Node.js for AI Applications?

Node’s non-blocking I/O is a good fit for LLM work. Most of the time you are waiting on the API, and Node handles that without spinning up threads or blocking the event loop. Running JavaScript on both ends also removes friction: the same Message type from the OpenAI SDK works client-side and server-side. The npm ecosystem has solid coverage, including the official OpenAI SDK, LangChain.js, and the MongoDB vector search driver.

Getting Started with OpenAI API in Node.js

Prerequisites

  • Node.js 20+
  • An OpenAI API key
  • Basic JavaScript or TypeScript

Project Setup

mkdir ai-nodejs-app
cd ai-nodejs-app
npm init -y
npm install openai express dotenv

Create a .env file:

OPENAI_API_KEY=your_openai_api_key_here

Basic OpenAI Integration

// index.js
import OpenAI from 'openai';
import dotenv from 'dotenv';

dotenv.config();

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

async function generateResponse(prompt) {
  const completion = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      { role: 'system', content: 'You are a helpful assistant.' },
      { role: 'user', content: prompt }
    ],
    max_tokens: 1000,
  });

  return completion.choices[0].message.content;
}

const response = await generateResponse('Explain Node.js in simple terms');
console.log(response);

The response object includes the generated text, token counts for both prompt and completion, and a finish reason. Log the token counts early; they are the main cost driver.

Building an AI-Powered Chatbot

The OpenAI API is stateless, so conversation history lives in your app. Each request sends the full message array so the model has context. The Map below stores history by session ID in memory, which works for prototyping but needs a persistent store in production.

// server.js
import express from 'express';
import OpenAI from 'openai';
import dotenv from 'dotenv';

dotenv.config();

const app = express();
app.use(express.json());

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const conversations = new Map();

app.post('/api/chat', async (req, res) => {
  try {
    const { sessionId, message } = req.body;

    if (!conversations.has(sessionId)) {
      conversations.set(sessionId, [
        {
          role: 'system',
          content: 'You are a helpful assistant specialized in Node.js development.'
        }
      ]);
    }

    const history = conversations.get(sessionId);
    history.push({ role: 'user', content: message });

    const completion = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: history,
      max_tokens: 1000,
      temperature: 0.7,
    });

    const assistantMessage = completion.choices[0].message.content;
    history.push({ role: 'assistant', content: assistantMessage });

    res.json({ success: true, message: assistantMessage, usage: completion.usage });
  } catch (error) {
    console.error('Chat error:', error);
    res.status(500).json({ success: false, error: error.message });
  }
});

app.listen(3000, () => console.log('Chatbot server running on port 3000'));

Implementing Streaming Responses

Without streaming, users wait several seconds staring at a blank input before the full response arrives. Server-Sent Events let you push tokens as they arrive. If you are new to Node.js streams, Understanding Backpressure and Stream Optimization covers the underlying mechanics worth knowing before you scale this in production.

app.post('/api/chat/stream', async (req, res) => {
  const { message } = req.body;

  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  try {
    const stream = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [
        { role: 'system', content: 'You are a helpful assistant.' },
        { role: 'user', content: message }
      ],
      stream: true,
    });

    for await (const chunk of stream) {
      const content = chunk.choices[0]?.delta?.content || '';
      if (content) {
        res.write(`data: ${JSON.stringify({ content })}\n\n`);
      }
    }

    res.write(`data: ${JSON.stringify({ done: true })}\n\n`);
    res.end();
  } catch (error) {
    res.write(`data: ${JSON.stringify({ error: error.message })}\n\n`);
    res.end();
  }
});

Function Calling and AI Agents

Function calling lets you give the model a list of tools, then let it decide which one to call and with what arguments. You still execute the function yourself, then feed the result back into the conversation. Here is a minimal weather tool example:

const tools = [
  {
    type: 'function',
    function: {
      name: 'get_weather',
      description: 'Get the current weather for a location',
      parameters: {
        type: 'object',
        properties: {
          location: { type: 'string', description: 'City name' },
          unit: { type: 'string', enum: ['celsius', 'fahrenheit'] }
        },
        required: ['location']
      }
    }
  }
];

async function runAgent(userMessage) {
  const messages = [
    { role: 'system', content: 'You are a helpful assistant with access to tools.' },
    { role: 'user', content: userMessage }
  ];

  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages,
    tools,
    tool_choice: 'auto',
  });

  return response.choices[0].message;
}

RAG: Retrieval-Augmented Generation

The base model knows nothing about your internal data. RAG fixes that by embedding your documents, storing them in a vector index, and injecting the nearest matches into the system prompt at query time. The model answers from your data, not from its training weights.

class RAGSystem {
  async generateEmbedding(text) {
    const response = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: text,
    });
    return response.data[0].embedding;
  }

  async query(question) {
    const relevantDocs = await this.search(question, 3);
    const context = relevantDocs.map(doc => doc.content).join('\n\n');

    const completion = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [
        {
          role: 'system',
          content: `Answer based on this context:\n${context}`
        },
        { role: 'user', content: question }
      ],
    });

    return completion.choices[0].message.content;
  }
}

Vector Embeddings with MongoDB

MongoDB Atlas has a $vectorSearch aggregation stage that runs approximate nearest-neighbour search directly in the database. If you are already on MongoDB, this avoids adding a separate vector store to the stack.

async function vectorSearch(query, limit = 5) {
  const embeddingResponse = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: query,
  });

  const results = await collection.aggregate([
    {
      $vectorSearch: {
        index: 'vector_index',
        path: 'embedding',
        queryVector: embeddingResponse.data[0].embedding,
        numCandidates: 100,
        limit: limit
      }
    }
  ]).toArray();

  return results;
}

Using LangChain with Node.js

LangChain.js is worth reaching for when the raw API gets repetitive, mostly for its prompt template system and chain abstractions. A simple code-explanation chain looks like this:

import { ChatOpenAI } from '@langchain/openai';
import { ChatPromptTemplate } from '@langchain/core/prompts';

const model = new ChatOpenAI({ modelName: 'gpt-4o' });

const chain = ChatPromptTemplate.fromMessages([
  ['system', 'You are an expert programmer.'],
  ['human', 'Explain this code:\n{code}']
]).pipe(model);

const response = await chain.invoke({ code: 'const x = 42;' });

Production Best Practices

Rate Limiting

When you are managing many concurrent LLM calls, see Promise.all() Is Fine… Until It Isn’t! for patterns on controlling parallel execution safely.

import Bottleneck from 'bottleneck';

const limiter = new Bottleneck({ maxConcurrent: 5, minTime: 100 });

const rateLimitedCall = limiter.wrap(async (messages) => {
  return openai.chat.completions.create({ model: 'gpt-4o', messages });
});

Retry Logic with Exponential Backoff

async function withRetry(fn, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await fn();
    } catch (error) {
      if (error.status === 429 && attempt < maxRetries - 1) {
        await new Promise(r => setTimeout(r, 1000 * Math.pow(2, attempt)));
      } else {
        throw error;
      }
    }
  }
}

Response Caching

import NodeCache from 'node-cache';
const cache = new NodeCache({ stdTTL: 3600 });

async function getCachedCompletion(prompt) {
  const cacheKey = `completion:${hashString(prompt)}`;
  const cached = cache.get(cacheKey);
  if (cached) return cached;

  const result = await openai.chat.completions.create({ /* ... */ });
  cache.set(cacheKey, result);
  return result;
}

Error Handling and Rate Limiting

The OpenAI API returns distinct status codes worth handling explicitly. A 401 means your key is wrong or missing — store your key properly and review the Node.js Security Checklist. A 429 is a rate limit hit. context_length_exceeded means the message array is too long for the model’s context window. Everything else is a transient service error worth wrapping generically. For a broader treatment of production error patterns, see Node.js Error Handling: Production Patterns.

async function safeCompletion(messages) {
  try {
    return await withRetry(() =>
      openai.chat.completions.create({ model: 'gpt-4o', messages })
    );
  } catch (error) {
    if (error.status === 401) throw new Error('Invalid API key');
    if (error.status === 429) throw new Error('Rate limit exceeded');
    if (error.code === 'context_length_exceeded') throw new Error('Input too long');
    throw new Error('AI service unavailable');
  }
}

Summary

The patterns here cover most of what you need to get an LLM integration into production. Streaming cuts perceived latency significantly. Function calling turns the model into an agent that can query your own APIs. RAG grounds answers in your actual data rather than the model’s training weights. Add backoff and response caching before you go live; token costs and rate limits will bite you faster than you expect. The OpenAI API reference and LangChain.js docs are the two tabs you will keep open throughout.