NVIDIA NIM

Run hundreds of GPU-hosted open models through NVIDIA NIM's OpenAI-compatible endpoint: agents call Llama, Nemotron, and DeepSeek chat models with tool calling, generate embeddings and rerank passages with NeMo Retriever for RAG, and discover models from the live catalog.

5 actions API key available

NVIDIA NIM website ↗ Platform Documentation ↗ Capability data verified 2026-07-25

A question arrives from the internal answer desk channel

Create Embeddings embeds the question with input type Query against the production embedder

The vector store returns a broad candidate set for the query

Rerank Documents scores the candidates with a cross-encoder and reorders them by relevance

The agent compares the top logit score against the grounding threshold; weak matches stop here

Chat Completion drafts an answer from the surviving passages, citations attached

Questions that failed the grounding check route to a subject matter expert instead of receiving a confident guess

Unanswered questions log as knowledge gaps for the documentation backlog

What This Integration Enables

NVIDIA NIM is open-weight AI as managed infrastructure: Llama, Nemotron, DeepSeek, Qwen, Mistral, and Gemma served from NVIDIA's GPU cloud through one OpenAI-compatible endpoint, alongside the NeMo Retriever family that handles the unglamorous half of RAG, embeddings and reranking. That combination is the point. Most teams assemble retrieval pipelines from three vendors and hope the seams hold; NIM puts the chat model, the embedder, and the cross-encoder reranker behind one key, and FlowRunner agents orchestrate them into pipelines with checkpoints where quality is decided. - Chat Completion exposes the full surface: multi-turn messages, tool calling, sampling controls, and NIM-specific extensions like the Nemotron reasoning toggle and guided decoding - Simple Chat covers the single-prompt case and surfaces thinking traces from reasoning models - Create Embeddings enforces the asymmetric Passage and Query input types that NVIDIA's QA embedders require - Rerank Documents scores up to 512 passages per request, the precision step after broad vector search - List Models reads the live catalog, so flows pin models deliberately instead of hardcoding ids - Retrieval-layer changes run through [human-in-the-loop](/concepts/human-in-the-loop) approval before they touch production indexes

Without FlowRunner

Open models mean GPU operations Running Llama or Nemotron seriously means provisioning, patching, and babysitting inference hardware

RAG stops at nearest-neighbor Vector search returns the roughly relevant, and the model answers from whatever floated to the top

Retrieval quality degrades invisibly An embedding mismatch or a wrong input type quietly ruins recall, and the pipeline still reports success

With FlowRunner

Open weights behind a managed endpoint Hundreds of models from the live catalog, called with the same request shape as any OpenAI-compatible API

A precision step before every answer Cross-encoder reranking narrows broad retrieval to the passages that actually support the reply

Index changes are governed changes Re-embedding the corpus is a proposed, approved operation, not a side effect of an agent's model swap

Use Case Scenarios

A knowledge base that answers with receipts

The team indexes internal documentation by chunking each page and running Create Embeddings with input type Passage, storing vectors in [Pinecone](/integrations/pinecone). Questions embed with input type Query, retrieval pulls fifty candidates, and Rerank Documents narrows them to the five that matter. Chat Completion answers from those five, citing each. Questions whose top rerank score falls below threshold go to a person, and the question text lands in the docs backlog. The answer desk gets faster and the documentation gets better from the same flow.

Ticket triage with a reasoning model's homework attached

When a ticket event fires in [Zendesk](/integrations/zendesk), Simple Chat on a Nemotron reasoning model classifies severity and product area, returning its thinking trace alongside the verdict. The summary and trace post to [Slack](/integrations/slack), so when the on-call engineer disagrees with a classification, they can see exactly where the reasoning went sideways and tune the prompt. Triage stops being a black box that the team either trusts blindly or ignores.

Support drafts grounded in the articles that actually apply

Inbound support email in [Freshdesk](/integrations/freshdesk) triggers a retrieval pass: candidate help articles are pulled, Rerank Documents orders them by true relevance to the customer's actual question, and Chat Completion drafts a reply grounded in the top passages. The draft, with its source articles linked, waits for the support agent's approval before sending. Response quality rises because the model reads the right three articles, not the nearest ten.

Human-in-Loop Highlight

The consequential operation on this page is the Create Embeddings run that rebuilds an index. NVIDIA's embedders are asymmetric and version-sensitive: vectors from different models are incomparable, Matryoshka dimensions must match what the store expects, and passages embedded with the Query input type measurably degrade retrieval while looking perfectly healthy. An agent that swaps embedding models or re-embeds part of a corpus on its own initiative does not crash anything; it quietly poisons every downstream answer flow, and the failure surfaces weeks later as "the bot got worse." So index rebuilds are proposed operations in FlowRunner: the agent presents the model id, output dimensions, input type, corpus size, and estimated token cost, and a person authorizes the rebuild as an all-or-nothing cutover, never a mixed state. "Re-embed 41,000 passages with nvidia/llama-3.2-nv-embedqa-1b-v2 at 1024 dimensions, replacing the e5-v5 index. Proceed?" The retrieval layer is shared infrastructure. Changes to it get a signature.

Agent processes routinely

Detects exception requiring judgment

Clear match Continues automatically

Ambiguous Routes to human via preferred channel

Human decides

Agent resumes with decision

Agent Capabilities

5 actions

Chat

Chat Completion Runs a full chat completion against any NIM-hosted model through the OpenAI-compatible endpoint: complete messages arrays, sampling controls, seed and stop sequences, tool and function calling, and an escape hatch for NIM-specific fields such as the Nemotron reasoning toggle and nvext guided decoding. Returns the raw completion with choices, tool calls, finish reason, and token usage.
Simple Chat Sends a single prompt to any NIM chat model and returns just the answer text, with optional system prompt and sampling controls. Reasoning models such as Nemotron and DeepSeek-R1 also return their thinking trace when the NIM exposes it.

Embeddings

Create Embeddings Generates vectors with NeMo Retriever embedding NIMs. The QA and E5 embedders are asymmetric: Passage for indexing documents, Query for search queries, and using the wrong one measurably degrades retrieval. Supports truncation, base64 encoding, and Matryoshka output dimensions on capable models.
Rerank Documents Scores and reorders up to 512 candidate passages against a query with a cross-encoder reranking NIM, returning them from most to least relevant with original index, raw logit score, and text. The precision step after broad vector search.

Models

List Models Lists every model hosted on the NIM cloud endpoint, chat, reasoning, embedding, reranking, and multimodal, from NVIDIA, Meta, DeepSeek, Qwen, Mistral, Microsoft, and other publishers, with the namespaced publisher/model ids other operations expect. Supports free-text and publisher filters.

Frequently Asked Questions

What can FlowRunner do with NVIDIA NIM?

FlowRunner agents can run Chat Completion, Simple Chat, and Create Embeddings in NVIDIA NIM, plus 2 more actions.

Does connecting NVIDIA NIM to FlowRunner require OAuth?

No. NVIDIA NIM connects to FlowRunner with an API key, no OAuth flow required.

Can NVIDIA NIM trigger a FlowRunner workflow automatically?

NVIDIA NIM doesn't currently expose triggers in FlowRunner. It connects as an action step inside workflows started by another trigger.

Start building with NVIDIA NIM

$100 in credits. No card required. Connect in minutes.

Start Building Free Talk to us