Evaluate AI agent response relevance using OpenAI and cosine similarity
Ready for 3.0Ready for 3.0. This template uses no nodes that n8n 3.0 removes. No removed nodes, versions or helpers found. Static check, last run 2026-10-19.
Nodes used (11)
- OpenAI Chat Model
@n8n/n8n-nodes-langchain.lmChatOpenAi - Evaluation Trigger
n8n-nodes-base.evaluationTrigger - Edit Fields (Set)
n8n-nodes-base.set - Evaluation
n8n-nodes-base.evaluation - No Operation, do nothing
n8n-nodes-base.noOp - AI Agent
@n8n/n8n-nodes-langchain.agent - Chat Trigger
@n8n/n8n-nodes-langchain.chatTrigger - Structured Output Parser
@n8n/n8n-nodes-langchain.outputParserStructured - Basic LLM Chain
@n8n/n8n-nodes-langchain.chainLlm - HTTP Request
n8n-nodes-base.httpRequest - Code
n8n-nodes-base.code
What it does
This n8n template demonstrates how to calculate the evaluation metric "Relevance" which in this scenario, measures the relevance of the agent's response to the user's question. The scoring approach is adapted from the open-source evaluations project RAGAS and you can see the source here https://github.com/explodinggradients/ragas/blob/main/ragas/src/ragas/metrics/answerrelevance.py How it works This evaluation works best for Q&A agents. For our scoring, we analyse the agent's response and ask another AI to generate a question from it. This generated question is then compared to the original question using cosine similarity. A high score indicates relevance and the agent's successful ability to answer the question whereas a low score means agent may have added too much irrelevant info, went off script or hallucinated. Requirements n8n version 1.94+ Check out this Google Sheet for a sample data https://docs.google.com/spreadsheets/d/1YOnu2JJjlxd787AuYcg-wKbkjyjyZFgASYVV0jsij5Y/edit?usp=sharing