Evaluations metric: answer similarity
Ready for 3.0Ready for 3.0. This template uses no nodes that n8n 3.0 removes. No removed nodes, versions or helpers found. Static check, last run 2026-10-19.
Nodes used (12)
- OpenAI Chat Model
@n8n/n8n-nodes-langchain.lmChatOpenAi - Evaluation Trigger
n8n-nodes-base.evaluationTrigger - Edit Fields (Set)
n8n-nodes-base.set - Evaluation
n8n-nodes-base.evaluation - No Operation, do nothing
n8n-nodes-base.noOp - AI Agent
@n8n/n8n-nodes-langchain.agent - Chat Trigger
@n8n/n8n-nodes-langchain.chatTrigger - HTTP Request
n8n-nodes-base.httpRequest - Split Out
n8n-nodes-base.splitOut - Aggregate
n8n-nodes-base.aggregate - Merge
n8n-nodes-base.merge - Code
n8n-nodes-base.code
What it does
This n8n template demonstrates how to calculate the evaluation metric "Similarity" which in this scenario, measures the consistency of the agent. The scoring approach is adapted from the open-source evaluations project RAGAS and you can see the source here https://github.com/explodinggradients/ragas/blob/main/ragas/src/ragas/metrics/answersimilarity.py How it works This evaluation works best where questions are close-ended or about facts where the answer can have little to no deviation. For our scoring, we generate embeddings for both the AI's response and ground truth and calculate the cosine similarity between them. A high score indicates LLM consistency with expected results whereas a low score could signal model hallucination. Requirements n8n version 1.94+ Check out this Google Sheet for a sample data https://docs.google.com/spreadsheets/d/1YOnu2JJjlxd787AuYcg-wKbkjyjyZFgASYVV0jsij5Y/edit?usp=sharing