From sitemap crawling to vector storage: Creating an efficient workflow for RAG

by Mariela Slavenova ✓ · 1,331 views · created 2025-09-18 · Market Research, AI RAG

What this workflow does

This AI workflow has 35 nodes and runs manually, when you click “Execute workflow”. Along the way it connects Embeddings OpenAI, Supabase, Postgres and Supabase Vector Store. It also calls kiekens.com and crawl4ai-app-nrcsv.ondigitalocean.app over HTTP.

How it runs

Trigger: Manually, when you click “Execute workflow”.

1. HTTP Request: calls kiekens.com
2. XML: converts between XML and JSON
3. Split Out: splits a list into separate items
4. Loop Over Items (Loop Over Items (Split in Batches)): loops over the items in batches
5. Split Out: splits a list into separate items
6. Format the URL (Edit Fields (Set)): sets 1 field: loc
7. Loop Over Items (Loop Over Items (Split in Batches)): loops over the items in batches
8. Check if the URL is in the Supabase Table (Supabase): get many
9. Get a row - scrape_queue Table (Supabase): get
10. Format the Output from the Supabase node (Code): runs custom JavaScript code

…plus 21 more steps. See the full node list below.

Services it uses

Embeddings OpenAISupabasePostgresSupabase Vector Store

HTTP calls to: kiekens.com, crawl4ai-app-nrcsv.ondigitalocean.app

Before you upgrade to n8n 3.0

Nodes used (15)

• Manual Triggern8n-nodes-base.manualTrigger
• HTTP Requestn8n-nodes-base.httpRequest
• XMLn8n-nodes-base.xml
• Split Outn8n-nodes-base.splitOut
• Loop Over Items (Split in Batches)n8n-nodes-base.splitInBatches
• Waitn8n-nodes-base.wait
• Ifn8n-nodes-base.if
• Default Data Loader@n8n/n8n-nodes-langchain.documentDefaultDataLoader
• Character Text Splitter@n8n/n8n-nodes-langchain.textSplitterCharacterTextSplitter
• Embeddings OpenAI@n8n/n8n-nodes-langchain.embeddingsOpenAi
• Edit Fields (Set)n8n-nodes-base.set
• Supabasen8n-nodes-base.supabase
• Coden8n-nodes-base.code
• Postgresn8n-nodes-base.postgres
• Supabase Vector Store@n8n/n8n-nodes-langchain.vectorStoreSupabase

From the author

This template crawls a website from its sitemap, deduplicates URLs in Supabase, scrapes pages with Crawl4AI, cleans and validates the text, then stores content + metadata in a Supabase vector store using OpenAI embeddings. It’s a reliable, repeatable pipeline…

View the original on n8n.io → Check your own copy