Process incoming files and notify via email with GitHub storage

File Processing Pipeline with Email and GitHub This workflow automatically ingests newly-uploaded files, validates and transforms their contents, stores the processed files in a GitHub repository, and sends email notifications upon completion. It is ideal for teams that regularly receive data drops and need an auditable, automated pipeline to clean, version, and distribute those files. Pre-conditions/Requirements Prerequisites - n8n instance (self-hosted or n8n cloud) - GitHub and SMTP credentials with appropriate permissions - Basic understanding of JavaScript for custom validation/transform code Required Credentials | Credential | Purpose | Minimum Scope | |------------|---------|---------------| | GitHub OAuth2 | Commit transformed files to a repository | repo | | SMTP (Email Send) | Dispatch success/failure notifications | send | Repository & Email Setup Requirements | Item | Example | Notes | |------|---------|-------| | GitHub Repository | [email protected]:org/data-pipeline.git | Branch must exist (default: main) | | Email Recipients | [email protected], [email protected] | Comma-separated list | How it works This workflow automatically ingests newly-uploaded files, validates and transforms their contents, stores the processed files in a GitHub repository, and sends email notifications upon completion. It is ideal for teams that regularly receive data drops and need an auditable, automated pipeline to clean, version, and distribute those files. Key Steps: - Schedule Trigger: Executes the pipeline on a defined schedule (e.g., hourly). - HTTP Request: Pulls a file manifest or fetches file blobs from an upstream service. - SplitInBatches: Iterates over each file individually for isolated processing. - Code (Validate): Runs custom JavaScript to ensure file integrity and schema compliance. - If (Validation Pass?): Routes good files forward and diverts failures. - Code (Transform): Performs any data re-formatting, enrichment, or cleansing. - Set: Prepares commit metadata (file path, commit message). - GitHub: Creates or updates a file in the target repository/branch. - Email Send: Notifies stakeholders of overall job status, including summary metrics. Set up steps Setup Time: 10-15 minutes 1. Clone the Workflow: Import the JSON file or copy from the n8n community templates gallery. 2. Configure Schedule Trigger: Set your desired cron expression (e.g., 0 for hourly). 3. HTTP Request Node: - Enter the endpoint that lists or returns your new uploads. - Add authentication headers if required. 4. Adjust Validation Code: Open the first Code node and replace the schema rules with your own logic. 5. Modify Transformation Logic: In the second Code node, apply formatting (e.g., CSV → JSON) or enrichment steps. 6. GitHub Node: - Select your GitHub OAuth2 credential. - Specify repository, branch, and file path template (e.g., processed/{{ $json.filename }}). 7. Email Send Node: - Set SMTP credential. - Add recipients and customize subject/body with variables. 8. Test Run: Click “Execute Workflow” to ensure end-to-end success. 9. Activate: Toggle workflow to “Active” for autonomous execution. Node Descriptions Core Workflow Nodes: - Sticky Note – Documentation and inline comments for maintainers. - Schedule Trigger – Initiates the automation on a fixed timetable. - HTTP Request – Retrieves incoming file list or contents. - SplitInBatches – Paginates large manifests to conserve memory. - Code (Validate / Transform) – Custom JS powering data quality checks and format conversion. - If – Branching logic for pass/fail handling. - Set – Crafts commit metadata and email summaries. - GitHub – Commits each transformed file to the repository. - Email Send – Dispatches completion notices or error alerts. Data Flow: 1. Schedule Trigger → HTTP Request → SplitInBatches → Code (Validate) → If - true → Code (Transform) → Set → GitHub - false → Set (Error) 2. Both branches converge → Email Send Customization Examples Custom Validation Rule javascript // Reject