Flowfile

Documentation: Website - Core - Worker - Frontend

Flowfile is a visual ETL tool that combines drag-and-drop workflow building with the speed of Polars dataframes. Build data pipelines visually, transform data using powerful nodes, and analyze results - all without writing code.

⚡ Technical Design

Flowfile operates as three interconnected services:

• Designer (Electron + Vue): Visual interface for building data flows
• Core (FastAPI): ETL engine using Polars for data transformations (:63578)
• Worker (FastAPI): Handles computation and caching of data operations (:63579)

Each flow is represented as a directed acyclic graph (DAG), where nodes represent data operations and edges represent data flow between operations.

🔥 Example Use Cases

• Data Cleaning & Transformation
  • Complex joins (fuzzy matching)
  • Text to rows transformations
  • Advanced filtering and grouping
  • Custom formulas and expressions
  • Filter data based on conditions

---

• Performance
  • Build to scale out of core
  • Using polars for data processing

---

Data Integration

• Standardize data formats
• Handle messy Excel files

---

• ETL Operations
  • Data quality checks

🚀 Getting Started

Prerequisites

• Python 3.10+
• Node.js 16+
• Poetry (Python package manager)
• Docker & Docker Compose (option, for Docker setup)
• Make (optional, for build automation)

Installation Options

1. Desktop Application (Recommended)

The desktop version offers the best experience with a native interface and integrated services. You can either:

Option A: Download Pre-built Application

• Download the latest release from GitHub Releases
• Run the installer for your platform (Windows, macOS, or Linux)

Option B: Build from Source:

git clone https://github.com/edwardvaneechoud/Flowfile.git
cd Flowfile

# Build packaged executable
make    # Creates platform-specific executable

# Or manually:
poetry install
poetry run build_backends
cd flowfile_frontend
npm install
npm run build      # All platforms

2. Docker Setup

Perfect for quick testing or deployment scenarios. Runs all services in containers with proper networking and volume management:

# Clone and start all services
git clone https://github.com/edwardvaneechoud/Flowfile.git
cd Flowfile
docker compose up -d

# Access services:
Frontend: http://localhost:8080 # main service
Core API: http://localhost:63578/docs
Worker API: http://localhost:63579/docs

Just place your files that you want to transform in the directory in shared_data and you're all set!

3. Manual Setup (Development)

Ideal for development work when you need direct access to all services and hot-reloading:

git clone https://github.com/edwardvaneechoud/Flowfile.git
cd Flowfile

# Install Python dependencies
poetry install

# Start backend services
poetry run flowfile_worker  # Starts worker on :63579
poetry run flowfile_core   # Starts core on :63578

# Start web frontend
cd flowfile_frontend
npm install
npm run dev:web  # Starts web interface on :8080

📋 TODO

Core Features

• Add cloud storage support
  • S3 integration
  • Azure Data Lake Storage (ADLS)
• Multi-flow execution support
• Polars code reverse engineering
  • Generate Polars code from visual flows
  • Import existing Polars scripts

Documentation

• Add comprehensive docstrings
• Create detailed node documentation
• Add architectural documentation
• Improve inline code comments
• Create user guides and tutorials

Infrastructure

• Implement proper testing
• Add CI/CD pipeline
• Improve error handling
• Add monitoring and logging

📝 License

MIT License

Acknowledgments

Built with Polars, Vue.js, FastAPI, Vueflow and Electron.