Skip to content
Drew-ChasePublic

About

A client/server solution for batch processing ffmpeg operations from multiple systems across the internet.

Resources

Stars

2 stars

Watchers

1 watching

Forks

Repository files navigation

FFNodes

Have you ever wanted to encode many video or audio files at once, across multiple devices? FFNodes is a Rust-based tool designed to simplify this process by leveraging the power of ffmpeg and distributed computing.

Table of Contents

Features

  • Distributed encoding across multiple devices
  • GUID-based authentication for secure client connections
  • Priority-based job queue (largest and most complex files first)
  • Smart file re-probing (only probes when file modification time changes)
  • Automatic file system watching with batch processing
  • Client heartbeat monitoring with automatic job reassignment on timeout
  • FFmpeg template system with hardware acceleration support
  • Real-time WebSocket progress monitoring
  • Automatic job recovery and retry with exponential backoff
  • Support for 47+ video/audio formats

Architecture

FFNodes uses a client-server architecture for distributed video encoding:

System Components

  1. Server - Central coordinator that manages:

    • Media file discovery and metadata extraction
    • Job queue with priority-based assignment
    • Client registration and heartbeat monitoring
    • File system watching for automatic job creation
    • REST API and WebSocket endpoints
  2. Clients - Worker nodes that:

    • Connect to the server via GUID authentication
    • Request encoding jobs from the queue
    • Execute FFmpeg with hardware acceleration
    • Report progress and completion status
    • Send periodic heartbeats

Core Subsystems

Media File Scanner

  • Recursively scans configured directories for supported video/audio files
  • Extracts metadata using FFprobe (duration, resolution, bitrate, codec, etc.)
  • Calculates encoding complexity: (width × height) × bitrate × duration
  • Smart re-probing: only re-probes files when last_modified timestamp changes
  • Processes up to 10 files concurrently

File Watcher

  • Monitors configured directories for file system changes using notify
  • Batch processing every 30 seconds (configurable) to reduce overhead
  • Automatically adds new files to database and creates encoding jobs
  • Removes deleted files from database and cancels associated jobs

Job Queue

  • Priority-based queue: priority = file_size × encoding_complexity / (retry_count + 1)
  • Largest and most complex files encode first
  • Failed jobs automatically deprioritized based on retry count
  • In-memory cache with SQLite persistence
  • Job states: pending → assigned → in_progress → completed/failed

Client Manager

  • GUID-based authentication (server generates UUID on first launch)
  • Tracks client metadata: display name, computer name, connection time
  • Heartbeat monitoring with configurable timeout (default 300 seconds)
  • Automatic disconnection detection and job reassignment

Job Scheduler

  • Background task runs every 60 seconds
  • Detects stale jobs (assigned but no progress for > timeout)
  • Automatically requeues stale jobs when clients disconnect or timeout
  • Increments retry count and recalculates priority

Template System

  • Server provides FFmpeg command template to clients
  • Template format: -c:v h264{HWACCEL_CODE} -preset medium -crf 23 -i {INPUT} {OUTPUT}
  • Clients specify hardware acceleration:
    • _nvenc - NVIDIA NVENC
    • _amf - AMD AMF
    • _qsv - Intel Quick Sync
    • `` (empty) - Software encoding
  • Template parser performs simple string replacement

Data Flow

  1. Server Startup:

    • Load configuration (or generate with new GUID)
    • Initialize SQLite database with WAL mode
    • Start job scheduler for stale job monitoring
    • Start file watcher for automatic discovery
    • Perform initial media file scan
    • Start HTTP server and WebSocket endpoint
  2. Client Connection:

    • Client sends handshake with server GUID, display name, computer name
    • Server validates GUID and registers client
    • Server returns auth token (client ID) and FFmpeg template
  3. Job Assignment:

    • Client requests job with client ID
    • Server retrieves highest priority pending job
    • Server assigns job to client and updates state
    • Server returns job details with input/output paths
  4. Job Execution:

    • Client starts job (updates state to in_progress)
    • Client executes FFmpeg with template and hardware acceleration
    • Client sends periodic progress updates (frame, fps, speed)
    • Client sends heartbeats to maintain connection
  5. Job Completion:

    • Client reports completion with output metadata (size, bitrate)
    • OR client reports failure with error message (increments retry count)
    • Server updates job state and media file record
    • Job removed from queue or requeued if failed
  6. Monitoring:

    • WebSocket connection provides real-time progress updates
    • REST endpoints provide system status, client list, active jobs

Database Schema

media_files table:

  • Stores file metadata and processing state
  • Primary key: path (file path)
  • Fields: size, bitrate, duration, resolution, frames, encoding_complexity, retry_count, processed, last_modified

encoding_jobs table:

  • Stores job queue state
  • Primary key: id (UUID)
  • Fields: media_file_path, status, priority, assigned_client, timestamps, output details, error_message
  • Foreign keys: media_files(path), clients(id)

clients table:

  • Stores connected client information
  • Primary key: id (UUID)
  • Fields: display_name, computer_name, connected_at, last_heartbeat, disconnected_at

encoding_progress table:

  • Stores real-time encoding progress
  • Primary key: job_id
  • Fields: frame, fps, bitrate, speed, updated_at
  • Foreign key: encoding_jobs(id)

Installation

FFNodes is available on Windows, Linux, and MacOS, for both x86 and ARM architectures.

Note: FFNodes requires ffmpeg to be installed on your system and accessible from the command line. You can find ffmpeg here.

Server

The server is a simple REST API that can be used to manage encoding jobs. You can Install the server in a few ways:

  1. Using Cargo Install: cargo install ffnodes-server --git https://github.com/Drew-Chase/FFNodes.git
  2. Using our installer script: curl -fsSL https://raw.githubusercontent.com/Drew-Chase/FFNodes/main/install.sh | bash (linux only)
  3. Using our MSI installer found in the releases page (windows only)
  4. Building from source: git clone https://github.com/Drew-Chase/FFNodes.git && cd FFNodes && cargo build --release --bin ffnodes-server.
  5. Using Docker: docker run -p 8080:8080 ghcr.io/drew-chase/ffnodes-server:latest
  6. Using Homebrew: brew install drew-chase/tap/ffnodes-server (macos only)
  7. Using winget: winget install FFNodes.FFNodesServer (windows only)

Client

The client is a GUI application that can be used to manage encoding jobs. You can install the client in a few ways:

  1. Using Cargo Install: cargo install ffnodes-client --git https://github.com/Drew-Chase/FFNodes.git
  2. Using our appimage file found in the releases page (linux only)
  3. Using our MSI installer found in the releases page (windows only)
  4. Building from source: git clone https://github.com/Drew-Chase/FFNodes.git && cd FFNodes && cargo build --release --bin ffnodes-client.
  5. Using Homebrew: brew install drew-chase/tap/ffnodes-client (macos only)
  6. Using winget: winget install FFNodes.FFNodesClient (windows only)

Docker Deployment (Server Only)

The ffnodes-server can be deployed as a Docker container for easy deployment and management. Note: Only the server is containerized. The client is a desktop application that connects to the server.

Quick Start

# Pull latest image
docker pull ghcr.io/drew-chase/ffnodes/ffnodes-server:latest

# Run with default configuration
docker run -d \
  --name ffnodes-server \
  -p 8080:8080 \
  -v $(pwd)/data:/app/data \
  -v $(pwd)/logs:/app/logs \
  ghcr.io/drew-chase/ffnodes/ffnodes-server:latest

Using Docker Compose

Create docker-compose.yml:

version: '3.8'

services:
  ffnodes-server:
    image: ghcr.io/drew-chase/ffnodes/ffnodes-server:latest
    container_name: ffnodes-server
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - ./data:/app/data
      - ./logs:/app/logs
      - /path/to/videos:/videos:ro  # Mount video directories read-only
    environment:
      - RUST_LOG=info
      - FFNODES_PORT=8080
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:8080/api/status"]
      interval: 30s
      timeout: 10s
      retries: 3

Run:

docker-compose up -d

Docker Configuration

The container expects configuration at /app/data/config.json. On first run, a default configuration will be generated.

Extract the server GUID:

docker logs ffnodes-server | grep "Server GUID"

Mount existing config:

docker run -d \
  -p 8080:8080 \
  -v $(pwd)/config.json:/app/data/config.json \
  -v $(pwd)/data:/app/data \
  ghcr.io/drew-chase/ffnodes/ffnodes-server:latest

Persistent Data

The container uses the following volumes for persistent data:

  • /app/data - Configuration and SQLite database (app.db)
  • /app/logs - Application logs
  • Video directories - Mount source directories as read-only with :ro flag

Multi-Architecture Support

The Docker image supports multiple architectures:

  • linux/amd64 - Intel/AMD x86_64
  • linux/arm64 - ARM 64-bit (Raspberry Pi 4, Apple Silicon, AWS Graviton)

Docker automatically pulls the correct architecture for your platform.

Building Locally

# Clone repository
git clone https://github.com/Drew-Chase/FFNodes.git
cd FFNodes

# Build for current architecture
docker build -t ffnodes-server .

# Build for specific platform
docker buildx build --platform linux/arm64 -t ffnodes-server:arm64 .

Configuration

The server uses a config.json file for configuration. On first launch, it will generate a default configuration with a new server GUID.

Configuration File Location

  • Debug mode: target/dev-env/server/config.json
  • Production: ./config.json (in the same directory as the executable)

Configuration Options

{
  "port": 8080,
  "ffmpeg": "path/to/ffmpeg",
  "ffprobe": "path/to/ffprobe",
  "watch_directories": [
    "/path/to/video/library1",
    "/path/to/video/library2"
  ],
  "server_guid": "550e8400-e29b-41d4-a716-446655440000",
  "ffmpeg_template": "-c:v h264{HWACCEL_CODE} -preset medium -crf 23 -i {INPUT} {OUTPUT}",
  "client_timeout_seconds": 300,
  "notify_batch_interval_seconds": 30,
  "max_concurrent_jobs_per_client": 4
}

Configuration Fields

  • port - HTTP server port (default: 8080)
  • ffmpeg - Path to FFmpeg binary (auto-detected if in PATH)
  • ffprobe - Path to FFprobe binary (auto-detected if in PATH)
  • watch_directories - Array of directories to monitor for media files
  • server_guid - Unique server identifier for client authentication (auto-generated)
  • ffmpeg_template - FFmpeg command template with placeholders:
    • {HWACCEL_CODE} - Hardware acceleration suffix (e.g., _nvenc, _amf, _qsv, or empty)
    • {INPUT} - Input file path
    • {OUTPUT} - Output file path
  • client_timeout_seconds - Time before reassigning jobs from unresponsive clients (default: 300)
  • notify_batch_interval_seconds - File watcher batch processing interval (default: 30)
  • max_concurrent_jobs_per_client - Maximum jobs per client (default: 4)

Supported Video Formats

FFNodes automatically detects and processes these formats: webm, mkv, flv, vob, ogv, ogg, rrc, gifv, mng, mov, avi, qt, wmv, yuv, rm, asf, amv, mp4, m4p, m4v, mpg, mp2, mpeg, mpe, mpv, m4v, svi, 3gp, 3g2, mxf, roq, nsv, flv, f4v, f4p, f4a, f4b, mts, m2ts, ts

Usage

Starting the Server

  1. Launch the server:

    ffnodes-server
  2. On first launch, the server will:

    • Generate config.json with a new GUID
    • Create app.db SQLite database
    • Display the server GUID in logs (needed for client connections)
    • Start the HTTP server on configured port
  3. The server will log:

    Server GUID: 550e8400-e29b-41d4-a716-446655440000
    Starting production server at http://127.0.0.1:8080...
    

Client Workflow

  1. Handshake: Connect to the server with GUID

    POST /api/handshake
    {
      "server_guid": "550e8400-e29b-41d4-a716-446655440000",
      "display_name": "My Workstation",
      "computer_name": "DESKTOP-ABC123"
    }
    

    Response includes auth token and FFmpeg template.

  2. Request Job: Poll for available jobs

    POST /api/jobs/request/{client_id}
    

    Returns highest priority job or 204 No Content if queue is empty.

  3. Start Job: Mark job as in progress

    POST /api/jobs/{job_id}/start
    
  4. Report Progress: Send periodic updates

    POST /api/jobs/{job_id}/progress
    {
      "frame": 1250,
      "fps": 45.2,
      "bitrate": "2500kbits/s",
      "speed": "1.5x"
    }
    
  5. Complete Job: Report success or failure

    POST /api/jobs/{job_id}/complete
    {
      "output_size": 125829120,
      "output_bitrate": 2500000
    }
    

    or

    POST /api/jobs/{job_id}/fail
    {
      "error": "FFmpeg process exited with code 1"
    }
    
  6. Heartbeat: Maintain connection

    POST /api/heartbeat/{client_id}
    

    Send every 60 seconds to avoid timeout.

Monitoring

  • System Status: GET /api/status

    {
      "total_media_files": 1523,
      "pending_jobs": 45,
      "active_jobs": 8,
      "connected_clients": 3
    }
  • Connected Clients: GET /api/clients

    [
      {
        "id": "client-uuid",
        "display_name": "My Workstation",
        "computer_name": "DESKTOP-ABC123",
        "active_jobs": 2,
        "connected_at": "2025-01-15T10:30:00Z"
      }
    ]
  • Active Jobs: GET /api/jobs/active

    [
      {
        "id": "job-uuid",
        "media_file_path": "/videos/movie.mp4",
        "status": "in_progress",
        "assigned_client": "client-uuid",
        "priority": 15000000,
        "started_at": "2025-01-15T10:35:00Z"
      }
    ]
  • Real-time Progress: WebSocket at ws://localhost:8080/api/ws/progress

    • Receives JSON events for job assignments, progress updates, completions, and client connections

API Documentation

For complete API specification with request/response schemas, see API.md.

API Endpoints Summary

Authentication

  • POST /api/handshake - Register client with server GUID

Job Management

  • POST /api/jobs/request/{client_id} - Request next job
  • POST /api/jobs/{job_id}/start - Mark job as started
  • POST /api/jobs/{job_id}/progress - Update progress
  • POST /api/jobs/{job_id}/complete - Mark job complete
  • POST /api/jobs/{job_id}/fail - Mark job failed
  • GET /api/jobs/active - Get all active jobs

Monitoring

  • GET /api/status - Get system status
  • GET /api/clients - Get connected clients
  • POST /api/heartbeat/{client_id} - Send heartbeat

WebSocket

  • GET /api/ws/progress - Real-time progress updates

Technology

FFNodes is built using Rust for performance and safety.

Server

The server uses the following technologies:

  • Actix Web - For the REST API
  • SQLX with SQLite - For storing job metadata and found media files
  • Tokio - For asynchronous tasks
  • Notify - For watching for media files

Client

The client uses the following technologies:

About

A client/server solution for batch processing ffmpeg operations from multiple systems across the internet.

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Used by

Contributors

Languages