Flowlitics Backend

This project is a Node.js RESTful backend API server designed to manage and execute complex workflows (Flows). It provides a robust API for defining, managing, and running workflows composed of various node types, including those interacting with AI agents and requiring human intervention. The system supports asynchronous, event-driven execution with persistence and scheduling capabilities.

⚠️ Not Production-Ready

This project is currently under active development and is not ready for production use. Expect frequent and potentially breaking changes to the codebase, features, and APIs. Use with caution and be prepared for instability.

❤️ Interested in collaborating or sponsoring? You're welcome to join! For more information, please see the bottom of this README.

Objective

The primary goal is to create a flexible and scalable engine for orchestrating complex processes that require coordination between automated tasks (like AI agent interactions) and manual steps (human-in-the-loop).

Features

• Workflow Definition: Define flows composed of interconnected nodes (Start, Action, Decision, Delay, End, Custom) and edges with optional conditions.

• Agent Integration: Integrate various AI agents and define 'human' agents to represent manual steps. See the Available Agents section for details.

• Asynchronous Execution: Execute workflows asynchronously using a queue-based, event-driven model. The executor processes nodes until a waiting state is encountered (e.g., human node pending), freeing up resources. Execution resumes upon external events (e.g., human node resolution).

• State Management: Persistent storage of flow definitions, executions, and execution steps using MongoDB.

• Human-in-the-Loop: Explicit support for nodes requiring manual input or approval, pausing execution until resolved via API.

• Pluggable Node Types: Extend functionality by adding custom node types via a plugin system.

• Scheduling: Schedule flow executions using cron expressions.

• API: Comprehensive RESTful API for managing flows, nodes, agents, executions, plugins, and schedules.

• Logging: Structured logging for monitoring system activity and execution progress.

• TypeScript: Developed using TypeScript for improved code quality and maintainability.

Architecture Highlights

The system employs an event-driven architecture centered around the queueManager and flowExecutionService.

• Queueing: New executions or resumed executions are added to a queue.

• Execution: A limited number of concurrent workers (flowExecutionService) pick up jobs from the queue.

• Partial Processing: The worker executes nodes sequentially within a flow until it encounters a node that cannot proceed (e.g., a 'pending' human node or a node waiting for input from a pending upstream node).

• Passive Waiting: When a flow is blocked, the worker releases it. The execution status remains 'running' in the database, but it's passively waiting for an external event.

• Event Trigger: An external event (e.g. an API call resolving a human node) updates the state of a specific node step.

• Re-queueing/Resumption: The event processing logic checks if the flow can now proceed. If so, the execution is re-enqueued, and a worker will pick it up to continue processing from the newly unblocked node(s).

• Finalization: When no more nodes can be processed and no nodes are pending, the execution is finalized as 'completed' or 'failed'.

This model ensures efficient resource utilization by not blocking workers on long-waiting tasks.

Key Components

• Example API Client: The ApiClient class in client-demo.ts provides methods to interact with the backend APIs for managing agents, flows, nodes, and executions.
• Flow Execution: Demonstrates creating a complex workflow with AI and human-controlled nodes, starting execution, and monitoring its progress.
• Mermaid Syntax: Thanks to Mermaid Live Editor Syntax, it is also possible to graphically represent the flow of all involved processes.

How to Run

1. Install Dependencies:

   git clone [email protected]:flowlitics/backend.git flowlitics-backend
   cd flowlitics-backend
   npm install

2. Set Up Environment:

   • Create a .env file with the necessary configuration. See .env.example for a template of all required variables.

3. Start the Server:

   npm start # or npm run dev

4. Run the Client Demo:

   Optionally, run the client-demo.ts to test workflow creation and execution:

   npm run client

Available Agents

You can use different agents out-of-the-box (after configuration via env file). Currently, the following AI agents/services are supported:

• Human: Represents a step requiring manual intervention
• Gemini: Google's AI language model
• Codestral: Mistral AI's coding assistant
• Mistral: Open-source language models
• Ollama: For running local AI models
• Tavily Search: AI-powered search engine
• OpenAI: Various models from OpenAI
• Anthropic Claude: Anthropic Claude models
• Groq: Various models from Groq (llama, deepseek, gemma, playai, qwen, compound, ..)

Future Agent Developments:

• Support for custom agent configurations per agent instance
• Integration of additional agent types (Grok, Tavily Extrct, ..).
• Integration of additional agent capabilities (OCR, file upload, ..)

Example Workflow Execution

The client-demo.ts script provides a practical example of how to use the API to:

1. Create AI and Human agents.
2. Define a complex flow structure with various node types (start, custom, decision, action, end).
3. Connect nodes with edges, including conditional edges.
4. Start an execution of the flow with initial data.
5. Simulate the resolution of a human-controlled node via an API call, triggering the flow to resume.
6. Poll the execution status until completion.
7. Retrieve the flow definition in Mermaid syntax for visualization.

Here you can find More Examples

Visual Representation Example

Flows can be visualized using Mermaid which can be retrieved via the API.

Thanks to Mermaid Live Editor Syntax.

Gigantic Flow Example

Here you can find More Visual Examples

Future Enhancements

Based on the project roadmap and current structure, planned enhancements include:

• Add authentication and authorization mechanisms
• Improved error handling, validation, and logging
• Enhanced monitoring and real-time updates (e.g. WebSocket integration)
• Integration with external services (e.g. Telegram bot for human task notifications)
• Advanced workflow features (e.g. fallback nodes, loops).
• Workflow templating, export, and import

Support the project

This open-source project grows thanks to everyone's support. If you appreciate this work and want to keep it active, consider making a small donation.

Even a small contribution, like the cost of a coffee ☕, can make a difference!

Why Donate?

• You support the continuous development and maintenance of the project.
• You contribute to creating new features and improvements.

How to Donate?

You can make a donation through:

Lighjtning Network [email protected]

Bitcoin address bc1qtly7cqy8zxzs79ksmdsfnz7hjyhhd3t2f9mvvj

Ethereum address 0xa4A79Be4e7AE537Cb9ee65DB92E6368425b2d63D

Thank you for your support! ❤️

License

This project is licensed under the MIT License.