Multi-workflow applications: composable, reusable workflow configs that call each other

[intel352]

Summary

Support applications that execute multiple workflows defined in separate, reusable config files. Workflows should be able to reference, call, and pass execution to each other — enabling modular, composable workflow design.

Motivation

Today a workflow config is a single monolithic unit. Real-world applications are better modeled as a composition of smaller, focused workflows. For example, a chat platform with responders might decompose into:

• Main loop workflow — handles incoming messages, routing, session management
• Queue & assignment workflow — manages conversation queueing, responder availability, and assignment logic
• Escalation workflow — handles escalation rules, timeouts, and supervisor routing

The main loop workflow would trigger/call the queue & assignment workflow when a new conversation arrives, and that workflow might trigger escalation under certain conditions. Each workflow is independently testable, reusable, and maintainable.

Proposed Scope

1. Multi-Workflow Config Loading

• An application config can reference multiple workflow config files
• Each workflow has its own namespace but shares the application's module registry
• Example top-level config:

application:
  name: chat-platform
  workflows:
    - file: workflows/main-loop.yaml
    - file: workflows/queue-assignment.yaml
    - file: workflows/escalation.yaml

2. Cross-Workflow Invocation

• A workflow step/action can trigger another workflow by name
• Support both synchronous (call and wait for result) and asynchronous (fire-and-forget / event-based) invocation
• Data passing: input parameters passed to the called workflow, output/result returned to the caller
• Example step config:

steps:
  - name: assign-conversation
    type: workflow-call
    config:
      workflow: queue-assignment
      mode: sync
      input:
        conversation_id: "{{ .conversation_id }}"
        priority: "{{ .priority }}"
      output_mapping:
        assigned_responder: "{{ .result.responder_id }}"

3. Shared Context & Isolation

• Workflows share the application-level service registry (event bus, data stores, etc.)
• Each workflow maintains its own execution context (state, variables)
• Cross-workflow communication via explicit invocation or shared event bus topics

4. Engine Support

• StdEngine loads and manages multiple workflow definitions
• TriggerWorkflow() supports routing to the correct workflow by name
• Workflow lifecycle management: start/stop individual workflows independently
• Error propagation across workflow boundaries with configurable behavior (fail-fast, retry, fallback)

5. UI Representation

• Workflow builder shows a multi-workflow view — list/tabs of workflows in the application
• Visual indication of cross-workflow calls (links between workflow canvases)
• Ability to drill into a called workflow from the caller's canvas
• Application-level overview showing workflow topology/dependencies

6. Testing & Observability

• Individual workflow configs can be tested in isolation
• Integration testing of composed multi-workflow applications
• Tracing/correlation IDs span across workflow boundaries
• Logging and metrics attribute events to the originating workflow

Example Use Case: Chat Platform

┌─────────────────────┐
│   Main Loop WF      │
│                     │
│  on: new_message    │──► ┌──────────────────────┐
│  → route            │    │  Queue & Assignment   │
│  → workflow-call ───┼───►│                      │
│  → respond          │    │  → check availability │
│                     │◄───│  → assign responder   │──► ┌─────────────────┐
│                     │    │  → return assignment   │    │  Escalation WF  │
└─────────────────────┘    └──────────────────────┘    │                 │
                                                        │  → check timeout│
                                                        │  → notify super │
                                                        └─────────────────┘

Acceptance Criteria

• Application config can reference multiple workflow YAML files
• Workflows can invoke other workflows synchronously (call/return)
• Workflows can invoke other workflows asynchronously (fire-and-forget via event bus)
• Input/output data passing works between workflows with mapping support
• Each workflow maintains isolated execution context
• Engine manages lifecycle of multiple workflows (start/stop/restart individually)
• UI displays multi-workflow application with navigation between workflows
• UI shows cross-workflow call relationships visually
• Correlation IDs propagate across workflow boundaries for tracing
• Example multi-workflow config included (e.g., chat platform scenario)
• Integration tests cover cross-workflow invocation patterns
• Documentation covers multi-workflow application design patterns