3D Workflow Engine: beyond DAGs for intelligent document processing

Context

A financial services group processing thousands of documents monthly - loan applications, compliance filings, insurance claims, regulatory correspondence - was struggling with their existing workflow automation. They had invested in a traditional DAG-based orchestration platform (comparable to Airflow/Temporal/Camunda) to automate document processing pipelines.

The problem was not the technology itself. It was that flat, sequential workflow models cannot express the complexity of real document processing: varying formats, multiple languages, ambiguous content, feedback loops between extraction and validation, and business rules that change mid-flight.

The problem: why 2D workflows fail for complex domains

The client's document processing pipeline was a classic directed acyclic graph: Parse PDF → Extract entities → Validate → Store. It worked for simple, standardized documents. For everything else, it broke.

Specific failures:

• Fixed execution order: when entity extraction failed on a malformed document, the entire downstream path was blocked. No alternative routing, no strategic reconsideration of the approach
• No strategic reasoning: DAG nodes are executors, not thinkers. When processing a complex financial statement, a node cannot step back and ask: "Should I be using a different extraction strategy given what I've learned about this document's structure?"
• Destructive failure propagation: when one task failed, all downstream tasks were automatically blocked, even if they could partially execute with available data
• No learning across executions: each workflow run was isolated. If the system successfully extracted data from a rare document format on Monday, it had no memory of that success on Tuesday. Valuable operational knowledge remained locked in undocumented tribal knowledge
• Cannot adapt to changing requirements: when regulatory formats changed, the entire pipeline had to be stopped, reconfigured, and redeployed. No capacity for in-flight adaptation

The operations team had built increasingly complex retry logic, branching conditions, and error handlers - but they were fighting the fundamental limitation: traditional workflow engines flatten multi-dimensional problems into a single execution layer.

The solution: 3D Workflow Architecture

CMX had been researching this exact problem. We had identified that human experts solving complex problems naturally operate across multiple cognitive layers: meta-cognitive ("What problem am I solving?"), strategic ("What approach should I take?"), tactical ("What specific steps do I need?"), and operational ("How do I execute this step?"). Traditional workflow engines flatten all of this into one layer - they can only "execute steps," not "decide what steps to take based on evolving understanding."

We proposed and built 2yFlow: a 3D distributed workflow engine that introduces abstraction layers with bidirectional inter-layer connectivity.

The three dimensions

• X-axis (Temporal): Sequential task dependencies within a layer - the traditional causal chain: "Task B needs Task A's output"
• Y-axis (Parallel): Concurrent task alternatives and variations at the same abstraction level - enabling A/B testing, multi-hypothesis reasoning, and fault tolerance
• Z-axis (Abstraction): Hierarchical problem decomposition from meta-cognitive goals down to operational execution

The four abstraction layers

• Layer 3 - Meta-Cognitive: High-level goals, success criteria, constraints. "Process this loan application with 99% accuracy and regulatory compliance"
• Layer 2 - Strategic: Approach selection, resource allocation, risk assessment. "This document has handwritten annotations - use OCR + LLM hybrid strategy"
• Layer 1 - Tactical: Task decomposition, scheduling, dependency management. "Run these three extraction methods in parallel and merge the best results"
• Layer 0 - Operational: Concrete tool invocations, API calls, computations. "Call the Mistral model with this structured prompt on this text block"

The critical innovation: bidirectional feedback

Not just top-down decomposition, but bottom-up learning and lateral coordination. When an operational extraction fails, the signal propagates upward: the tactical layer adjusts the task plan, the strategic layer may switch approaches entirely, and the meta-cognitive layer reassesses whether the current goals are achievable with available data.

This mirrors how human experts solve problems - moving fluidly between strategic thinking, tactical planning, and operational execution while maintaining resilient feedback loops.

What CMX delivered

Phase 1 - Architecture design and formal model (6 weeks)

We designed the 3D workflow architecture grounded in cognitive science foundations and formal verification:

• Thought-action space model: a unified architecture where thoughts and actions coexist at multiple abstraction levels, with bidirectional flow enabling action-informed thinking and thought-guided action
• Inter-layer communication protocol: formal definitions of how signals propagate between layers: decomposition commands flow downward, result aggregations flow upward, and coordination signals flow laterally
• Emergent intelligence architecture: integration points for reinforcement learning (RL), deep learning (DL), and large language models (LLMs) within the workflow engine, enabling the system to learn from every execution

Phase 2 - Engine implementation (10 weeks)

We built the 2yFlow distributed engine:

• 3D task space: tasks exist in a three-dimensional coordinate system, with explicit temporal, parallel, and abstraction relationships
• Self-healing execution: when a task fails, the engine does not simply retry or abort. It escalates to the strategic layer, which can select alternative approaches, redistribute resources, or reformulate the problem
• Cross-execution memory: a knowledge layer that persists learning across workflow runs. Successful extraction strategies for specific document types are remembered and reapplied. Failure patterns are cataloged and avoided
• Dynamic reconfiguration: workflows adapt in-flight based on runtime context, without stopping or redeploying

Phase 3 - Document processing deployment (6 weeks)

We deployed 2yFlow for the client's document processing operation:

• Loan document analysis: multi-format financial documents processed through adaptive extraction strategies that select themselves based on document characteristics
• Regulatory correspondence: complex, ambiguous documents where the strategic layer chooses between template matching, LLM-based extraction, and human escalation based on confidence assessments
• Claims processing: high-volume standardized documents where the system learns optimal processing paths and pre-emptively routes edge cases to specialized strategies

Phase 4 - Evaluation and knowledge transfer (2 weeks)

• Comparative benchmarks against the client's previous DAG-based pipeline
• Operational documentation and team training
• Integration architecture for extending 2yFlow to additional business domains

Results

• +34% extraction accuracy on complex, non-standardized documents compared to the previous static pipeline
• Self-healing recovery: workflow failures that previously required manual intervention are now handled autonomously through strategic-layer reconsideration
• Cross-execution learning: the system improves with each batch, reducing error rates on previously-seen document formats by an additional 12% per month
• 4 cognitive layers operating in production: the first enterprise deployment of a workflow engine that reasons across abstraction levels, not just executes tasks
• Dynamic adaptation: when a new regulatory document format appeared mid-quarter, the system adapted its processing strategy without pipeline changes

Why this matters

The entire workflow orchestration industry - Airflow, Prefect, Temporal, Camunda - is built on the same 2D DAG paradigm. These engines are powerful for predictable, sequential processes. But modern enterprises face problems that are multi-dimensional, ambiguous, and evolving: recruitment workflows, compliance processes, document synthesis, strategic decision-making.

2yFlow represents CMX's conviction that the next generation of enterprise automation requires architectural thinking, not just better tooling. Workflows need to reason, learn, and adapt - the same way human experts do.

This is not theoretical. It is running in production, processing real documents, making real decisions, and learning from every execution.