Dnp Workflow Engine

Workflow Engine Documentation

WorkflowEngine — Detailed Documentation

=====================================

This document provides a comprehensive overview of the WorkflowEngine project: features, configuration, plugin model, steps and examples, operational notes, and the set of additional support items required to reach feature parity with WWF (Windows Workflow Foundation) where appropriate.

Table of contents

  • Overview
  • Architecture
  • Plugin model and discovery
  • Configuration and appsettings
  • Built-in steps and examples
  • Start / Validate / MoveFile / Complete
  • If / RuleEvaluator
  • ParallelForEach (durable)
  • Retry wrapper
  • EmitMetric (Counter & Histogram)
  • HttpCall (auth & resiliency)
  • Switch (regex & glob)
  • Scheduler (cron recurring)
  • HumanTask (human-in-the-loop)
  • UI and API endpoints
  • Event model & resume behavior
  • Testing & simulation
  • Operational considerations
  • Support & roadmap — features needed to match WWF support

Overview

--------

WorkflowEngine is a modular, plugin-driven orchestration and automation platform intended for a wide range of scenarios. Typical uses include file/record processing, message-driven microservices, API orchestration, scheduled jobs, human-in-the-loop workflows, and custom connector pipelines. It provides:

  • Declarative workflows (JSON-based) supporting linear and state-machine modes.
  • Plugin step factories and dynamic step discovery.
  • Roslyn-based rules engine to evaluate expressions against WorkflowContext.
  • Durable workflow state persisted to DB to enable suspend/resume patterns.
  • Parallel and iterative steps with durable resume support.
  • Service invocation step with robust resiliency features.
  • Metrics emission and basic UI for inspection.
  • Cron-based scheduler for recurring jobs and a schedule management UI.

New/Updated capabilities (high-level)

-------------------------------------

This project has received a set of enhancements to support human workflows, event-driven resume, richer events, and improved test coverage. Key additions since earlier documentation:

  • HumanTask step and repository: steps can create persistent human tasks and store the created task id into workflow context.
  • Razor Pages UI + API for human tasks: a simple /Tasks Razor page and api/tasks endpoints to list, assign, and complete tasks.
  • Event model extended: WorkflowEvent now includes EventType and MetadataJson to carry semantic type and structured metadata.
  • Event namespacing: when events are processed the event payload and metadata are exposed under namespaced keys in WorkflowContext.ContextData (e.g. event, eventpayload, eventpayload:{EventType}, event_metadata:{EventType}) to avoid collisions with existing context keys.
  • WorkflowEventProcessor enhancements: payload/metadata are parsed and attached to the resumed WorkflowContext, and trigger keys (triggerevent, triggerevent_type) are set to enable event-driven transitions.
  • State-machine event-driven transitions: transitions with OnEvent now match either the event's EventType or EventName, enabling workflows to be resumed and routed by semantic event types.
  • Resume via events: completing a human task enqueues a WorkflowEvent (type TaskCompleted) referencing the workflow instance; the event processor picks it up and resumes the suspended workflow.
  • Tests: new unit/integration tests cover human task creation, event processing, namespaced payload/metadata, and state-machine resume behavior. The SimulationHarness continues to provide in-process deterministic testing.
  • Workflow definition versioning: workflow entities now include a Version and suspended WorkflowState records store the WorkflowDefinitionVersion and an optional WorkflowDefinitionJson snapshot so instances can be resumed against the exact definition that started them. Promote/demote operations and versioned registration are supported.

Architecture

------------

Key components (summary):

  • WorkflowExecutor — executes workflow definitions step-by-step, handling control flow and state persistence.
  • WorkflowRegistry — registers steps and workflows, discovers plugin DLLs and additional DLLs from configuration.
  • WorkflowStateRepository — persists WorkflowState objects containing ContextData and current step info.
  • WorkflowEventProcessor — background service that polls WorkflowEvent entries and resumes suspended workflows. It now merges event payload/metadata into the resumed WorkflowContext under namespaced keys.
  • SchedulerService — cron interpreter (NCrontab) that triggers recurring schedules and creates trigger events.
  • SimulationHarness — test helper used to run workflow definitions in-process for unit and integration tests.

Plugin model and discovery

--------------------------

Discovery order and priority remain unchanged. The HumanTask step and other built-in steps are registered as built-in factories and can be overridden by plugins placed in the plugin folder or additional plugin DLLs configured through appsettings.

Built-in steps and examples

---------------------------

HumanTask (new)

  • IsHumanTask flag and related properties are available in WorkflowStepConfig to mark a step that will create a human task.
  • Configurable expressions: HumanTaskTitleExpression, HumanTaskDescriptionExpression, HumanTaskAssigneeExpression accept literals or expressions using the project's rules evaluator (prefix expr: supported in the simple evaluator used).
  • HumanTaskResultKey controls where the created task id is stored in the workflow ContextData.
  • HumanTaskRepository persists tasks and exposes query and assignment APIs used by the UI and API.

Event-driven state machines

  • TransitionConfig.OnEvent may reference a semantic event name. The engine now evaluates event-driven transitions by checking the trigger values added to ContextData by the WorkflowEventProcessor.
  • When WorkflowEventProcessor handles an event it will (if it finds a suspended workflow instance):
  • Parse Payload and MetadataJson into CLR objects.
  • Store them under namespaced keys in ContextData (e.g. eventpayload, eventpayload:TaskCompleted).
  • Set triggerevent and triggerevent_type to the event's name and type so transitions can match on either.
  • When resuming a suspended instance the engine will attempt to load the exact workflow definition version recorded on the WorkflowState:
  • First it attempts to load the persisted workflow entity by name+version from the DB and uses its JSON to resume.
  • If the entity is not available it will fall back to an embedded WorkflowDefinitionJson snapshot stored on the WorkflowState (if present).
  • If neither is available it falls back to the registry/default workflow and logs a warning. This preserves deterministic behavior for in-flight instances when workflow definitions change.

UI and API endpoints

--------------------

Human tasks UI (Razor Pages)

  • /Tasks Razor page lists open tasks and provides simple assign/complete forms.
  • The UI uses the new api/tasks endpoints for actions.

Tasks API (added)

  • GET /api/tasks?assignedTo={user} — list open tasks (optionally filter by assignee).
  • POST /api/tasks/assign — assign a task (body: { taskId, user }).
  • POST /api/tasks/complete — complete a task (body: { taskId, result, workflowInstanceId? }). Completing a task will mark the task completed and — when linked to a workflow instance — enqueue a WorkflowEvent of EventType = TaskCompleted referencing the workflow instance id so the WorkflowEventProcessor can resume the instance.

Workflow administration (new)

  • Admin UI: a new Razor Pages admin page /Admin/Workflows lists workflows grouped by name and shows versions. It highlights the production version and shows Created/Promoted timestamps. The page supports Promote and Demote actions with confirmation prompts.
  • Admin API: api/workflows endpoints to list workflows and perform POST /api/workflows/promote/{id} and POST /api/workflows/demote/{id}. The API uses repository methods to implement promote/demote logic and preserve version semantics.

Event model & resume behavior

-----------------------------

  • WorkflowEvent now has:
  • EventType — optional semantic type of the event (e.g. TaskCompleted).
  • MetadataJson — optional free-form JSON for structured metadata.
  • When events are processed the engine exposes in WorkflowContext.ContextData the following keys for the resumed workflow:
  • _triggerevent — the event's EventName (string)
  • _triggerevent_type — the event's EventType (string)
  • __event — dictionary with EventName, EventType, Payload, Metadata (full event info)
  • __event:{EventType} — same full event info namespaced by event type
  • eventpayload and eventpayload:{EventType} — parsed payload object (dictionary/list/primitive)
  • eventmetadata and eventmetadata:{EventType} — parsed metadata object
  • Namespacing is used to avoid accidental collisions with other context keys; the engine intentionally avoids merging payload object properties into top-level context keys.

Database & migrations

  • Example create scripts under Example.Service/Scripts/ have been updated for all supported providers to include the new WorkflowStates table (stores WorkflowDefinitionVersion, WorkflowDefinitionJson) and Workflows.Version column. Provider-specific SQL migration examples are available in Migrations/ for SQL Server, MySQL, and PostgreSQL. Convert these into EF Core migrations if you use EF migrations in production.

Testing & simulation

--------------------

  • SimulationHarness remains available for in-process deterministic tests.
  • New tests were added covering:
  • Human task step creation and persistence.
  • WorkflowEventProcessor parsing payloads/metadata and populating namespaced keys.
  • State-machine resume on EventType match (transition OnEvent matches EventType or EventName).
  • Versioned resume tests: ensure instances started with workflow version N resume against version N even after a newer version is registered.

Operational considerations

-------------------------

  • Resume via events is intentionally implemented by enqueuing WorkflowEvent entries and letting WorkflowEventProcessor pick them up. This is safer for multi-node deployments compared to invoking the executor directly from the UI/API because it guarantees a single resume path and respects host lifecycle/backoff behavior.
  • Namespacing payload/metadata prevents accidental overwrites of existing context keys. Workflows should read payload values from the namespaced keys or from __event.
  • If you need to expose payload properties to top-level context keys you can do so explicitly in an entry action step.

Support & Roadmap — Features to reach WWF parity (status)

--------------------------------------------------------

Below are the roadmap items and the current status based on implemented features in this branch.

1) Designer and visual workflow authoring — Not implemented

  • No web-based authoring or drag/drop designer yet.

2) Rich state machine constructs and long-running workflows — Partial

  • State machine execution, entry/exit actions and event-driven transitions are implemented. Additional features (history visualization, state timeouts, durable bookmarks) are planned.

3) Persistence and versioning — Implemented (core)

  • Core versioning is implemented: workflows store a Version and WorkflowState records include WorkflowDefinitionVersion and optional WorkflowDefinitionJson snapshots. The executor resumes using the recorded version when available. Promotion/demotion flows and simple version management UI/API are implemented. Remaining work: automated migration tooling, in-flight instance migration helpers, and richer version history management in the UI.

4) Distributed scheduling and leader election — Not implemented

  • Scheduler is single-node correct; leader election for multi-node clusters remains to be implemented.

5) Durable timers and durable bookmark support — Partial

  • Durable resume via persisted WorkflowState and WorkflowEvent has been implemented (tasks enqueue events to resume workflows). "Bookmarks" and richer durable timer semantics are still an area for enhancement.

6) Visual debugging and step-level tracing — Not implemented

  • Basic logs are recorded; a full visual debugger UI is not present.

7) Security/permissions on plugins and runtime code — Not implemented

  • Plugin loading is permissive; consider signed plugin validation and sandboxing for production.

8) Advanced expression and workflow language parity — Not implemented

  • Expression support exists via the rules evaluator; richer typed expression models are not implemented.

9) Workflow authoring and lifecycle tooling — Partial

  • Workflows can be saved/registered; CI/CLI tooling for promotion and lifecycle actions is still work to be done.

10) High-availability and multi-region coordination — Not implemented

  • Coordination and active/active strategies are not part of this change.

11) Testing and simulation harness — Implemented

  • SimulationHarness and numerous integration tests are present and used in CI. New tests for event namespacing and state-machine resume were added.

12) Comprehensive observability — Partial

  • The engine emits metrics and logs; integration with OpenTelemetry/exporters is left to host configuration. Metric key conventions and dashboards remain to be created.

13) Out-of-the-box activity library — Partial

  • Additional steps (HumanTask, HttpCall resiliency, ParallelForEach, EmitMetric) expand the activity set. More connectors and activities can be added.

14) Human workflows and task lists — Implemented (basic)

  • Persistent HumanTask model and repository, a Razor Pages UI (/Tasks) for listing/assigning/completing tasks, and API endpoints were added. Completing a task enqueues a WorkflowEvent that resumes the related suspended workflow. Further features (task history, escalation, roles) are future work.

15) Compatibility layers — Not implemented

  • No compatibility layer for WWF XAML provided.