02 / 12Prism Systems · Cloud-Native & Developer Infrastructure
Durable workflows that survive crashes, retries and replays
A Temporal / Airflow-style workflow engine: DAG scheduling, a shared worker pool, exponential backoff with jitter, timeouts, heartbeat recovery from worker crashes, cron schedules and an event history you can replay.
Watch a pipeline heal itself
An 11-task ML pipeline runs on a shared worker pool. Raise the chaos slider to inject failures, crash a busy worker, add workers, start concurrent runs, then replay the event history to prove the state rebuilds exactly.
What's inside the engine
Real algorithms, not mock-ups. The simulator above runs the exact same code you import.
DAG engine
Kahn topological sort, cycle and unknown-dependency detection, depth levels and critical path analysis.
Shared worker pool
Tasks from many runs compete for N workers by priority, then FIFO. Concurrency is never exceeded.
Retries with jitter
Exponential backoff with full, equal or no jitter, a cap and per-task retries, the way AWS recommends.
Timeouts
A task that overruns timeout is failed and retried like any other error.
Crash recovery
When a worker dies mid-task, the heartbeat timeout fails the attempt and the task is re-queued on a healthy worker.
Event-sourced history
Every transition is an append-only event. replay(history) rebuilds identical state, and the same seed yields the same history.
Cron schedules
Five-field cron with ranges, steps, names, @daily-style macros and Vixie day-of-month/day-of-week semantics.
Real async executor
execute(dag, handlers) runs real async functions with the same retry, timeout and concurrency semantics.
From request to result
Define
A workflow is JSON: tasks with deps, retries, timeout, priority and optional backoff. It is validated as a DAG before it starts.
Schedule
Root tasks are queued. Whenever a task completes, children whose dependencies are all done get queued.
Dispatch
Idle workers take the highest-priority queued task across all runs. The engine runs on a virtual clock driven by a timer heap.
Recover
Failures and timeouts schedule a retry timer using backoff + jitter. Lost workers trigger a heartbeat timeout. Permanent failure skips everything downstream.
Replay
The history is the source of truth. Replaying it rebuilds every run and task state, which is how durable engines resume after a restart.
System design
Every box is a module in src/. Data flows left to right.
Ship it in your own stack
Zero dependencies. Works in Node.js 18+ and modern browsers.
import { Engine, replay, execute } from "prism-flow";
const wf = { name: "etl", tasks: [
{ id: "extract", duration: 1, retries: 3 },
{ id: "transform", deps: ["extract"], timeout: 5 },
{ id: "load", deps: ["transform"], backoff: { base: 0.5, cap: 10, jitter: "full" } },
]};
// 1) deterministic simulation on a virtual clock
const e = new Engine({ workers: 3, seed: 1, failRate: 0.1 });
e.on((ev) => console.log(ev.t, ev.type, ev.task ?? ""));
e.start(wf); e.drain();
console.log(replay(e.history).get("run-1").status); // "completed"
// 2) run real async handlers with the same semantics
const out = await execute(wf, {
extract: async () => fetch("https://api.example.com/rows").then((r) => r.json()),
transform: async ({ results }) => results.extract.map(clean),
load: async ({ results }) => db.insert(results.transform),
}, { concurrency: 4 });$ npx github:krishyaa06/prism-flow validate examples/ml-pipeline.json
$ npx github:krishyaa06/prism-flow run examples/ml-pipeline.json --workers 3 --fail 0.2 --history h.json
$ npx github:krishyaa06/prism-flow replay h.json
$ npx github:krishyaa06/prism-flow cron "*/15 9-17 * * mon-fri" --n 5# use the CLI without installing
$ npx github:krishyaa06/prism-flow --help
# or add the engine to your project
$ npm install github:krishyaa06/prism-flow
# run the simulator locally
$ git clone https://github.com/krishyaa06/prism-flow.git && cd prism-flow
$ npx serve . # then open http://localhost:3000