05 / 12Prism Systems · Cloud-Native & Developer Infrastructure
Logs you can search in milliseconds, PII already masked
A Loki-style log aggregation engine: auto-detecting parsers for JSON, logfmt, nginx and syslog, ingest-time PII and secret redaction, label-indexed chunks with trigram bloom filters, and a LogQL query engine with metric aggregations.
Ingest six services, query them live
Six services write four log formats into the pipeline. Trigger a DB outage or credential stuffing, toggle redaction, then run LogQL. Chunks the query reads glow, chunks the bloom filter rules out flash red.
What's inside the engine
Real algorithms, not mock-ups. The simulator above runs the exact same code you import.
Format auto-detection
JSON (nested fields flattened to http_status), logfmt, nginx/apache combined with request_time, RFC 3164 syslog and plain text.
Normalisation
Levels from level, severity, pino numbers, syslog PRI or HTTP status. Timestamps from ISO, epoch s/ms/ns or nginx dates.
Ingest-time redaction
Emails, Luhn-valid cards, Aadhaar, phones, bearer tokens, JWTs, sk_live_ / AWS / GitHub keys, passwords and IPv4 are masked before storage.
Label-indexed chunks
Only low-cardinality labels are indexed. Each stream is a list of chunks with min/max time, sealed by size or age.
Trigram bloom filters
Every chunk carries a 32 Kbit bloom of its trigrams, so |= "timeout" skips chunks that cannot match. No false negatives.
LogQL engine
Selectors, line filters, json/logfmt/pattern/regexp, label filters with and/or, line_format, unwrap, drop/keep.
Metric queries
rate, count_over_time, bytes_rate, quantile_over_time, avg/min/max/sum_over_time, sum/avg/count by, topk, + - * /.
Guard rails
Retention compaction, a stream cardinality limit, out-of-order tolerance inside open chunks, and helpful parse errors.
From request to result
Parse
Each line's format is detected, fields are extracted and flattened, and level and timestamp are normalised.
Redact
Nine rules mask PII and secrets in the raw line. Counts per rule are tracked for compliance reporting.
Label
Static labels (service, env) plus the normalised level form the stream key. High-cardinality fields stay in the line.
Chunk
Lines append to the stream's open chunk, which updates min/max time and the trigram bloom, then seals at 512 lines or 10 minutes.
Query
The planner resolves streams through the label index, prunes chunks by time and bloom, then streams lines through the compiled pipeline and range/vector aggregations.
System design
Every box is a module in src/. Data flows left to right.
Ship it in your own stack
Zero dependencies. Works in Node.js 18+ and modern browsers.
import { LogStore, Ingestor, query, explain } from "prism-logs";
const store = new LogStore({ chunkLines: 512, retentionMs: 7 * 864e5 });
const ingest = new Ingestor(store, { labels: ["service", "env"] }); // redaction on by default
ingest.push('{"level":"error","msg":"timeout acquiring connection","db":{"wait_ms":7200},"user":"riya@gmail.com"}', { service: "checkout", env: "prod" });
ingest.push('ts=2026-10-26T10:00:00Z level=info msg="payment captured" psp=razorpay latency_ms=142', { service: "payments", env: "prod" });
const logs = query(store, '{service="checkout"} |= "timeout" | json | db_wait_ms > 6000', { limit: 50 });
console.log(logs.entries[0].line); // ..."user":"r***@gmail.com"...
console.log(logs.stats); // { chunksScanned, chunksBloom, lines, bytes, ms, ... }
const errs = query(store, 'sum by (service) (rate({level="error"}[1m]))', { start, end, step: 15e3 });
console.log(errs.series); // [{ labels: { service }, values: [...] }]
console.log(explain(errs.ast));$ npx github:krishyaa06/prism-logs query '{service="checkout"} |= "timeout" | json | db_wait_ms > 6000'
$ npx github:krishyaa06/prism-logs query 'sum by (service) (count_over_time({level=~"error|fatal"}[1m]))'
$ npx github:krishyaa06/prism-logs query '{service="app"} | logfmt | latency_ms > 300' --file app.log --service app
$ cat app.log | npx github:krishyaa06/prism-logs redact -
$ npx github:krishyaa06/prism-logs bench --n 100000# use the CLI without installing
$ npx github:krishyaa06/prism-logs --help
# or add the engine to your project
$ npm install github:krishyaa06/prism-logs
# run the simulator locally
$ git clone https://github.com/krishyaa06/prism-logs.git && cd prism-logs
$ npx serve . # then open http://localhost:3000