Generative AIProtected2026

Bastion

AI cybersecurity assessment SaaS

Compresses a 10–12-week security engagement into one workflow: a multi-stage Claude pipeline (an estimated ~80% token reduction) where every finding cites the evidence it came from and a human signs off.

NIST CSF 2.0 · evidence to findingsReal pipeline output · precomputed

Evidence · Access Control Policy (POL-AC-002)6 controls in scope

# Access Control Policy
Document ID: POL-AC-002
Version: 2.0

[…]

### 3.3 Account Deprovisioning
Upon employee termination, all access must be revoked within 24 hours. HR must notify IT Security via the termination checklist. Remote access (VPN, email) is disabled immediately. Application access is removed within 24 hours.

> NOTE: The current offboarding process relies on a manual ticketing system. HR submits a ticket and IT processes it during business hours. Average time to full access revocation is 2-3 business days, not 24 hours as specified.

[…]

### 4.1 Periodic Reviews
Access reviews should be conducted quarterly for all systems containing sensitive data. Reviews are performed by system owners with IT Security oversight.

> OBSERVATION: As of the most recent audit (Q2 2024), no formal periodic access review has been completed in the last 12 months. System owners report they review access "informally" but no documentation exists.

[…]

## 7. Third-Party Access

Third-party vendors requiring system access are provisioned through the Vendor Access Request form. Vendor accounts are time-limited (maximum 90 days, renewable). Vendor sessions are not monitored or recorded.

> NOTE: Three vendors currently have persistent VPN access that has not been reviewed or time-limited: CloudOps Inc (since 2022), DataMigrate LLC (since 2023), and SecureAudit Partners (since 2021). Two of these vendors have domain admin equivalent privileges.

Excerpt of a synthetic policy about a fictional company. The full document is case pr-00-golden-access-policy in the published 20-case golden eval.

Renders what the analyze stage really produced for this document.

Synthetic scenario, fictional company. No real client or assessment data. The findings are real pipeline output, precomputed: copied verbatim from the cached golden-eval run of case pr-00-golden-access-policy (claude-sonnet-4-20250514, Bastion repo 968c574).

Preview on synthetic controls only, with no real client or assessment data.

Protected work

Bastion runs real cybersecurity assessments on confidential client evidence, so source and data stay private. It's live as an invite-only deployment. Request a walkthrough for a guided tour of the running app.

Request access

Bastion (Next.js 15 + Supabase + Claude) replaces the spreadsheet/Word/PowerPoint sprawl of a 10–12 week security engagement with one workflow. It runs live as an invite-only deployment (a guided walkthrough is available on request), while this card's preview uses synthetic controls only, with no real client or assessment data.

Its AI is a real multi-stage pipeline: local section filtering → Claude Haiku relevance triage → Claude Sonnet deep analysis (an estimated ~80% token reduction vs. passing the full document corpus to Sonnet directly) → Haiku self-verification → bottom-up discovery → a Pinecone RAG layer. Every output cites the evidence it came from and a human approves it.

It supports 5 frameworks (NIST CSF 2.0, CIS v8, ISO 27001, SOC 2, CMMC 2.0), enforces the client/internal split at the database layer via Supabase RLS, and includes Stripe multi-tenancy and PDF/DOCX/PPTX report generation.

Next.js 15
React 19
TypeScript
Supabase
Claude (Sonnet + Haiku)
Pinecone (RAG)
Stripe
Vercel

Architecture · multi-stage Claude gap-analysis pipeline

Evidence vault
Client documents + controls, isolated per tenant at the database layer via Supabase RLS.
Local section filtering
A deterministic pre-filter narrows the corpus before any LLM tokens are spent.
Claude Haiku: relevance triage
A cheap pass keeps only the sections worth deep analysis.
Claude Sonnet: deep gap analysis
Full gap analysis on the survivors (an estimated ~80% fewer tokens than analysing full context).
Claude Haiku: self-verification
A second pass checks each finding back against the cited evidence.
Bottom-up discovery + Pinecone RAG
Surfaces related findings across the vault; every output cites the evidence it came from.
Human sign-off
A consultant approves before anything reaches a client-ready report.

Frameworks · NIST / CIS / ISO / SOC 2 / CMMC: 5
Every finding: cites evidence + human sign-off
Tenant isolation: Supabase RLS (client/internal)
AI token reduction: ~80% (estimated, engagement-observed)
Golden-eval cases: 10/20 (strict bar)
Citation validity: 95.6% (109/114 quotes verbatim)

View measured eval

Where the numbers come from

Golden-eval cases: 10/20 is the strict all-or-nothing per-case bar…
Golden-eval cases: 10/20 is the strict all-or-nothing per-case bar (every critical expectation matched, zero violations), with every case and transcript rendered at /measured. The same run reports the rates underneath: 49/59 critical findings recalled, 109/114 citations verified against the evidence, and 94% severity calibration.
The number is scorer-corrected, not retried: the first run was published raw…
The number is scorer-corrected, not retried: the first run was published raw at 1/20, then a documented scoring review (artifact _provenance.scoringReview; full table in the harness README) audited all 28 missed expectations. 17 were scorer defects (correct gap-to-risk escalations rejected by an exact-type match), 1 was a keyword phrasing miss, 10 were real model misses and still count as misses. Citation checking was also corrected for markdown markup the model faithfully omits; 5 genuinely unverifiable quotes still count against it. The model responses are the unedited first-run outputs throughout; only the scorer and 2 documented keyword nets changed, and no case was re-run.
The harness (scripts/studies/bastion-eval in the portfolio repo) mirrors Bastion's analyze-evidence prompt…
The harness (scripts/studies/bastion-eval in the portfolio repo) mirrors Bastion's analyze-evidence prompt and model id verbatim at the recorded Bastion repo sha and scores findings against expectations authored before the run. Scope is single-stage: only the Sonnet evidence-to-findings call is exercised; the RAG retrieval, bottom-up discovery, and Haiku verification stages are not. Every scenario is a synthetic document about a fictional company, never client material. The unedited first-run model responses are committed at scripts/studies/bastion-eval/responses, and node scripts/studies/bastion-eval/run-eval.mjs --rescore-only re-scores them with zero API access. With the private Bastion checkout present, it validates response hashes against rebuilt prompts; without it, it falls back to offline scoring from the committed responses and case fixtures. Rescore-only preserves generatedAt, so a clean run can reproduce the published artifact byte-for-byte. The raw 1/20 first-run artifact itself was not preserved (the scorer was corrected in place before the repo kept versioned score artifacts), so 1/20 is the review's recorded starting point, not a regenerable file.
AI token reduction: the ~80% is an estimate observed while running engagements…
AI token reduction: the ~80% is an estimate observed while running engagements on the deployed pipeline, not a published measurement. The saving comes from the section-filtering and Haiku-triage stages that the public eval deliberately does not exercise, so the committed eval responses cannot reproduce it; a precise figure requires production token telemetry that only the owner's private deployment records.

A Bastion-generated executive-summary report: a NIST CSF 2.0 maturity assessment cover, a maturity overview (38% average maturity, 74 of 106 subcategories assessed, 3 critical or high gaps, 3 strengths), and a six-function radar chart versus the technology-sector median, over a DRAFT watermark. — The deliverable Bastion produces: a client-ready executive summary that replaces an engagement's slide sprawl. Synthetic data only, the fictional Acme Corporation on Sentinel Advisory Group letterhead with the DRAFT watermark intact; no real client or assessment data.

What I'd improve

The 20-case golden eval this card used to promise as future work is now built, with every case and transcript rendered at /measured. Its first run scored 1 of 20 and was published raw; the mandated scoring review then audited all 28 missed expectations against the cached responses and found the scorer, not the model, caused most of them: it rejected findings the model had correctly escalated from gap to risk, and rejected quotes that matched the evidence's prose but not its markdown markup. With the scorer corrected and the same unedited model responses re-scored, 10 of 20 cases clear the strict bar (every critical expectation matched, zero violations), with 83.1% critical-finding recall, 95.6% citation validity, and 94% severity calibration. The 10 expectations still missed are real model misses: 5 rated below the expected severity, 3 strength findings never produced, 2 seeded gaps not surfaced. Next steps are extending the eval across the full multi-stage pipeline (RAG retrieval, bottom-up discovery, and the Haiku verification pass) and using those real misses to tighten the analyze prompt.

Private · request access

Want something like this? Get in touch →

# Access Control Policy Document ID: POL-AC-002 Version: 2.0 […] ### 3.3 Account Deprovisioning Upon employee termination, all access must be revoked within 24 hours. HR must notify IT Security via the termination checklist. Remote access (VPN, email) is disabled immediately. Application access is removed within 24 hours. > NOTE: The current offboarding process relies on a manual ticketing system. HR submits a ticket and IT processes it during business hours. Average time to full access revocation is 2-3 business days, not 24 hours as specified. […] ### 4.1 Periodic Reviews Access reviews should be conducted quarterly for all systems containing sensitive data. Reviews are performed by system owners with IT Security oversight. > OBSERVATION: As of the most recent audit (Q2 2024), no formal periodic access review has been completed in the last 12 months. System owners report they review access "informally" but no documentation exists. […] ## 7. Third-Party Access Third-party vendors requiring system access are provisioned through the Vendor Access Request form. Vendor accounts are time-limited (maximum 90 days, renewable). Vendor sessions are not monitored or recorded. > NOTE: Three vendors currently have persistent VPN access that has not been reviewed or time-limited: CloudOps Inc (since 2022), DataMigrate LLC (since 2023), and SecureAudit Partners (since 2021). Two of these vendors have domain admin equivalent privileges.

Architecture · multi-stage Claude gap-analysis pipeline

Evidence vault

Client documents + controls, isolated per tenant at the database layer via Supabase RLS.

Local section filtering

A deterministic pre-filter narrows the corpus before any LLM tokens are spent.

Claude Haiku: relevance triage

A cheap pass keeps only the sections worth deep analysis.

Claude Sonnet: deep gap analysis

Full gap analysis on the survivors (an estimated ~80% fewer tokens than analysing full context).

Claude Haiku: self-verification

A second pass checks each finding back against the cited evidence.

Bottom-up discovery + Pinecone RAG

Surfaces related findings across the vault; every output cites the evidence it came from.

Human sign-off

A consultant approves before anything reaches a client-ready report.

Where the numbers come from

Golden-eval cases: 10/20 is the strict all-or-nothing per-case bar…

Golden-eval cases: 10/20 is the strict all-or-nothing per-case bar (every critical expectation matched, zero violations), with every case and transcript rendered at /measured. The same run reports the rates underneath: 49/59 critical findings recalled, 109/114 citations verified against the evidence, and 94% severity calibration.

The number is scorer-corrected, not retried: the first run was published raw…

The number is scorer-corrected, not retried: the first run was published raw at 1/20, then a documented scoring review (artifact _provenance.scoringReview; full table in the harness README) audited all 28 missed expectations. 17 were scorer defects (correct gap-to-risk escalations rejected by an exact-type match), 1 was a keyword phrasing miss, 10 were real model misses and still count as misses. Citation checking was also corrected for markdown markup the model faithfully omits; 5 genuinely unverifiable quotes still count against it. The model responses are the unedited first-run outputs throughout; only the scorer and 2 documented keyword nets changed, and no case was re-run.

The harness (scripts/studies/bastion-eval in the portfolio repo) mirrors Bastion's analyze-evidence prompt…

The harness (scripts/studies/bastion-eval in the portfolio repo) mirrors Bastion's analyze-evidence prompt and model id verbatim at the recorded Bastion repo sha and scores findings against expectations authored before the run. Scope is single-stage: only the Sonnet evidence-to-findings call is exercised; the RAG retrieval, bottom-up discovery, and Haiku verification stages are not. Every scenario is a synthetic document about a fictional company, never client material. The unedited first-run model responses are committed at scripts/studies/bastion-eval/responses, and node scripts/studies/bastion-eval/run-eval.mjs --rescore-only re-scores them with zero API access. With the private Bastion checkout present, it validates response hashes against rebuilt prompts; without it, it falls back to offline scoring from the committed responses and case fixtures. Rescore-only preserves generatedAt, so a clean run can reproduce the published artifact byte-for-byte. The raw 1/20 first-run artifact itself was not preserved (the scorer was corrected in place before the repo kept versioned score artifacts), so 1/20 is the review's recorded starting point, not a regenerable file.

AI token reduction: the ~80% is an estimate observed while running engagements…

AI token reduction: the ~80% is an estimate observed while running engagements on the deployed pipeline, not a published measurement. The saving comes from the section-filtering and Haiku-triage stages that the public eval deliberately does not exercise, so the committed eval responses cannot reproduce it; a precise figure requires production token telemetry that only the owner's private deployment records.

What I'd improve