# Tutorial: AI eval & field monitoring

| | |
|---|---|
| **Audience** | Operators running the 9-host field trial or local eval |
| **Time** | 20-40 minutes |
| **Outcome** | You can find Mean Session Length and Novelty Capture Rate in the dashboard, run the eval suite, and compare stub / SLM / Ollama tiers |

## What you will learn

1. Find Mean Session Length in the field-trial dashboard
2. Read Novelty Capture Rate without over-trusting it
3. Run the pinned shell+redis realism eval suite locally
4. Run the local SLM tier and compare latency/output
5. Compare stub / SLM-only / full pipeline side by side in the dashboard

## Prerequisites

- Local field-trial collector + dashboard (`scripts/field_trial/`)
- Optional: Ollama + `RUN_OLLAMA_EVAL=1` for eval CI section

## 1. Mean Session Length

Open `http://127.0.0.1:8787/` (default dashboard). Overview columns include session
duration (ms) aggregated from sensor `metrics.sample` events. Metric definitions:
[Decoy realism AI](../how-to/decoy-realism-ai.md).

## 2. Novelty Capture Rate

The **Novelty** column is the share of sessions tagged `passive:no_signature_matched`
(no known ATT&CK signature). It is a **passive KPI**, not a detection/block signal.

## 3. Eval suite (Phase 1)

Pinned allowlisted shell leaves + `redis.value` realism checks against live Ollama:

```bash
cd services/ai-broker
ollama serve   # separate terminal, model pulled
RUN_OLLAMA_EVAL=1 pytest tests/eval/ -q
```

Without `RUN_OLLAMA_EVAL=1` or without Ollama, pytest reports **skipped** (not failure).
See `services/ai-broker/tests/eval/README.md`.

## 4. Local SLM tier (Phase 1)

```bash
export CYBERHALLUCINET_BROKER_ADAPTER=local
export CYBERHALLUCINET_LOCAL_MODEL_PATH=/path/to/model.gguf
export CYBERHALLUCINET_LOCAL_MODEL_DIGEST=<sha256-hex>
export CYBERHALLUCINET_LOCAL_MODEL_DEVICE=cpu
# Optional tiered routing:
export CYBERHALLUCINET_TIER_ROUTING_ENABLED=1
```

`redis.value` and single-line `shell.stdout` fills use the local tier first;
digest mismatch still fails closed exactly like before runtime wiring.

Compare latency against `static_stub` and `ollama` in the dashboard once Phase 2
field spread lands.

## 5. Three-tier field comparison (Phase 2)

Spread the three CHN field hosts across tiers for dashboard comparison rigor:

| Host | Tier | Adapter |
|------|------|---------|
| `chn-us-a` | Control | `static_stub` |
| `chn-eu-a` | SLM-only | `local` |
| `chn-ap-a` | Full pipeline | `ollama` + `CYBERHALLUCINET_TIER_ROUTING_ENABLED=1` |

**Human-approval checkpoint: operator sign-off required before redeploy** to live
Internet-facing hosts. Config overlays:
`scripts/field_trial/stacks/chn/overlays/`: plan in
[`RUN_MANIFEST_TIER.md`](../../scripts/field_trial/RUN_MANIFEST_TIER.md).

After approved deploy, compare Mean Session Length, Dwell Time Distortion Ratio,
and generative slot drill-down across hosts. Do not redeploy without reviewing
inference cost and behavioral exposure on real attacker traffic.