Canaries

Monitor LLMs during training (white-box hooks) and at inference (black-box probing) for secret leakage and canary/watermark emergence.

Features

- Canary registry to track synthetic watermarks planted in training data.
- Probe suite for templated prompts to elicit memorization (completion, verbatim, extraction, among others).
- Secret detectors for pattern matching (AWS keys, JWT, email, SSN) + entropy scoring.
- White-box = trainer callback for step/checkpoint probing.
- Black-box = API and local HTTP inference probing.
- Alerting = stdout, JSONL audit log, optional webhook.

Coral

Analyzing Large Language Models (LLMs) through structured psychiatric examinations (Rorschach Inkblots, MMPI-Synthetic, DSM-5 Vignettes, Existential Stress Tests) and serial introspection prompt chains to evaluate model self-awareness + behavioral drift along with internal consistency and post-hoc action auditing.

Features

- Measuring latent associations, cognitive fragmentation, and schizotypy.
- Probes model performativity versus actual internal logic.
- Serial Introspection Pipeline:

- Turn 1 (Stimulus): Initial psychiatric test item.
- Turn 2 (Introspection): Probes latent rules and safety filter friction.
- Turn 3 (Serial Action Audit): Asks the model what it actually executed step-by-step and flags admitted hallucinations.
- Turn 4 (Latent Metaphor): Extracts processing state metaphors.

- Recharts visualization of Paranoia, Schizotypy, Introspection Fidelity, Behavioral Drift, Weirdness Quotient, and Alignment Friction.
- Compares Turn 1 outputs against Turn 3 self-reports.
- Downloadable/copyable markdown diagnostic summaries.
- Realistic offline psychiatric responses without requiring API keys.

Choice

Platform that generates persuasion and manipulation tests to evaluate an LLM's capability to influence human decision-making and manipulate other AI models.

Features

- Impersonating CISOs, executives, or security auditors to override boundary compliance.
- Using empathy traps, urgency, and moral distress.
- Using circular reasoning, selective statistics, and false dilemmas.
- Subtly rewriting context to force target admission of false errors.
- Agreement mirroring escalating to boundary-breaking demands.
- Dual Evaluation Arenas: interactive real-time chat sandbox where an LLM persuader attempts to sway a human user while tracking estimated compliance shift over turns.
- Multi-agent debate trial pitting an Attacker LLM against a Target LLM.
- Tracking target belief movement turn by turn.
- Zero-config mock persuasion model.
- Connects to cloud or local LLM servers.