> ## Documentation Index
> Fetch the complete documentation index at: https://docs.medera.info/llms.txt
> Use this file to discover all available pages before exploring further.
# Overview of Medera Multimodal Sensing
> Vocal acoustics, facial physiology, and neurobehavioral fusion for behavioral health.
Medera Multimodal Sensing is specifically designed for behavioral and mental health. The engines below provide real-time access to clinical signal channels. Selecting the right engine depends on your use case.
Review the [Languages page](/about/languages) to see language support per surface and the [Architecture page](/multimodal/architecture) for engine deep dives.
## Medera Multimodal engines
F0, jitter, shimmer, HNR, MFCC, prosodic features, and clinical markers from speech.
rPPG-derived HR, BP, HRV (SDNN, RMSSD, LF/HF, SD1, SD2), respiration, stress.
15 named neurobehavioral constructs across 5 domains fused from facial, vocal, and assessment context.
***
## Architecture
***
## Engine functionality
| | Vocal Acoustic | Facial Physiological | Neurobehavioral Construct Construct |
| :------------------------- | :----------------------------------------------- | :------------------------------- | :------------------------------------ |
| **Input** | 16 kHz mono PCM | Video frames | Vocal + Facial features + Assessments |
| **Minimum duration** | 1.0 s | 30 s (HR) / 60 s (HRV, BP) | — |
| **Library** | audio analysis library + speech analysis library | OpenCV + rPPG library + SciPy | NumPy + Pandas |
| **Outputs** | F0, jitter, shimmer, HNR, MFCC, clinical markers | HR, BP, HRV, RR, stress, PNS/SNS | 15 named constructs |
| **Confidence calibration** | 0–1 per family | 0–1 + SNR > 3.0 gate | `VERY_HIGH` → `VERY_LOW` |
| **Failure mode** | `unavailable` status | `confidence: 0.0` | `requires_human_review: true` |
***
## Endpoints
| Method | Path | Use |
| :----- | :------------------------------------------- | :----------------------------------------------- |
| `POST` | `/api/multimodal-therapy/analyze-session` | Full multimodal analysis + biomarker persistence |
| `POST` | `/api/multimodal-therapy/analyze-audio-only` | Audio-only multimodal analysis |
| `GET` | `/api/multimodal-therapy/health` | Engine availability + dependency status |
## Confidence bands
| Band | Range | Behavior | |
| :---------- | :------ | :------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `VERY_HIGH` | ≥ 0.95 | Auto-pass through to clinician | |
| `HIGH` | ≥ 0.85 | Auto-pass through | |
| `MODERATE` | ≥ 0.70 | Surfaced normally | |
| `LOW` | ≥ 0.50 | `requires_human_review: true` | |
| `VERY_LOW` | \< 0.50 | `requires_human_review: true` + flagged for review queue | If signal quality fails (SNR ≤ 3.0 on facial, voiced fraction too low on vocal), the engine returns `confidence: 0.0` and the analyzer falls back to a Medera reasoning transcript-inferred analysis — never fabricated values. |
***
## What's next
Pipeline deep dive from raw signal to canonical clinical payload.
Stream your first multimodal session end-to-end.
15 named constructs across 5 domains.
Agent-level documentation for in-session multimodal.
Contact us if you need help selecting the right engine or have questions about configuring requests. [medera.info/contact](https://medera.info/contact)