Quickstart — Co-Therapy multimodal

The Co-Therapy Agent is a silent multimodal partner — it never speaks to the patient. It runs the Vocal Acoustic Engine, the Facial Physiological Engine, and the Neurobehavioral Construct Computer to produce a canonical clinical payload with 15 named construct activations.

Pipeline

Create the session

cURL curl -X POST https://api.medera.info/api/co-therapy-sessions \ -H "X-API-Key: $MEDERA_API_KEY" \ -d '{ "patientId": "pat_abc", "providers": ["prv_xyz"], "scheduledStart": "2026-05-28T15:00:00Z", "scheduledDurationMinutes": 50, "videoPlatform": "telehealth", "recordingEnabled": true, "deploymentId": "dep_abc" }' 

const session = await client.coTherapy.create({ patientId: 'pat_abc', providers: ['prv_xyz'], scheduledStart: '2026-05-28T15:00:00Z', scheduledDurationMinutes: 50, videoPlatform: 'telehealth', recordingEnabled: true, deploymentId: 'dep_abc', }); 

session = client.co_therapy.create(patient_id="pat_abc", providers=["prv_xyz"], scheduled_start="2026-05-28T15:00:00Z", scheduled_duration_minutes=50, video_platform="telehealth", recording_enabled=True, deployment_id="dep_abc",) 

Start the session

curl -X POST https://api.medera.info/api/co-therapy-sessions/{id}/start \ -H "X-API-Key: $MEDERA_API_KEY" 

Analyze the session

Upload audio + video for full multimodal analysis. The analyzer runs the engines, computes the neurobehavioral profile, and (optionally) retrieves clinical RAG evidence.

cURL curl -X POST https://api.medera.info/api/multimodal-therapy/analyze-session \ -H "X-API-Key: $MEDERA_API_KEY" \ -F "session_id={id}" \ -F "audio_file=@session.wav" \ -F "video_file=@session.mp4" \ -F "transcript=The patient reports..." \ -F "duration_seconds=2400" \ -F "patient_id=pat_abc" 

const form = new FormData; form.append('session_id', session.id); form.append('audio_file', audioBlob, 'session.wav'); form.append('video_file', videoBlob, 'session.mp4'); form.append('transcript', transcript); form.append('duration_seconds', '2400'); form.append('patient_id', 'pat_abc'); const res = await fetch('https://api.medera.info/api/multimodal-therapy/analyze-session', { method: 'POST', headers: { 'X-API-Key': process.env.MEDERA_API_KEY! }, body: form, }); 

with open('session.wav', 'rb') as a, open('session.mp4', 'rb') as v: res = client.multimodal.analyze_session(session_id=session.id, audio_file=a, video_file=v, transcript=transcript, duration_seconds=2400, patient_id="pat_abc",) 

Read the canonical payload

The response is a canonical payload with flat ClinicalMetric envelopes:

{ "facial": { "heart_rate": { "value": 78, "unit": "bpm", "confidence": 0.91 }, "hrv_sdnn": { "value": 42, "unit": "ms", "confidence": 0.83 }, "stress_index": { "value": 62, "unit": "0-100", "confidence": 0.78 } }, "vocal": { "f0_mean": { "value": 168, "unit": "Hz", "confidence": 0.92 }, "jitter": { "value": 0.85, "unit": "%", "confidence": 0.88 }, "prosodic_flatness": { "value": 0.48, "unit": "0-1", "confidence": 0.79 } }, "rdoc": { "constructs": [ { "name": "potential_threat_anxiety", "domain": "negative_valence", "score": 0.74, "confidence": "HIGH", "severity": "high", "interpretation": "Elevated anxiety markers...", "contributors": [ { "feature": "f0_mean", "value": 168, "contribution": 0.31 } ] } ] }, "clinical_context": { "summary": "...", "evidence_blocks": [...], "differentials": [...] } } 

The Co-Therapy Agent never addresses or interacts with the patient. Recording requires explicit per-session consent (consent_recorded: true).

What’s next

Co-Therapy Agent

Agent-level documentation.

Multimodal architecture

Pipeline deep dive.

Neurobehavioral Construct Constructs

15 named constructs.

Multimodal API

REST surface.

​Pipeline

​What’s next

Co-Therapy Agent

Multimodal architecture

Neurobehavioral Construct Constructs

Multimodal API

Pipeline

What’s next