feat: complete real-time voice-ai engine upgrades and frontend integration guide
This commit is contained in:
@@ -0,0 +1,256 @@
|
||||
# CureSelect AI — Real-Time Voice Intake & EHR Frontend Integration Guide
|
||||
|
||||
**Version**: `2.0.0 (Production Release)`
|
||||
**Target Audience**: Frontend Engineers (React, Next.js, Vue, React Native, Flutter, Vanilla TypeScript)
|
||||
**Backend Protocol**: Full-Duplex WebSockets (`ws://`) & HTTP/1.1 REST
|
||||
**Authentication Header**: `X-API-Key: skyheal_secure_api_key_2026` or `Authorization: Bearer <TOKEN>`
|
||||
**Default Dashboard**: `http://localhost:8000/` or `http://localhost:8000/dashboard`
|
||||
|
||||
---
|
||||
|
||||
## 1. Architectural Overview
|
||||
|
||||
The **CureSelect AI Voice Engine** is a high-speed, full-duplex conversational health intake system. It processes real-time microphone audio or text turns, classifies speaker perspectives, detects emergency distress, checks RxNorm drug safety, redacts HIPAA/ABDM PII, and generates real-time HL7 FHIR R4 Bundles and Doctor SOAP Notes.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client as Frontend (React / Web / Mobile)
|
||||
participant WS as WebSocket Manager (ws/auto)
|
||||
participant Engine as AI Orchestrator & Multi-Models
|
||||
participant EHR as EHR Export (FHIR / SOAP Note)
|
||||
|
||||
Client->>WS: Connect (ws://localhost:8000/v1/voice-onboarding/ws/auto)
|
||||
WS-->>Client: {"type": "connected", "session_id": "VOICE-AUTO-8F12A9C4"}
|
||||
WS-->>Client: {"type": "ai_response", "reply": "Hello Patient, welcome to CureSelect..."}
|
||||
|
||||
Client->>WS: Stream Binary Mic Bytes (MediaRecorder WebM/PCM)
|
||||
WS-->>Client: {"type": "backchannel", "reply": "Mm-hmm..."} (< 15ms)
|
||||
WS-->>Client: {"type": "speculative_audio_chunk", "ttfa_ms": 180} (< 200ms)
|
||||
WS-->>Client: {"type": "transcript", "speaker_type": "caregiver", "speaker_label": "Caregiver (Mother)"}
|
||||
WS-->>Client: {"type": "ai_response", "reply": "I'm sorry to hear about your mother's headache...", "audio_base64": "..."}
|
||||
|
||||
Client->>EHR: GET /v1/voice-onboarding/soap-note/{session_id}?format=html
|
||||
EHR-->>Client: Render Printable Doctor SOAP Note Report
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. WebSocket Connection Lifecycle (`ws://localhost:8000/v1/voice-onboarding/ws/auto`)
|
||||
|
||||
### 2.1 Connection Setup
|
||||
Connect to the zero-effort auto-session WebSocket endpoint:
|
||||
```typescript
|
||||
const ws = new WebSocket("ws://localhost:8000/v1/voice-onboarding/ws/auto");
|
||||
```
|
||||
|
||||
### 2.2 Inbound WebSocket Event Schema
|
||||
|
||||
#### 1. Connection Initialized (`connected`)
|
||||
Sent immediately upon socket handshake:
|
||||
```json
|
||||
{
|
||||
"type": "connected",
|
||||
"session_id": "VOICE-AUTO-9DEC79EC",
|
||||
"message": "Connected to CureSelect Real-Time Voice Engine"
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. Speaker Diarized Transcript (`transcript`)
|
||||
Sent as soon as patient or caregiver speech is recognized:
|
||||
```json
|
||||
{
|
||||
"type": "transcript",
|
||||
"role": "patient",
|
||||
"speaker_type": "caregiver",
|
||||
"speaker_label": "Caregiver (Mother)",
|
||||
"text": "My mother has severe headache for 10 days and she is taking Metformin 500mg."
|
||||
}
|
||||
```
|
||||
*Note*: `speaker_type` is either `"patient"` or `"caregiver"`.
|
||||
|
||||
#### 3. Low-Latency Acoustic Backchannel (`backchannel`)
|
||||
Emitted in **< 15ms** during patient pauses to signal active listening:
|
||||
```json
|
||||
{
|
||||
"type": "backchannel",
|
||||
"reply": "Mm-hmm...",
|
||||
"audio_base64": "..."
|
||||
}
|
||||
```
|
||||
|
||||
#### 4. Speculative Audio Chunk (`speculative_audio_chunk`)
|
||||
Emitted in **< 200ms** on the first 3-4 words generated by the LLM:
|
||||
```json
|
||||
{
|
||||
"type": "speculative_audio_chunk",
|
||||
"clause_text": "I am sorry to hear that.",
|
||||
"audio_base64": "...",
|
||||
"ttfa_ms": 180
|
||||
}
|
||||
```
|
||||
|
||||
#### 5. Complete AI Response & State Update (`ai_response`)
|
||||
Contains full reply, TTS audio, next stage, and updated clinical state:
|
||||
```json
|
||||
{
|
||||
"type": "ai_response",
|
||||
"reply": "Any known drug allergies?",
|
||||
"audio_id": "aud_771829",
|
||||
"audio_base64": "...",
|
||||
"next_stage": "symptom_collection",
|
||||
"completed": false,
|
||||
"state_updates": {
|
||||
"symptoms": [{"name": "headache", "duration": "10 days", "severity": "severe"}],
|
||||
"medications": [{"name": "Metformin", "dosage": "500mg daily"}]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 6. Emergency Nurse Escalation Alert (`nurse_escalation_alert`)
|
||||
Emitted when Pain Score >= 8/10 or Distress Score > 0.75:
|
||||
```json
|
||||
{
|
||||
"type": "nurse_escalation_alert",
|
||||
"session_id": "VOICE-AUTO-9DEC79EC",
|
||||
"distress_score": 0.85,
|
||||
"pain_score": 9,
|
||||
"message": "CRITICAL: Patient reported severe pain (9/10). Nurse escalation triggered!"
|
||||
}
|
||||
```
|
||||
|
||||
#### 7. RxNorm / FDA Drug Safety Warning (`drug_safety_warning`)
|
||||
Emitted when drug interactions or drug-allergy contraindications are flagged:
|
||||
```json
|
||||
{
|
||||
"type": "drug_safety_warning",
|
||||
"warnings": [
|
||||
{
|
||||
"severity": "CRITICAL",
|
||||
"category": "DRUG_INTERACTION",
|
||||
"description": "High risk of bleeding: Combined use of Warfarin and Aspirin."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Outbound Client Messages
|
||||
|
||||
### 3.1 Streaming Audio Bytes
|
||||
Stream raw WebM / PCM audio blobs from `MediaRecorder`:
|
||||
```javascript
|
||||
mediaRecorder.ondataavailable = (e) => {
|
||||
if (e.data.size > 0 && ws.readyState === WebSocket.OPEN) {
|
||||
ws.send(e.data); // Send raw Blob
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### 3.2 Text Turn Fallback
|
||||
```json
|
||||
{
|
||||
"type": "text_turn",
|
||||
"text": "10 days-ah severe headache iruku, Metformin 500mg sapudren."
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 Full-Duplex Barge-In Interruption
|
||||
Send when the user speaks or clicks "Interrupt AI" while AI audio is playing:
|
||||
```json
|
||||
{
|
||||
"type": "interrupt"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. REST API Integration Endpoints
|
||||
|
||||
All REST requests require the authentication header:
|
||||
`X-API-Key: skyheal_secure_api_key_2026`
|
||||
|
||||
### 4.1 HL7 FHIR R4 Bundle Export
|
||||
- **Endpoint**: `GET /v1/voice-onboarding/fhir/{session_id}`
|
||||
- **Response**: Production-grade HL7 FHIR R4 Bundle (`Patient`, `Encounter`, `Condition`, `MedicationStatement`, `AllergyIntolerance`).
|
||||
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/v1/voice-onboarding/fhir/VOICE-AUTO-9DEC79EC" \
|
||||
-H "X-API-Key: skyheal_secure_api_key_2026"
|
||||
```
|
||||
|
||||
### 4.2 Doctor SOAP Note & Printable PDF/HTML Report
|
||||
- **Endpoint**: `GET /v1/voice-onboarding/soap-note/{session_id}?format=json` (or `format=html`)
|
||||
- **Query Params**:
|
||||
- `format=json`: Returns structured Subjective, Objective, Assessment (ICD-10-CM & SNOMED-CT codes), and Plan JSON.
|
||||
- `format=html`: Renders a clean printable HTML document for 1-click doctor chart printing.
|
||||
|
||||
```bash
|
||||
# Get JSON SOAP Note with ICD-10 & SNOMED CT
|
||||
curl -X GET "http://localhost:8000/v1/voice-onboarding/soap-note/VOICE-AUTO-9DEC79EC?format=json" \
|
||||
-H "X-API-Key: skyheal_secure_api_key_2026"
|
||||
|
||||
# Get Printable HTML Report
|
||||
curl -X GET "http://localhost:8000/v1/voice-onboarding/soap-note/VOICE-AUTO-9DEC79EC?format=html" \
|
||||
-H "X-API-Key: skyheal_secure_api_key_2026"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. React / Next.js Custom Hook Example
|
||||
|
||||
```typescript
|
||||
import { useState, useEffect, useRef } from "react";
|
||||
|
||||
export function useVoiceIntakeWebSocket(url: string = "ws://localhost:8000/v1/voice-onboarding/ws/auto") {
|
||||
const [sessionId, setSessionId] = useState<string>("");
|
||||
const [messages, setMessages] = useState<Array<any>>([]);
|
||||
const [clinicalState, setClinicalState] = useState<any>({});
|
||||
const [escalationAlert, setEscalationAlert] = useState<string | null>(null);
|
||||
const [drugWarnings, setDrugWarnings] = useState<Array<any>>([]);
|
||||
const wsRef = useRef<WebSocket | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
const ws = new WebSocket(url);
|
||||
wsRef.current = ws;
|
||||
|
||||
ws.onmessage = (event) => {
|
||||
const data = JSON.parse(event.data);
|
||||
if (data.type === "connected") {
|
||||
setSessionId(data.session_id);
|
||||
} else if (data.type === "transcript") {
|
||||
setMessages(prev => [...prev, { role: data.role, text: data.text, label: data.speaker_label }]);
|
||||
} else if (data.type === "ai_response") {
|
||||
setMessages(prev => [...prev, { role: "assistant", text: data.reply, label: "AI Assistant" }]);
|
||||
if (data.audio_base64) {
|
||||
const audio = new Audio("data:audio/wav;base64," + data.audio_base64);
|
||||
audio.play().catch(console.error);
|
||||
}
|
||||
if (data.state_updates) {
|
||||
setClinicalState(data.state_updates);
|
||||
}
|
||||
} else if (data.type === "nurse_escalation_alert") {
|
||||
setEscalationAlert(data.message);
|
||||
} else if (data.type === "drug_safety_warning") {
|
||||
setDrugWarnings(data.warnings);
|
||||
}
|
||||
};
|
||||
|
||||
return () => ws.close();
|
||||
}, [url]);
|
||||
|
||||
const sendTextTurn = (text: string) => {
|
||||
if (wsRef.current?.readyState === WebSocket.OPEN) {
|
||||
wsRef.current.send(JSON.stringify({ type: "text_turn", text }));
|
||||
}
|
||||
};
|
||||
|
||||
const interruptAI = () => {
|
||||
if (wsRef.current?.readyState === WebSocket.OPEN) {
|
||||
wsRef.current.send(JSON.stringify({ type: "interrupt" }));
|
||||
}
|
||||
};
|
||||
|
||||
return { sessionId, messages, clinicalState, escalationAlert, drugWarnings, sendTextTurn, interruptAI };
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,870 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>CureSelect AI - Clinical Voice Intake & Real-Time EHR Dashboard</title>
|
||||
<!-- Google Fonts & FontAwesome Icons -->
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700;800&family=JetBrains+Mono:wght@400;500;600&display=swap" rel="stylesheet">
|
||||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css">
|
||||
|
||||
<style>
|
||||
:root {
|
||||
--bg-dark: #090d16;
|
||||
--bg-card: rgba(17, 24, 39, 0.75);
|
||||
--bg-card-hover: rgba(31, 41, 55, 0.85);
|
||||
--border-color: rgba(255, 255, 255, 0.08);
|
||||
--accent-blue: #3b82f6;
|
||||
--accent-cyan: #06b6d4;
|
||||
--accent-green: #10b981;
|
||||
--accent-red: #ef4444;
|
||||
--accent-amber: #f59e0b;
|
||||
--accent-purple: #8b5cf6;
|
||||
--text-main: #f3f4f6;
|
||||
--text-muted: #9ca3af;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
font-family: 'Inter', sans-serif;
|
||||
}
|
||||
|
||||
body {
|
||||
background-color: var(--bg-dark);
|
||||
background-image:
|
||||
radial-gradient(at 0% 0%, rgba(59, 130, 246, 0.12) 0px, transparent 50%),
|
||||
radial-gradient(at 100% 100%, rgba(139, 92, 246, 0.12) 0px, transparent 50%);
|
||||
color: var(--text-main);
|
||||
min-height: 100vh;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
/* Top Header Navigation */
|
||||
header {
|
||||
background: rgba(15, 23, 42, 0.8);
|
||||
backdrop-filter: blur(16px);
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
padding: 16px 32px;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 100;
|
||||
}
|
||||
|
||||
.logo-container {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.logo-icon {
|
||||
width: 40px;
|
||||
height: 40px;
|
||||
background: linear-gradient(135deg, var(--accent-blue), var(--accent-cyan));
|
||||
border-radius: 10px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
font-size: 20px;
|
||||
color: white;
|
||||
box-shadow: 0 0 20px rgba(6, 182, 212, 0.4);
|
||||
}
|
||||
|
||||
.logo-text {
|
||||
font-size: 20px;
|
||||
font-weight: 700;
|
||||
background: linear-gradient(90deg, #ffffff, #93c5fd);
|
||||
-webkit-background-clip: text;
|
||||
-webkit-text-fill-color: transparent;
|
||||
}
|
||||
|
||||
.status-badge {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
background: rgba(16, 185, 129, 0.1);
|
||||
border: 1px solid rgba(16, 185, 129, 0.3);
|
||||
color: var(--accent-green);
|
||||
padding: 6px 14px;
|
||||
border-radius: 20px;
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.pulse-dot {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
background-color: var(--accent-green);
|
||||
border-radius: 50%;
|
||||
box-shadow: 0 0 10px var(--accent-green);
|
||||
animation: pulse 1.8s infinite;
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
0% { transform: scale(0.95); opacity: 0.8; }
|
||||
50% { transform: scale(1.3); opacity: 1; }
|
||||
100% { transform: scale(0.95); opacity: 0.8; }
|
||||
}
|
||||
|
||||
/* Critical Escalation Banner */
|
||||
.nurse-banner {
|
||||
display: none;
|
||||
background: linear-gradient(90deg, #7f1d1d, #b91c1c);
|
||||
border-bottom: 2px solid #f87171;
|
||||
color: white;
|
||||
padding: 14px 32px;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
font-weight: 600;
|
||||
animation: bannerFlash 1s infinite alternate;
|
||||
}
|
||||
|
||||
@keyframes bannerFlash {
|
||||
from { background: linear-gradient(90deg, #7f1d1d, #b91c1c); }
|
||||
to { background: linear-gradient(90deg, #991b1b, #dc2626); }
|
||||
}
|
||||
|
||||
.nurse-banner.active {
|
||||
display: flex;
|
||||
}
|
||||
|
||||
/* Main Container Grid */
|
||||
.main-container {
|
||||
flex: 1;
|
||||
padding: 24px 32px;
|
||||
display: grid;
|
||||
grid-template-columns: 1.2fr 0.8fr;
|
||||
gap: 24px;
|
||||
max-width: 1600px;
|
||||
margin: 0 auto;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
/* Card Styles */
|
||||
.glass-card {
|
||||
background: var(--bg-card);
|
||||
backdrop-filter: blur(16px);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 16px;
|
||||
padding: 24px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
box-shadow: 0 10px 30px rgba(0,0,0,0.3);
|
||||
}
|
||||
|
||||
.card-title {
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
color: #e5e7eb;
|
||||
margin-bottom: 16px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
/* Left Column: Voice Interaction & Live Feed */
|
||||
.voice-section {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
.mic-control-box {
|
||||
background: rgba(15, 23, 42, 0.6);
|
||||
border: 1px solid rgba(255, 255, 255, 0.05);
|
||||
border-radius: 16px;
|
||||
padding: 24px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
text-align: center;
|
||||
position: relative;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.waveform-canvas {
|
||||
width: 100%;
|
||||
height: 80px;
|
||||
margin: 10px 0;
|
||||
}
|
||||
|
||||
.mic-button {
|
||||
width: 80px;
|
||||
height: 80px;
|
||||
border-radius: 50%;
|
||||
background: linear-gradient(135deg, var(--accent-blue), var(--accent-cyan));
|
||||
border: none;
|
||||
color: white;
|
||||
font-size: 30px;
|
||||
cursor: pointer;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
box-shadow: 0 0 30px rgba(6, 182, 212, 0.4);
|
||||
transition: all 0.3s ease;
|
||||
outline: none;
|
||||
}
|
||||
|
||||
.mic-button.recording {
|
||||
background: linear-gradient(135deg, #ef4444, #dc2626);
|
||||
box-shadow: 0 0 40px rgba(239, 68, 68, 0.6);
|
||||
animation: pulseRecord 1.5s infinite;
|
||||
}
|
||||
|
||||
@keyframes pulseRecord {
|
||||
0% { transform: scale(1); }
|
||||
50% { transform: scale(1.08); }
|
||||
100% { transform: scale(1); }
|
||||
}
|
||||
|
||||
.action-buttons {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
margin-top: 16px;
|
||||
}
|
||||
|
||||
.btn {
|
||||
background: rgba(255, 255, 255, 0.08);
|
||||
border: 1px solid var(--border-color);
|
||||
color: var(--text-main);
|
||||
padding: 8px 16px;
|
||||
border-radius: 8px;
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
cursor: pointer;
|
||||
transition: all 0.2s ease;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.btn:hover {
|
||||
background: rgba(255, 255, 255, 0.15);
|
||||
}
|
||||
|
||||
.btn-danger {
|
||||
background: rgba(239, 68, 68, 0.2);
|
||||
border-color: rgba(239, 68, 68, 0.4);
|
||||
color: #fca5a5;
|
||||
}
|
||||
|
||||
.btn-danger:hover {
|
||||
background: rgba(239, 68, 68, 0.35);
|
||||
}
|
||||
|
||||
/* Conversation Transcript Area */
|
||||
.chat-area {
|
||||
flex: 1;
|
||||
min-height: 380px;
|
||||
max-height: 480px;
|
||||
overflow-y: auto;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 14px;
|
||||
padding-right: 8px;
|
||||
}
|
||||
|
||||
.chat-bubble {
|
||||
max-width: 85%;
|
||||
padding: 14px 18px;
|
||||
border-radius: 14px;
|
||||
font-size: 14px;
|
||||
line-height: 1.5;
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.chat-bubble.ai {
|
||||
background: rgba(30, 41, 59, 0.8);
|
||||
border: 1px solid rgba(59, 130, 246, 0.2);
|
||||
align-self: flex-start;
|
||||
border-bottom-left-radius: 2px;
|
||||
}
|
||||
|
||||
.chat-bubble.patient {
|
||||
background: rgba(14, 116, 144, 0.3);
|
||||
border: 1px solid rgba(6, 182, 212, 0.3);
|
||||
align-self: flex-end;
|
||||
border-bottom-right-radius: 2px;
|
||||
}
|
||||
|
||||
.chat-bubble.caregiver {
|
||||
background: rgba(126, 34, 206, 0.25);
|
||||
border: 1px solid rgba(168, 85, 247, 0.3);
|
||||
align-self: flex-end;
|
||||
border-bottom-right-radius: 2px;
|
||||
}
|
||||
|
||||
.bubble-meta {
|
||||
font-size: 11px;
|
||||
color: var(--text-muted);
|
||||
margin-bottom: 4px;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.speaker-tag {
|
||||
padding: 2px 8px;
|
||||
border-radius: 10px;
|
||||
font-size: 10px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.tag-patient { background: rgba(6, 182, 212, 0.2); color: var(--accent-cyan); }
|
||||
.tag-caregiver { background: rgba(168, 85, 247, 0.2); color: #c084fc; }
|
||||
.tag-ai { background: rgba(59, 130, 246, 0.2); color: #93c5fd; }
|
||||
|
||||
/* Right Column: Real-Time Clinical Intake State & EHR Controls */
|
||||
.clinical-section {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
.metrics-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, 1fr);
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.metric-card {
|
||||
background: rgba(15, 23, 42, 0.6);
|
||||
border: 1px solid var(--border-color);
|
||||
padding: 14px;
|
||||
border-radius: 10px;
|
||||
}
|
||||
|
||||
.metric-label {
|
||||
font-size: 12px;
|
||||
color: var(--text-muted);
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
.metric-value {
|
||||
font-size: 18px;
|
||||
font-weight: 700;
|
||||
color: #f3f4f6;
|
||||
}
|
||||
|
||||
/* Drug Warning Alert Card */
|
||||
.warning-card {
|
||||
display: none;
|
||||
background: rgba(245, 158, 11, 0.12);
|
||||
border: 1px solid rgba(245, 158, 11, 0.3);
|
||||
border-radius: 12px;
|
||||
padding: 14px;
|
||||
margin-top: 10px;
|
||||
}
|
||||
|
||||
.warning-card.active {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.warning-header {
|
||||
color: var(--accent-amber);
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-bottom: 6px;
|
||||
}
|
||||
|
||||
/* State JSON Tree View */
|
||||
.state-box {
|
||||
background: #050811;
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 10px;
|
||||
padding: 14px;
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 12px;
|
||||
color: #38bdf8;
|
||||
max-height: 250px;
|
||||
overflow-y: auto;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
/* Modal Overlay */
|
||||
.modal-overlay {
|
||||
display: none;
|
||||
position.fixed;
|
||||
top: 0;
|
||||
left: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
background: rgba(0, 0, 0, 0.8);
|
||||
backdrop-filter: blur(8px);
|
||||
z-index: 200;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.modal-overlay.active {
|
||||
display: flex;
|
||||
}
|
||||
|
||||
.modal-card {
|
||||
background: #0f172a;
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 16px;
|
||||
width: 90%;
|
||||
max-width: 850px;
|
||||
max-height: 85vh;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
box-shadow: 0 20px 50px rgba(0,0,0,0.5);
|
||||
}
|
||||
|
||||
.modal-header {
|
||||
padding: 18px 24px;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.modal-body {
|
||||
padding: 24px;
|
||||
overflow-y: auto;
|
||||
flex: 1;
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 13px;
|
||||
background: #090d16;
|
||||
color: #a7f3d0;
|
||||
}
|
||||
|
||||
.modal-footer {
|
||||
padding: 16px 24px;
|
||||
border-top: 1px solid var(--border-color);
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
gap: 12px;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<!-- Header Navigation -->
|
||||
<header>
|
||||
<div class="logo-container">
|
||||
<div class="logo-icon"><i class="fa-solid fa-heart-pulse"></i></div>
|
||||
<div class="logo-text">CureSelect AI Voice Intake</div>
|
||||
</div>
|
||||
<div style="display: flex; gap: 16px; align-items: center;">
|
||||
<div class="status-badge">
|
||||
<div class="pulse-dot"></div>
|
||||
<span id="wsStatus">Connected (ws://localhost:8000/v1/voice-onboarding/ws/auto)</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<!-- Critical Nurse Escalation Banner -->
|
||||
<div id="nurseBanner" class="nurse-banner">
|
||||
<div style="display: flex; align-items: center; gap: 12px;">
|
||||
<i class="fa-solid fa-triangle-exclamation fa-lg"></i>
|
||||
<div>
|
||||
<span style="font-size: 15px;">CRITICAL NURSE ESCALATION ALERT:</span>
|
||||
<span id="bannerText" style="font-weight: 400; margin-left: 8px;">High Pain Score (9/10) reported. Immediate clinical review required!</span>
|
||||
</div>
|
||||
</div>
|
||||
<button class="btn btn-danger" onclick="dismissBanner()">Acknowledge</button>
|
||||
</div>
|
||||
|
||||
<!-- Main Workspace Container -->
|
||||
<div class="main-container">
|
||||
|
||||
<!-- Left Column: Voice Interaction & Live Feed -->
|
||||
<div class="voice-section">
|
||||
|
||||
<div class="glass-card mic-control-box">
|
||||
<div class="card-title" style="margin-bottom: 8px;">
|
||||
<i class="fa-solid fa-microphone-lines" style="color: var(--accent-cyan);"></i>
|
||||
Real-Time Audio Streamer
|
||||
</div>
|
||||
|
||||
<canvas id="waveform" class="waveform-canvas"></canvas>
|
||||
|
||||
<button id="micBtn" class="mic-button" onclick="toggleMic()">
|
||||
<i class="fa-solid fa-microphone"></i>
|
||||
</button>
|
||||
|
||||
<div id="micStatusText" style="margin-top: 12px; font-size: 13px; color: var(--text-muted);">
|
||||
Click microphone to start full-duplex voice intake
|
||||
</div>
|
||||
|
||||
<div class="action-buttons">
|
||||
<button class="btn btn-danger" onclick="triggerInterrupt()">
|
||||
<i class="fa-solid fa-hand"></i> Interrupt AI (Barge-In)
|
||||
</button>
|
||||
<button class="btn" onclick="sendSampleTurn('Tanglish')">
|
||||
<i class="fa-solid fa-language"></i> Tanglish Turn
|
||||
</button>
|
||||
<button class="btn" onclick="sendSampleTurn('Caregiver')">
|
||||
<i class="fa-solid fa-people-roof"></i> Caregiver Turn
|
||||
</button>
|
||||
<button class="btn" onclick="sendSampleTurn('HighPain')">
|
||||
<i class="fa-solid fa-heart-crack"></i> High Pain (9/10)
|
||||
</button>
|
||||
<button class="btn" onclick="sendSampleTurn('DrugInteraction')">
|
||||
<i class="fa-solid fa-pills"></i> Drug Warning
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Conversation Transcript Area -->
|
||||
<div class="glass-card" style="flex: 1;">
|
||||
<div class="card-title">
|
||||
<i class="fa-solid fa-comments" style="color: var(--accent-purple);"></i>
|
||||
Live Conversation Stream
|
||||
</div>
|
||||
<div id="chatArea" class="chat-area">
|
||||
<div class="chat-bubble ai">
|
||||
<div class="bubble-meta">
|
||||
<span class="speaker-tag tag-ai">AI Assistant</span>
|
||||
<span>Greeting</span>
|
||||
</div>
|
||||
Hello Patient, welcome to CureSelect Healthcare. Could you please share what health symptoms you are experiencing today?
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- Right Column: Real-Time Clinical Intake State & EHR Controls -->
|
||||
<div class="clinical-section">
|
||||
|
||||
<div class="glass-card">
|
||||
<div class="card-title">
|
||||
<i class="fa-solid fa-notes-medical" style="color: var(--accent-green);"></i>
|
||||
Live Clinical Metrics
|
||||
</div>
|
||||
|
||||
<div class="metrics-grid">
|
||||
<div class="metric-card">
|
||||
<div class="metric-label">Pain Score</div>
|
||||
<div id="metricPain" class="metric-value">3 / 10</div>
|
||||
</div>
|
||||
<div class="metric-card">
|
||||
<div class="metric-label">Distress Score</div>
|
||||
<div id="metricDistress" class="metric-value">0.20</div>
|
||||
</div>
|
||||
<div class="metric-card">
|
||||
<div class="metric-label">TTFA Latency</div>
|
||||
<div id="metricTtfa" class="metric-value">⚡ 180 ms</div>
|
||||
</div>
|
||||
<div class="metric-card">
|
||||
<div class="metric-label">Speech Pace</div>
|
||||
<div id="metricPace" class="metric-value">1.0x (Standard)</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- RxNorm / FDA Drug Safety Warning Card -->
|
||||
<div id="drugWarningCard" class="warning-card">
|
||||
<div class="warning-header">
|
||||
<i class="fa-solid fa-shield-virus"></i> RxNorm / FDA Drug Safety Warning
|
||||
</div>
|
||||
<div id="warningCardText" style="font-size: 12px; color: #fef08a;">
|
||||
No drug interaction warnings detected.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- State JSON Tree View -->
|
||||
<div class="glass-card" style="flex: 1;">
|
||||
<div class="card-title" style="justify-content: space-between;">
|
||||
<div>
|
||||
<i class="fa-solid fa-code" style="color: var(--accent-blue);"></i>
|
||||
Extracted Clinical State
|
||||
</div>
|
||||
<div style="display: flex; gap: 8px;">
|
||||
<button class="btn" onclick="openFhirModal()">
|
||||
<i class="fa-solid fa-fire"></i> FHIR R4
|
||||
</button>
|
||||
<button class="btn" onclick="openSoapModal()">
|
||||
<i class="fa-solid fa-file-medical"></i> SOAP Note
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="jsonState" class="state-box">
|
||||
{
|
||||
"reason_for_visit": "general_intake",
|
||||
"symptoms": [],
|
||||
"medical_history": [],
|
||||
"medications": [],
|
||||
"allergies": []
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- Modal Dialog for FHIR & SOAP Note Viewers -->
|
||||
<div id="modalOverlay" class="modal-overlay">
|
||||
<div class="modal-card">
|
||||
<div class="modal-header">
|
||||
<div id="modalTitle" style="font-size: 16px; font-weight: 600; color: white;">HL7 FHIR R4 Bundle JSON</div>
|
||||
<button class="btn" onclick="closeModal()"><i class="fa-solid fa-xmark"></i> Close</button>
|
||||
</div>
|
||||
<div id="modalBody" class="modal-body">
|
||||
Loading...
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button class="btn" onclick="copyModalContent()"><i class="fa-solid fa-copy"></i> Copy JSON</button>
|
||||
<button id="modalHtmlBtn" class="btn" style="display: none;" onclick="openPrintableHtml()"><i class="fa-solid fa-print"></i> Open Printable Report</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- WebSockets & Web Audio Script -->
|
||||
<script>
|
||||
let ws;
|
||||
let isRecording = false;
|
||||
let audioContext;
|
||||
let mediaStream;
|
||||
let currentSessionId = "";
|
||||
|
||||
// Establish WebSocket Connection
|
||||
function connectWebSocket() {
|
||||
ws = new WebSocket("ws://localhost:8000/v1/voice-onboarding/ws/auto");
|
||||
|
||||
ws.onopen = () => {
|
||||
document.getElementById("wsStatus").innerText = "Connected (ws://localhost:8000/v1/voice-onboarding/ws/auto)";
|
||||
};
|
||||
|
||||
ws.onmessage = (event) => {
|
||||
const data = JSON.parse(event.data);
|
||||
handleWSMessage(data);
|
||||
};
|
||||
|
||||
ws.onclose = () => {
|
||||
document.getElementById("wsStatus").innerText = "Disconnected. Retrying...";
|
||||
setTimeout(connectWebSocket, 2000);
|
||||
};
|
||||
}
|
||||
|
||||
// Handle WebSocket Message Events
|
||||
function handleWSMessage(data) {
|
||||
console.log("WS Event Received:", data);
|
||||
|
||||
if (data.type === "connected") {
|
||||
currentSessionId = data.session_id;
|
||||
} else if (data.type === "transcript") {
|
||||
addChatBubble(data.role, data.text, data.speaker_label, data.speaker_type);
|
||||
} else if (data.type === "backchannel") {
|
||||
console.log("⚡ Instant Backchannel:", data.reply);
|
||||
} else if (data.type === "speculative_audio_chunk") {
|
||||
document.getElementById("metricTtfa").innerText = `⚡ ${data.ttfa_ms} ms`;
|
||||
} else if (data.type === "ai_response") {
|
||||
addChatBubble("assistant", data.reply, "AI Assistant", "ai");
|
||||
if (data.audio_base64) {
|
||||
playAudioBase64(data.audio_base64);
|
||||
}
|
||||
if (data.state_updates) {
|
||||
updateClinicalState(data.state_updates);
|
||||
}
|
||||
} else if (data.type === "cancel_playback") {
|
||||
console.log("🛑 Playback Interrupted by User!");
|
||||
} else if (data.type === "nurse_escalation_alert") {
|
||||
showNurseBanner(data.message);
|
||||
document.getElementById("metricPain").innerText = `${data.pain_score} / 10`;
|
||||
document.getElementById("metricDistress").innerText = `${data.distress_score}`;
|
||||
} else if (data.type === "drug_safety_warning") {
|
||||
showDrugWarnings(data.warnings);
|
||||
}
|
||||
}
|
||||
|
||||
// Play Audio Chunk
|
||||
function playAudioBase64(b64) {
|
||||
try {
|
||||
const audio = new Audio("data:audio/wav;base64," + b64);
|
||||
audio.play().catch(e => console.error("Audio playback error:", e));
|
||||
} catch (e) {
|
||||
console.error("Audio error:", e);
|
||||
}
|
||||
}
|
||||
|
||||
// Render Chat Bubbles
|
||||
function addChatBubble(role, text, label, speakerType) {
|
||||
const chatArea = document.getElementById("chatArea");
|
||||
const bubble = document.createElement("div");
|
||||
|
||||
let bubbleClass = "chat-bubble ";
|
||||
let tagClass = "speaker-tag ";
|
||||
|
||||
if (role === "assistant" || speakerType === "ai") {
|
||||
bubbleClass += "ai";
|
||||
tagClass += "tag-ai";
|
||||
} else if (speakerType === "caregiver") {
|
||||
bubbleClass += "caregiver";
|
||||
tagClass += "tag-caregiver";
|
||||
} else {
|
||||
bubbleClass += "patient";
|
||||
tagClass += "tag-patient";
|
||||
}
|
||||
|
||||
bubble.className = bubbleClass;
|
||||
bubble.innerHTML = `
|
||||
<div class="bubble-meta">
|
||||
<span class="${tagClass}">${label || (role === 'assistant' ? 'AI Assistant' : 'Patient')}</span>
|
||||
<span>${new Date().toLocaleTimeString([], {hour: '2-digit', minute:'2-digit'})}</span>
|
||||
</div>
|
||||
${text}
|
||||
`;
|
||||
chatArea.appendChild(bubble);
|
||||
chatArea.scrollTop = chatArea.scrollHeight;
|
||||
}
|
||||
|
||||
// Update Clinical State Tree
|
||||
function updateClinicalState(state) {
|
||||
document.getElementById("jsonState").innerText = JSON.stringify(state, null, 2);
|
||||
}
|
||||
|
||||
// Mic Toggle
|
||||
async function toggleMic() {
|
||||
const micBtn = document.getElementById("micBtn");
|
||||
const statusText = document.getElementById("micStatusText");
|
||||
|
||||
if (!isRecording) {
|
||||
try {
|
||||
mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true });
|
||||
isRecording = true;
|
||||
micBtn.classList.add("recording");
|
||||
statusText.innerText = "Streaming microphone audio... Speak freely.";
|
||||
|
||||
const mediaRecorder = new MediaRecorder(mediaStream);
|
||||
mediaRecorder.ondataavailable = (e) => {
|
||||
if (e.data.size > 0 && ws.readyState === WebSocket.OPEN) {
|
||||
ws.send(e.data);
|
||||
}
|
||||
};
|
||||
mediaRecorder.start(250);
|
||||
window.currentMediaRecorder = mediaRecorder;
|
||||
} catch (e) {
|
||||
alert("Microphone permission denied or not available: " + e.message);
|
||||
}
|
||||
} else {
|
||||
isRecording = false;
|
||||
micBtn.classList.remove("recording");
|
||||
statusText.innerText = "Microphone paused. Click to resume.";
|
||||
if (window.currentMediaRecorder) window.currentMediaRecorder.stop();
|
||||
if (mediaStream) mediaStream.getTracks().forEach(t => t.stop());
|
||||
ws.send(JSON.stringify({ type: "finish_turn" }));
|
||||
}
|
||||
}
|
||||
|
||||
function triggerInterrupt() {
|
||||
ws.send(JSON.stringify({ type: "interrupt" }));
|
||||
}
|
||||
|
||||
function sendSampleTurn(type) {
|
||||
if (type === "Tanglish") {
|
||||
ws.send(JSON.stringify({
|
||||
type: "text_turn",
|
||||
text: "10 days-ah severe headache iruku, Metformin 500mg daily sapudren."
|
||||
}));
|
||||
} else if (type === "Caregiver") {
|
||||
ws.send(JSON.stringify({
|
||||
type: "text_turn",
|
||||
text: "My mother has severe headache for 10 days and she is taking Metformin 500mg."
|
||||
}));
|
||||
} else if (type === "HighPain") {
|
||||
ws.send(JSON.stringify({
|
||||
type: "text_turn",
|
||||
text: "My chest pain is 9 out of 10 unbearable, I cannot breathe properly!"
|
||||
}));
|
||||
} else if (type === "DrugInteraction") {
|
||||
ws.send(JSON.stringify({
|
||||
type: "text_turn",
|
||||
text: "I am taking Warfarin and Aspirin daily, and I have severe Penicillin allergy."
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
function showNurseBanner(msg) {
|
||||
document.getElementById("bannerText").innerText = msg;
|
||||
document.getElementById("nurseBanner").classList.add("active");
|
||||
}
|
||||
|
||||
function dismissBanner() {
|
||||
document.getElementById("nurseBanner").classList.remove("active");
|
||||
}
|
||||
|
||||
function showDrugWarnings(warnings) {
|
||||
const card = document.getElementById("drugWarningCard");
|
||||
const text = document.getElementById("warningCardText");
|
||||
if (warnings && warnings.length > 0) {
|
||||
text.innerText = warnings.map(w => `[${w.severity}] ${w.category}: ${w.description}`).join("\n\n");
|
||||
card.classList.add("active");
|
||||
}
|
||||
}
|
||||
|
||||
// Modals for FHIR & SOAP Notes
|
||||
async function openFhirModal() {
|
||||
if (!currentSessionId) return alert("No active session ID");
|
||||
document.getElementById("modalTitle").innerText = "HL7 FHIR R4 Bundle JSON";
|
||||
document.getElementById("modalBody").innerText = "Loading FHIR R4 Bundle...";
|
||||
document.getElementById("modalHtmlBtn").style.display = "none";
|
||||
document.getElementById("modalOverlay").classList.add("active");
|
||||
|
||||
try {
|
||||
const res = await fetch(`http://localhost:8000/v1/voice-onboarding/fhir/${currentSessionId}`, {
|
||||
headers: { "X-API-Key": "skyheal_secure_api_key_2026" }
|
||||
});
|
||||
const bundle = await res.json();
|
||||
document.getElementById("modalBody").innerText = JSON.stringify(bundle, null, 2);
|
||||
} catch (e) {
|
||||
document.getElementById("modalBody").innerText = "Error fetching FHIR Bundle: " + e.message;
|
||||
}
|
||||
}
|
||||
|
||||
async function openSoapModal() {
|
||||
if (!currentSessionId) return alert("No active session ID");
|
||||
document.getElementById("modalTitle").innerText = "Clinical SOAP Note (Subjective, Objective, Assessment, Plan)";
|
||||
document.getElementById("modalBody").innerText = "Compiling SOAP Note & ICD-10 Coding...";
|
||||
document.getElementById("modalHtmlBtn").style.display = "inline-flex";
|
||||
document.getElementById("modalOverlay").classList.add("active");
|
||||
|
||||
try {
|
||||
const res = await fetch(`http://localhost:8000/v1/voice-onboarding/soap-note/${currentSessionId}`, {
|
||||
headers: { "X-API-Key": "skyheal_secure_api_key_2026" }
|
||||
});
|
||||
const note = await res.json();
|
||||
document.getElementById("modalBody").innerText = JSON.stringify(note, null, 2);
|
||||
} catch (e) {
|
||||
document.getElementById("modalBody").innerText = "Error compiling SOAP Note: " + e.message;
|
||||
}
|
||||
}
|
||||
|
||||
function closeModal() {
|
||||
document.getElementById("modalOverlay").classList.remove("active");
|
||||
}
|
||||
|
||||
function copyModalContent() {
|
||||
const text = document.getElementById("modalBody").innerText;
|
||||
navigator.clipboard.writeText(text);
|
||||
alert("Copied to clipboard!");
|
||||
}
|
||||
|
||||
function openPrintableHtml() {
|
||||
if (currentSessionId) {
|
||||
window.open(`http://localhost:8000/v1/voice-onboarding/soap-note/${currentSessionId}?format=html`, '_blank');
|
||||
}
|
||||
}
|
||||
|
||||
// Initialize on load
|
||||
window.onload = connectWebSocket;
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user