docs: add API documentation for frontend integration and include response timing metrics
This commit is contained in:
@@ -1,532 +1,349 @@
|
|||||||
# AI Patient Onboarding — Backend API Documentation
|
# AI Patient Onboarding — Frontend Integration API Reference Guide
|
||||||
|
|
||||||
This document provides a comprehensive guide for integrating with the **AI Patient Onboarding FastAPI Backend**. It covers all REST API endpoints, request/response models, voice integration flows, medical document processing, and EHR integration.
|
Base URL: `http://127.0.0.1:8000`
|
||||||
|
Protocol: `HTTP / REST` + `JSON` + `multipart/form-data`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Table of Contents
|
## Overview
|
||||||
1. [Architecture & Server Overview](#1-architecture--server-overview)
|
|
||||||
2. [Environment Configuration (`.env`)](#2-environment-configuration-env)
|
The AI Patient Onboarding system provides two modes of integration for frontend web/mobile applications:
|
||||||
3. [Running the Backend Locally](#3-running-the-backend-locally)
|
|
||||||
4. [Voice Onboarding API (`/v1/voice-onboarding`)](#4-voice-onboarding-api-v1voice-onboarding)
|
1. **Voice AI Onboarding API** (`/v1/voice-onboarding/...`): Supports voice microphone recording, ASR (Speech-to-Text), synthesized audio response streaming (TTS), Google Gemini AI triage, response timing, and EHR JSON export.
|
||||||
- [4.1 Start Voice Session](#41-start-voice-session)
|
2. **Text AI Onboarding API** (`/v1/onboarding/...`): Supports pure text chat-based intake, Google Gemini AI reasoning, state extraction, and EHR JSON export.
|
||||||
- [4.2 Send Voice / Speech Turn](#42-send-voice--speech-turn)
|
|
||||||
- [4.3 Fetch Synthesized Voice Audio](#43-fetch-synthesized-voice-audio)
|
|
||||||
- [4.4 Complete & Confirm Session](#44-complete--confirm-session)
|
|
||||||
5. [Text Onboarding API (`/v1/onboarding`)](#5-text-onboarding-api-v1onboarding)
|
|
||||||
- [5.1 Start Text Session](#51-start-text-session)
|
|
||||||
- [5.2 Send Text Message](#52-send-text-message)
|
|
||||||
- [5.3 Complete Text Onboarding](#53-complete-text-onboarding)
|
|
||||||
6. [Medical Document & Vision API (`/v1/documents`)](#6-medical-document--vision-api-v1documents)
|
|
||||||
7. [System Health Check (`/health`)](#7-system-health-check-health)
|
|
||||||
8. [Connecting External Hospital EMR / EHR System](#8-connecting-external-hospital-emr--ehr-system)
|
|
||||||
9. [Multi-Language & Clinical Specialties](#9-multi-language--clinical-specialties)
|
|
||||||
10. [Client Integration Code Samples](#10-client-integration-code-samples)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Architecture & Server Overview
|
# SECTION 1: Voice AI Onboarding API (`/v1/voice-onboarding`)
|
||||||
|
|
||||||
- **Framework:** FastAPI (Python 3.10+) with SQLAlchemy ORM
|
### 1.1 Start Voice Onboarding Session
|
||||||
- **Default Base URL:** `http://127.0.0.1:8000`
|
Initializes a voice onboarding session and returns localized greeting text + synthesized spoken audio URL.
|
||||||
- **Interactive Swagger Documentation:** `http://127.0.0.1:8000/docs`
|
|
||||||
- **OpenAPI JSON Spec:** `http://127.0.0.1:8000/openapi.json`
|
|
||||||
- **CORS:** Pre-configured to allow all origins (`*`) for easy React web and mobile client integration.
|
|
||||||
|
|
||||||
```
|
- **Endpoint**: `POST /v1/voice-onboarding/start`
|
||||||
+------------------+ +-------------------------------+ +-----------------------+
|
- **Content-Type**: `application/json`
|
||||||
| Client Device | | FastAPI Backend Engine | | Hospital EHR / Database|
|
|
||||||
+------------------+ +-------------------------------+ +-----------------------+
|
|
||||||
| | |
|
|
||||||
| --- POST /start ----------------> | (Initialize Session + State) |
|
|
||||||
| <-- Greeting Text & Audio URL --- | (Synthesized TTS Audio) |
|
|
||||||
| | |
|
|
||||||
| --- POST /turn (Audio / Text) --> | (Sarvam STT -> GPT-4o -> State) |
|
|
||||||
| <-- Next Question & Audio URL --- | |
|
|
||||||
| | |
|
|
||||||
| --- POST /upload (Prescription) > | (Gemini Vision OCR -> Extract Meds) |
|
|
||||||
| <-- Extracted Findings ---------- | |
|
|
||||||
| | |
|
|
||||||
| --- POST /complete (confirmed) -> | (Extract EHR JSON) ---------------->| POST /patients
|
|
||||||
| <-- Final Success Status -------- | <--- 200 OK Record Saved -----------|
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
#### Request Payload
|
||||||
|
|
||||||
## 2. Environment Configuration (`.env`)
|
|
||||||
|
|
||||||
Create or update the `.env` file in the `ai-patient-onboarding/` directory:
|
|
||||||
|
|
||||||
```ini
|
|
||||||
APP_NAME="AI Patient Onboarding"
|
|
||||||
APP_ENV=development
|
|
||||||
APP_VERSION=1.0.0
|
|
||||||
|
|
||||||
# Database URL
|
|
||||||
DATABASE_URL="sqlite:///./data/onboarding.db"
|
|
||||||
|
|
||||||
# AI Provider API Keys
|
|
||||||
OPENAI_API_KEY="sk-..."
|
|
||||||
OPENAI_MODEL="gpt-4o-mini"
|
|
||||||
|
|
||||||
ANTHROPIC_API_KEY="sk-ant-..."
|
|
||||||
ANTHROPIC_MODEL="claude-3-5-sonnet-20241022"
|
|
||||||
|
|
||||||
GEMINI_API_KEY="AIzaSy..."
|
|
||||||
GEMINI_MODEL="gemini-1.5-flash"
|
|
||||||
|
|
||||||
SARVAM_API_KEY="your_sarvam_key"
|
|
||||||
|
|
||||||
# External Destination Hospital EHR / Patient API (Optional)
|
|
||||||
PATIENT_API_BASE_URL="https://your-hospital-emr.com/api/v1"
|
|
||||||
PATIENT_API_TOKEN="your_secure_bearer_token"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Running the Backend Locally
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Navigate to the backend directory
|
|
||||||
cd ai-patient-onboarding
|
|
||||||
|
|
||||||
# Activate Python Virtual Environment
|
|
||||||
# Windows PowerShell:
|
|
||||||
.\venv\Scripts\Activate.ps1
|
|
||||||
# Linux / macOS:
|
|
||||||
# source venv/bin/activate
|
|
||||||
|
|
||||||
# Install dependencies if not already installed
|
|
||||||
pip install -r requirements.txt
|
|
||||||
|
|
||||||
# Start FastAPI development server
|
|
||||||
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Voice Onboarding API (`/v1/voice-onboarding`)
|
|
||||||
|
|
||||||
### 4.1 Start Voice Session
|
|
||||||
|
|
||||||
Creates a new voice onboarding session, initializes dynamic state, and returns localized assistant greeting with audio.
|
|
||||||
|
|
||||||
- **URL:** `POST /v1/voice-onboarding/start`
|
|
||||||
- **Headers:** `Content-Type: application/json`
|
|
||||||
- **Request Body:**
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"patient": {
|
"patient": {
|
||||||
"id": "PAT-9842",
|
"id": "PAT-9624",
|
||||||
"name": "Arun Kumar",
|
"name": "Gokul Varthan",
|
||||||
"age": 42,
|
"age": 25,
|
||||||
"gender": "male",
|
"gender": "male"
|
||||||
"abha_id": "91-1234-5678-9012",
|
|
||||||
"phone": "+91 98765 43210",
|
|
||||||
"insurance_scheme": "PM-JAY",
|
|
||||||
"dietary_habits": "Vegetarian"
|
|
||||||
},
|
},
|
||||||
"category": {
|
"category": {
|
||||||
"type": "speciality",
|
"type": "general",
|
||||||
"code": "cardio",
|
"code": "general_medicine",
|
||||||
"name": "Cardiology"
|
"name": "General Medicine / Primary Care"
|
||||||
},
|
},
|
||||||
"language": "en-IN"
|
"language": "ta-IN"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
- **Response (`200 OK`):**
|
*Note: `category` is **optional**. If omitted, it defaults to `general_medicine` (Auto-Triage).*
|
||||||
|
|
||||||
|
#### Supported Languages
|
||||||
|
- `ta-IN`: Tamil (Pechu Tamil Spoken Dialect)
|
||||||
|
- `hi-IN`: Hindi (Conversational Spoken Hindi)
|
||||||
|
- `te-IN`: Telugu (Conversational Telugu)
|
||||||
|
- `kn-IN`: Kannada (Conversational Kannada)
|
||||||
|
- `en-IN`: Indian English (Empathetic Spoken English)
|
||||||
|
|
||||||
|
#### Response (`200 OK`)
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"success": true,
|
"success": true,
|
||||||
"session_id": "VOICE-78A1BF2C",
|
"session_id": "VOICE-4C5F63F0",
|
||||||
"patient_id": "PAT-9842",
|
"patient_id": "PAT-9624",
|
||||||
"category": {
|
"category": {
|
||||||
"type": "speciality",
|
"type": "general",
|
||||||
"code": "cardio",
|
"code": "general_medicine",
|
||||||
"name": "Cardiology"
|
"name": "General Medicine / Primary Care"
|
||||||
},
|
},
|
||||||
"language": "en-IN",
|
"language": "ta-IN",
|
||||||
"status": "in_progress",
|
"status": "in_progress",
|
||||||
"stage": "greeting",
|
"stage": "greeting",
|
||||||
"greeting_text": "Hello Arun Kumar, welcome to CureSelect Cardiology department. What symptoms or discomfort are you experiencing today?",
|
"greeting_text": "வணக்கம் Gokul Varthan, நான் உங்க AI அசிஸ்டன்ட் நிலா. உங்களுக்கு உடம்புக்கு என்ன பண்ணுது, என்ன பிரச்சனைன்னு சொல்லுங்க?",
|
||||||
"audio_url": "/v1/voice-onboarding/audio/voice_greeting_89bf1a.wav"
|
"audio_url": "/v1/voice-onboarding/audio/aud_32dd00c39aa1",
|
||||||
|
"response_time_ms": 36.33,
|
||||||
|
"response_time_sec": 0.036
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### 4.2 Send Voice / Speech Turn
|
### 1.2 Submit Voice/Text Conversation Turn
|
||||||
|
Submits the patient's voice audio recording or text fallback. The backend runs ASR (Sarvam Speech-to-Text), Google Gemini AI reasoning, and returns spoken audio URL + dynamic triage category.
|
||||||
|
|
||||||
Submits the patient's recorded audio or transcribed speech. The backend runs multi-model Indic speech recognition (Sarvam/Whisper), extracts medical entities, updates conversational state, and generates the next question with audio TTS.
|
- **Endpoint**: `POST /v1/voice-onboarding/{session_id}/turn`
|
||||||
|
- **Content-Type**: `multipart/form-data`
|
||||||
|
|
||||||
- **URL:** `POST /v1/voice-onboarding/{session_id}/turn`
|
#### Form Fields
|
||||||
- **Headers:** `Content-Type: multipart/form-data`
|
| Field Name | Type | Required | Description |
|
||||||
- **Form Data Parameters:**
|
| :--- | :--- | :--- | :--- |
|
||||||
- `audio` *(Optional File)*: WebM, WAV, or MP3 recorded audio file from the user's microphone.
|
| `audio` | `File` (WebM/WAV) | Optional | Audio blob recorded by microphone. |
|
||||||
- `text_fallback` *(Optional String)*: Real-time browser speech recognition transcript fallback.
|
| `text_fallback` | `string` | Optional | Text entered by patient. **Pass `null` or omit when sending `audio` so Sarvam ASR runs.** |
|
||||||
|
|
||||||
- **Response (`200 OK`):**
|
#### React / JavaScript Integration Example
|
||||||
```json
|
```javascript
|
||||||
{
|
const sendVoiceTurn = async (sessionId, audioBlob, textInput) => {
|
||||||
"success": true,
|
const formData = new FormData();
|
||||||
"session_id": "VOICE-78A1BF2C",
|
if (audioBlob) {
|
||||||
"patient_transcript": "I have had severe chest pain and breathlessness since yesterday evening.",
|
formData.append('audio', audioBlob, 'recording.webm');
|
||||||
"ai_text": "I understand. Does the chest pain radiate to your left arm or jaw, and are you currently taking any blood pressure or heart medications?",
|
|
||||||
"stage": "symptoms_inquiry",
|
|
||||||
"completed": false,
|
|
||||||
"audio_url": "/v1/voice-onboarding/audio/speech_reply_30ef2.wav"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 4.3 Fetch Synthesized Voice Audio
|
|
||||||
|
|
||||||
Streams audio file generated for assistant responses.
|
|
||||||
|
|
||||||
- **URL:** `GET /v1/voice-onboarding/audio/{audio_id}`
|
|
||||||
- **Response:** `audio/wav` or `audio/mpeg` binary stream.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 4.4 Complete & Confirm Session
|
|
||||||
|
|
||||||
Extracts complete structured clinical results. Supports a two-step confirmation workflow (review preview $\rightarrow$ permanent submission).
|
|
||||||
|
|
||||||
- **URL:** `POST /v1/voice-onboarding/{session_id}/complete`
|
|
||||||
- **Headers:** `Content-Type: application/json`
|
|
||||||
|
|
||||||
#### Step A: Request Review Preview (`confirmed: false`)
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"confirmed": false
|
|
||||||
}
|
|
||||||
```
|
|
||||||
**Response (`200 OK`):**
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"success": true,
|
|
||||||
"session_id": "VOICE-78A1BF2C",
|
|
||||||
"status": "awaiting_confirmation",
|
|
||||||
"result": {
|
|
||||||
"patient_id": "PAT-9842",
|
|
||||||
"abha_id": "91-1234-5678-9012",
|
|
||||||
"phone": "+91 98765 43210",
|
|
||||||
"insurance_scheme": "PM-JAY",
|
|
||||||
"category": {
|
|
||||||
"type": "speciality",
|
|
||||||
"code": "cardio",
|
|
||||||
"name": "Cardiology"
|
|
||||||
},
|
|
||||||
"chief_complaint": "Acute onset retrosternal chest pain and dyspnea",
|
|
||||||
"symptoms": [
|
|
||||||
{
|
|
||||||
"name": "Chest tightness",
|
|
||||||
"duration": "1 day",
|
|
||||||
"severity": "Moderate to Severe",
|
|
||||||
"description": "Retrosternal pressure aggravated by exertion"
|
|
||||||
}
|
}
|
||||||
],
|
if (textInput && textInput.trim()) {
|
||||||
"medical_history": ["Hypertension (5 years)"],
|
formData.append('text_fallback', textInput.trim());
|
||||||
"medications": ["Amlodipine 5mg OD"],
|
}
|
||||||
"allergies": ["Penicillin"],
|
|
||||||
"dietary_habits": "Vegetarian",
|
const response = await fetch(`http://127.0.0.1:8000/v1/voice-onboarding/${sessionId}/turn`, {
|
||||||
"additional_information": "Non-smoker",
|
method: 'POST',
|
||||||
"language": "en-IN",
|
body: formData,
|
||||||
"onboarding_completed": true
|
});
|
||||||
|
|
||||||
|
return await response.json();
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (`200 OK`)
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"session_id": "VOICE-4C5F63F0",
|
||||||
|
"patient_transcript": "எனக்கு ரொம்ப மன அழுத்தம் மற்றும் மனச்சோர்வு இருக்கு",
|
||||||
|
"ai_text": "புரியுதுங்க கோகுல், இந்த மன அழுத்தமும் மனச்சோர்வும் எத்தனை நாளா இருக்குன்னு சொல்ல முடியுமா?",
|
||||||
|
"stage": "reason_for_visit",
|
||||||
|
"completed": false,
|
||||||
|
"audio_url": "/v1/voice-onboarding/audio/aud_a1b2c3d4e5f6",
|
||||||
|
"suggested_specialty": {
|
||||||
|
"type": "speciality",
|
||||||
|
"code": "psychiatry",
|
||||||
|
"name": "Psychiatry & Mental Health Care",
|
||||||
|
"confidence": 0.95,
|
||||||
|
"reason": "Automatically classified based on clinical keyword: 'மன அழுத்தம்'"
|
||||||
},
|
},
|
||||||
"message": "Please review the collected information and confirm by voice or tap."
|
"response_time_ms": 1420.50,
|
||||||
|
"response_time_sec": 1.421
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Step B: Confirm & Submit to EHR (`confirmed: true`)
|
---
|
||||||
|
|
||||||
|
### 1.3 Stream Spoken Audio (TTS)
|
||||||
|
Streams synthesized audio response files.
|
||||||
|
|
||||||
|
- **Endpoint**: `GET /v1/voice-onboarding/audio/{audio_id}`
|
||||||
|
- **Response Headers**: `Content-Type: audio/wav`, `Content-Disposition: inline`
|
||||||
|
|
||||||
|
#### Audio Playback Code
|
||||||
|
```javascript
|
||||||
|
const playAudioStream = async (audioUrl) => {
|
||||||
|
if (!audioUrl) return;
|
||||||
|
const fullUrl = audioUrl.startsWith('http') ? audioUrl : `http://127.0.0.1:8000${audioUrl}`;
|
||||||
|
const audio = new Audio(fullUrl);
|
||||||
|
try {
|
||||||
|
await audio.play();
|
||||||
|
} catch (err) {
|
||||||
|
console.warn("Audio autoplay blocked by browser:", err);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 1.4 Complete Voice Session & Export EHR JSON
|
||||||
|
Finalizes intake and exports the complete structured EHR JSON.
|
||||||
|
|
||||||
|
- **Endpoint**: `POST /v1/voice-onboarding/{session_id}/complete`
|
||||||
|
- **Content-Type**: `application/json`
|
||||||
|
|
||||||
|
#### Request Payload
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"confirmed": true
|
"confirmed": true
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
**Response (`200 OK`):**
|
|
||||||
|
#### Response (`200 OK`)
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"success": true,
|
"success": true,
|
||||||
"session_id": "VOICE-78A1BF2C",
|
"session_id": "VOICE-4C5F63F0",
|
||||||
"status": "submitted_and_cleaned",
|
"status": "submitted_and_cleaned",
|
||||||
"result": { ... },
|
"message": "Voice onboarding successfully saved to patient record.",
|
||||||
"message": "Voice onboarding successfully saved to patient record."
|
"result": {
|
||||||
|
"patient_id": "PAT-9624",
|
||||||
|
"category": {
|
||||||
|
"type": "speciality",
|
||||||
|
"code": "psychiatry",
|
||||||
|
"name": "Psychiatry & Mental Health Care"
|
||||||
|
},
|
||||||
|
"chief_complaint": "Severe mental depression and stress",
|
||||||
|
"symptoms": [
|
||||||
|
{
|
||||||
|
"name": "Mental depression",
|
||||||
|
"duration": "2 months",
|
||||||
|
"severity": "Moderate",
|
||||||
|
"description": "Patient reports persistent sadness and low mood."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"medical_history": [],
|
||||||
|
"medications": [],
|
||||||
|
"allergies": ["No Known Drug Allergies (NKDA)"],
|
||||||
|
"additional_information": "Referred to specialist psychiatrist.",
|
||||||
|
"language": "ta-IN",
|
||||||
|
"onboarding_completed": true
|
||||||
|
},
|
||||||
|
"response_time_ms": 2840.10,
|
||||||
|
"response_time_sec": 2.840
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Text Onboarding API (`/v1/onboarding`)
|
# SECTION 2: Text AI Onboarding API (`/v1/onboarding`)
|
||||||
|
|
||||||
For text chat interfaces or non-voice kiosks.
|
### 2.1 Start Text Onboarding Session
|
||||||
|
- **Endpoint**: `POST /v1/onboarding/start`
|
||||||
|
- **Content-Type**: `application/json`
|
||||||
|
|
||||||
### 5.1 Start Text Session
|
#### Request Payload
|
||||||
- **URL:** `POST /v1/onboarding/start`
|
|
||||||
- **Body:** Same as Voice Start (`patient`, `category`, `language`).
|
|
||||||
- **Response:** Returns `session_id`, `greeting_message`, and initial state.
|
|
||||||
|
|
||||||
### 5.2 Send Text Message
|
|
||||||
- **URL:** `POST /v1/onboarding/{session_id}/message`
|
|
||||||
- **Body:**
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"message": "I've had knee pain for 3 days after a run."
|
"patient": {
|
||||||
|
"id": "PAT-9624",
|
||||||
|
"name": "Gokul Varthan",
|
||||||
|
"age": 25,
|
||||||
|
"gender": "male"
|
||||||
|
},
|
||||||
|
"category": {
|
||||||
|
"type": "general",
|
||||||
|
"code": "general_medicine",
|
||||||
|
"name": "General Medicine / Primary Care"
|
||||||
|
},
|
||||||
|
"language": "en"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
- **Response:**
|
|
||||||
|
#### Response (`200 OK`)
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"success": true,
|
"success": true,
|
||||||
"session_id": "ONB-84A12B1C",
|
"session_id": "ONB-8F3A2B1C",
|
||||||
"stage": "symptoms_inquiry",
|
"patient_id": "PAT-9624",
|
||||||
"message": "Is the knee swollen or locking when you walk?",
|
"category": {
|
||||||
|
"type": "general",
|
||||||
|
"code": "general_medicine",
|
||||||
|
"name": "General Medicine / Primary Care"
|
||||||
|
},
|
||||||
|
"language": "en",
|
||||||
|
"status": "in_progress",
|
||||||
|
"stage": "greeting",
|
||||||
|
"message": "Hello Gokul Varthan, welcome to CureSelect Healthcare. Could you please share what health symptoms you are experiencing today?"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2.2 Send Text Message
|
||||||
|
- **Endpoint**: `POST /v1/onboarding/{session_id}/message`
|
||||||
|
- **Content-Type**: `application/json`
|
||||||
|
|
||||||
|
#### Request Payload
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"message": "I have been experiencing a severe headache and fever since yesterday."
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (`200 OK`)
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"session_id": "ONB-8F3A2B1C",
|
||||||
|
"stage": "symptoms",
|
||||||
|
"message": "Understood Gokul. Have you taken any fever or pain reliever medication yet?",
|
||||||
"status": "in_progress"
|
"status": "in_progress"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5.3 Complete Text Onboarding
|
|
||||||
- **URL:** `POST /v1/onboarding/{session_id}/complete`
|
|
||||||
- **Body:** `{"confirmed": true}`
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. Medical Document & Vision API (`/v1/documents`)
|
### 2.3 Complete Text Onboarding & Export EHR JSON
|
||||||
|
- **Endpoint**: `POST /v1/onboarding/{session_id}/complete`
|
||||||
|
- **Content-Type**: `application/json`
|
||||||
|
|
||||||
Uploads patient medical documents (prescriptions, discharge summaries, lab reports) to extract medications, allergies, and diagnoses using Gemini 1.5 Flash Vision, automatically injecting findings into the active session state.
|
#### Request Payload
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"confirmed": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
- **URL:** `POST /v1/documents/{session_id}/upload`
|
#### Response (`200 OK`)
|
||||||
- **Headers:** `Content-Type: multipart/form-data`
|
|
||||||
- **Form Field:** `file` *(Binary image or PDF file)*
|
|
||||||
|
|
||||||
- **Response (`200 OK`):**
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"success": true,
|
"success": true,
|
||||||
"session_id": "VOICE-78A1BF2C",
|
"session_id": "ONB-8F3A2B1C",
|
||||||
"document_type": "prescription",
|
"status": "submitted_and_cleaned",
|
||||||
"summary": "Prescription from Cardiologist dated 15-Aug-2024",
|
"message": "Onboarding successfully completed, saved to patient record, and temporary session cleaned.",
|
||||||
"findings": ["Ejection Fraction 55%", "Normal Sinus Rhythm"],
|
"result": {
|
||||||
"medications_found": ["Atorvastatin 20mg", "Metoprolol 25mg"],
|
"patient_id": "PAT-9624",
|
||||||
"allergies_found": ["Sulfa drugs"],
|
"category": {
|
||||||
"message": "Medical document analyzed and integrated into onboarding session."
|
"type": "general",
|
||||||
}
|
"code": "general_medicine",
|
||||||
```
|
"name": "General Medicine / Primary Care"
|
||||||
|
},
|
||||||
---
|
"chief_complaint": "Severe headache and fever",
|
||||||
|
"symptoms": [
|
||||||
## 7. System Health Check (`/health`)
|
{
|
||||||
|
"name": "Headache and fever",
|
||||||
- **URL:** `GET /health`
|
"duration": "1 day",
|
||||||
- **Response (`200 OK`):**
|
"severity": "Severe",
|
||||||
```json
|
"description": "Patient reports onset yesterday."
|
||||||
{
|
}
|
||||||
"success": true,
|
],
|
||||||
"service": "AI Patient Onboarding",
|
"medical_history": [],
|
||||||
"status": "healthy",
|
"medications": [],
|
||||||
"models": {
|
"allergies": ["No Known Drug Allergies (NKDA)"],
|
||||||
"speech": "Sarvam AI (Indian Languages) / OpenAI Whisper",
|
"language": "en",
|
||||||
"conversation": "OpenAI GPT-4o-mini",
|
"onboarding_completed": true
|
||||||
"validation": "Claude 3.5 Sonnet",
|
|
||||||
"vision_documents": "Gemini 1.5 Flash"
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. Connecting External Hospital EMR / EHR System
|
# Architecture Sequence Diagram
|
||||||
|
|
||||||
When onboarding is finalized, the backend invokes `PatientSubmissionService` located at:
|
```mermaid
|
||||||
`app/services/patient_submission_service.py`
|
sequenceDiagram
|
||||||
|
autonumber
|
||||||
|
actor Patient
|
||||||
|
participant Frontend as React / Web App
|
||||||
|
participant API as FastAPI Server
|
||||||
|
participant Sarvam as Sarvam Indic ASR / TTS
|
||||||
|
participant Gemini as Google Gemini AI
|
||||||
|
|
||||||
### Customizing the Destination Forwarding:
|
Patient->>Frontend: Clicks "Start Intake"
|
||||||
```python
|
Frontend->>API: POST /v1/voice-onboarding/start
|
||||||
import httpx
|
API->>Sarvam: Generate Spoken Greeting (TTS)
|
||||||
from app.config import PATIENT_API_BASE_URL, PATIENT_API_TOKEN
|
API-->>Frontend: Returns session_id, greeting_text, audio_url, response_time_ms
|
||||||
|
Frontend->>Patient: Plays audio greeting & displays text
|
||||||
|
|
||||||
class PatientSubmissionService:
|
Patient->>Frontend: Speaks into Microphone / Types
|
||||||
async def submit(self, result: dict) -> dict:
|
Frontend->>API: POST /v1/voice-onboarding/{session_id}/turn (formData)
|
||||||
if not PATIENT_API_BASE_URL:
|
alt Voice Audio Sent
|
||||||
# Local fallback mock
|
API->>Sarvam: ASR Speech-to-Text
|
||||||
return {
|
Sarvam-->>API: Returns transcript
|
||||||
"success": True,
|
end
|
||||||
"message": "Patient data saved locally.",
|
API->>Gemini: Process context & generate dynamic turn
|
||||||
"patient_id": result.get("patient_id"),
|
Gemini-->>API: Returns dynamic response & auto-classified specialty
|
||||||
}
|
API->>Sarvam: Generate AI Spoken Response (TTS)
|
||||||
|
API-->>Frontend: Returns ai_text, audio_url, suggested_specialty, response_time_ms
|
||||||
|
|
||||||
# Forward JSON payload to hospital central server
|
Frontend->>Patient: Plays AI audio response & updates UI
|
||||||
async with httpx.AsyncClient(timeout=15.0) as client:
|
|
||||||
headers = {
|
|
||||||
"Authorization": f"Bearer {PATIENT_API_TOKEN}",
|
|
||||||
"Content-Type": "application/json",
|
|
||||||
}
|
|
||||||
response = await client.post(
|
|
||||||
f"{PATIENT_API_BASE_URL}/patients/{result.get('patient_id')}/onboarding",
|
|
||||||
json=result,
|
|
||||||
headers=headers,
|
|
||||||
)
|
|
||||||
|
|
||||||
if response.status_code in [200, 201]:
|
Patient->>Frontend: Final Confirmation
|
||||||
return {"success": True, "data": response.json()}
|
Frontend->>API: POST /v1/voice-onboarding/{session_id}/complete ({confirmed: true})
|
||||||
|
API-->>Frontend: Returns Final Structured EHR JSON
|
||||||
return {
|
|
||||||
"success": False,
|
|
||||||
"error": f"Hospital API error: {response.text}",
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 9. Multi-Language & Clinical Specialties
|
|
||||||
|
|
||||||
### Supported Languages
|
|
||||||
| Code | Language | Speech Engine |
|
|
||||||
|---|---|---|
|
|
||||||
| `en-IN` / `en` | English (Indian accent) | Sarvam / OpenAI Whisper |
|
|
||||||
| `hi-IN` / `hi` | Hindi (हिन्दी) | Sarvam Bulbul / Whisper |
|
|
||||||
| `ta-IN` / `ta` | Tamil (தமிழ்) | Sarvam Bulbul / Whisper |
|
|
||||||
| `te-IN` / `te` | Telugu (తెలుగు) | Sarvam Bulbul / Whisper |
|
|
||||||
| `kn-IN` / `kn` | Kannada (ಕನ್ನಡ) | Sarvam Bulbul / Whisper |
|
|
||||||
| `bn-IN` / `bn` | Bengali (বাংলা) | Sarvam Bulbul / Whisper |
|
|
||||||
| `ml-IN` / `ml` | Malayalam (മലയാളം) | Sarvam Bulbul / Whisper |
|
|
||||||
| `mr-IN` / `mr` | Marathi (मराठी) | Sarvam Bulbul / Whisper |
|
|
||||||
| `gu-IN` / `gu` | Gujarati (ગુજરાતી) | Sarvam Bulbul / Whisper |
|
|
||||||
|
|
||||||
### Clinical Specialties Supported
|
|
||||||
- `general_medicine` - General Medicine / Primary Care
|
|
||||||
- `cardio` - Cardiology
|
|
||||||
- `ortho` - Orthopedics
|
|
||||||
- `peds` - Pediatrics
|
|
||||||
- `derma` - Dermatology
|
|
||||||
- `neuro` - Neurology
|
|
||||||
- `gastro` - Gastroenterology
|
|
||||||
- `ent` - ENT / Otorhinolaryngology
|
|
||||||
- `gyn` - Obstetrics & Gynecology
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 10. Client Integration Code Samples
|
|
||||||
|
|
||||||
### 10.1 Python Client (`httpx`)
|
|
||||||
```python
|
|
||||||
import httpx
|
|
||||||
import asyncio
|
|
||||||
|
|
||||||
API_BASE = "http://127.0.0.1:8000"
|
|
||||||
|
|
||||||
async def main():
|
|
||||||
async with httpx.AsyncClient() as client:
|
|
||||||
# 1. Start Session
|
|
||||||
start_res = await client.post(f"{API_BASE}/v1/voice-onboarding/start", json={
|
|
||||||
"patient": {"id": "P-1001", "name": "Deepak Sharma", "age": 45, "gender": "male"},
|
|
||||||
"category": {"type": "speciality", "code": "cardio", "name": "Cardiology"},
|
|
||||||
"language": "hi-IN"
|
|
||||||
})
|
|
||||||
session = start_res.json()
|
|
||||||
session_id = session["session_id"]
|
|
||||||
print(f"Session started: {session_id}")
|
|
||||||
print(f"Greeting: {session['greeting_text']}")
|
|
||||||
|
|
||||||
# 2. Send Turn
|
|
||||||
turn_res = await client.post(
|
|
||||||
f"{API_BASE}/v1/voice-onboarding/{session_id}/turn",
|
|
||||||
data={"text_fallback": "मुझे सीने में भारीपन महसूस हो रहा है"}
|
|
||||||
)
|
|
||||||
turn_data = turn_res.json()
|
|
||||||
print(f"AI Response: {turn_data['ai_text']}")
|
|
||||||
|
|
||||||
# 3. Finalize & Submit
|
|
||||||
complete_res = await client.post(
|
|
||||||
f"{API_BASE}/v1/voice-onboarding/{session_id}/complete",
|
|
||||||
json={"confirmed": True}
|
|
||||||
)
|
|
||||||
print(f"EHR Saved: {complete_res.json()}")
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
asyncio.run(main())
|
|
||||||
```
|
|
||||||
|
|
||||||
### 10.2 JavaScript / TypeScript (`fetch`)
|
|
||||||
```typescript
|
|
||||||
const API_BASE = "http://127.0.0.1:8000";
|
|
||||||
|
|
||||||
// Start voice onboarding
|
|
||||||
export async function startVoiceOnboarding(patient: any, category: any, language = "en-IN") {
|
|
||||||
const res = await fetch(`${API_BASE}/v1/voice-onboarding/start`, {
|
|
||||||
method: "POST",
|
|
||||||
headers: { "Content-Type": "application/json" },
|
|
||||||
body: JSON.stringify({ patient, category, language })
|
|
||||||
});
|
|
||||||
return res.json();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Send audio turn
|
|
||||||
export async function sendVoiceTurn(sessionId: string, audioBlob?: Blob, textFallback?: string) {
|
|
||||||
const formData = new FormData();
|
|
||||||
if (audioBlob) formData.append("audio", audioBlob, "speech.webm");
|
|
||||||
if (textFallback) formData.append("text_fallback", textFallback);
|
|
||||||
|
|
||||||
const res = await fetch(`${API_BASE}/v1/voice-onboarding/${sessionId}/turn`, {
|
|
||||||
method: "POST",
|
|
||||||
body: formData
|
|
||||||
});
|
|
||||||
return res.json();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Upload medical report
|
|
||||||
export async function uploadPrescription(sessionId: string, file: File) {
|
|
||||||
const formData = new FormData();
|
|
||||||
formData.append("file", file);
|
|
||||||
|
|
||||||
const res = await fetch(`${API_BASE}/v1/documents/${sessionId}/upload`, {
|
|
||||||
method: "POST",
|
|
||||||
body: formData
|
|
||||||
});
|
|
||||||
return res.json();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Complete onboarding
|
|
||||||
export async function completeOnboarding(sessionId: string, confirmed: boolean) {
|
|
||||||
const res = await fetch(`${API_BASE}/v1/voice-onboarding/${sessionId}/complete`, {
|
|
||||||
method: "POST",
|
|
||||||
headers: { "Content-Type": "application/json" },
|
|
||||||
body: JSON.stringify({ confirmed })
|
|
||||||
});
|
|
||||||
return res.json();
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 10.3 cURL Commands
|
|
||||||
|
|
||||||
#### Start Session:
|
|
||||||
```bash
|
|
||||||
curl -X POST "http://127.0.0.1:8000/v1/voice-onboarding/start" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{
|
|
||||||
"patient": {"id": "P-101", "name": "Arun Kumar", "age": 40, "gender": "male"},
|
|
||||||
"category": {"type": "general", "code": "general_medicine", "name": "General Medicine"},
|
|
||||||
"language": "en-IN"
|
|
||||||
}'
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Send Voice Turn (with text fallback):
|
|
||||||
```bash
|
|
||||||
curl -X POST "http://127.0.0.1:8000/v1/voice-onboarding/VOICE-XXXXX/turn" \
|
|
||||||
-F "text_fallback=I have had high fever and sore throat since yesterday"
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Upload Prescription:
|
|
||||||
```bash
|
|
||||||
curl -X POST "http://127.0.0.1:8000/v1/documents/VOICE-XXXXX/upload" \
|
|
||||||
-F "file=@/path/to/prescription.pdf"
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Confirm & Submit:
|
|
||||||
```bash
|
|
||||||
curl -X POST "http://127.0.0.1:8000/v1/voice-onboarding/VOICE-XXXXX/complete" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"confirmed": true}'
|
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -350,6 +350,7 @@ async def complete_voice_onboarding(
|
|||||||
request: CompleteOnboardingRequest,
|
request: CompleteOnboardingRequest,
|
||||||
db: Session = Depends(get_db),
|
db: Session = Depends(get_db),
|
||||||
):
|
):
|
||||||
|
start_time = time.time()
|
||||||
onboarding_service = OnboardingService()
|
onboarding_service = OnboardingService()
|
||||||
completion_service = CompletionService()
|
completion_service = CompletionService()
|
||||||
result_service = ResultExtractionService()
|
result_service = ResultExtractionService()
|
||||||
@@ -384,6 +385,10 @@ async def complete_voice_onboarding(
|
|||||||
conversation_history=conversation_history,
|
conversation_history=conversation_history,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
elapsed = time.time() - start_time
|
||||||
|
resp_ms = round(elapsed * 1000, 2)
|
||||||
|
resp_sec = round(elapsed, 3)
|
||||||
|
|
||||||
if not request.confirmed:
|
if not request.confirmed:
|
||||||
return CompleteOnboardingResponse(
|
return CompleteOnboardingResponse(
|
||||||
success=True,
|
success=True,
|
||||||
@@ -391,6 +396,8 @@ async def complete_voice_onboarding(
|
|||||||
status="awaiting_confirmation",
|
status="awaiting_confirmation",
|
||||||
result=result,
|
result=result,
|
||||||
message="Please review the collected information and confirm by voice or tap.",
|
message="Please review the collected information and confirm by voice or tap.",
|
||||||
|
response_time_ms=resp_ms,
|
||||||
|
response_time_sec=resp_sec,
|
||||||
)
|
)
|
||||||
|
|
||||||
submission = await submission_service.submit(result)
|
submission = await submission_service.submit(result)
|
||||||
@@ -407,6 +414,8 @@ async def complete_voice_onboarding(
|
|||||||
status="submitted_and_cleaned",
|
status="submitted_and_cleaned",
|
||||||
result=result,
|
result=result,
|
||||||
message="Voice onboarding successfully saved to patient record.",
|
message="Voice onboarding successfully saved to patient record.",
|
||||||
|
response_time_ms=resp_ms,
|
||||||
|
response_time_sec=resp_sec,
|
||||||
)
|
)
|
||||||
|
|
||||||
session.status = "submission_pending_retry"
|
session.status = "submission_pending_retry"
|
||||||
@@ -418,4 +427,6 @@ async def complete_voice_onboarding(
|
|||||||
status="submission_pending_retry",
|
status="submission_pending_retry",
|
||||||
result=result,
|
result=result,
|
||||||
message="Patient API submission failed. Session preserved for retry.",
|
message="Patient API submission failed. Session preserved for retry.",
|
||||||
|
response_time_ms=resp_ms,
|
||||||
|
response_time_sec=resp_sec,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -98,3 +98,5 @@ class CompleteOnboardingResponse(BaseModel):
|
|||||||
status: str
|
status: str
|
||||||
result: Optional[PatientOnboardingResult] = None
|
result: Optional[PatientOnboardingResult] = None
|
||||||
message: str
|
message: str
|
||||||
|
response_time_ms: Optional[float] = None
|
||||||
|
response_time_sec: Optional[float] = None
|
||||||
|
|||||||
Reference in New Issue
Block a user