Whisper for radiology dictation: transcription with medical context

Why base Whisper isn’t enough

OpenAI Whisper is an excellent voice-to-text transcription model, but radiology dictation has three challenges that naive Whisper doesn’t solve alone:

1. Clinical vocabulary: names of findings, procedures, drugs and acronyms (POC, pneumothorax, Hounsfield) that generic Whisper gets wrong.
2. Noisy environment: the reading room is not a recording studio. Keyboards, cross-conversations, fans.
3. Coherence with the study: transcription must align with study type, anatomical region and protocol, not be “guessed” by a generic language model.

Adaptations for medical context

What we do at Soft[in]Health to bring Whisper to the clinic:

Light fine-tuning with radiology corpus

We don’t train Whisper from scratch (that would be irrational). We do a light fine-tuning on an internal corpus of anonymized radiology dictations, reinforcing the clinical lexicon.

Conditioned glossaries and prompts

We pass the study context (modality, region, protocol) as a conditioned prompt. So “head” reads as an anatomical region, not as colloquial slang.

Post-correction with medical NLP

After transcription, a second NLP model specialized in clinical terminology corrects finding names and normalizes them to SNOMED CT and RadLex.

Silent mode with VAD

Voice activity detection (VAD) before Whisper: we only transcribe segments where there’s voice, dramatically cutting cost and noise-induced error.

From text to structured draft

Transcribed text is not the report. The report has sections: technique, findings, impression, recommendation. A second NLP model parses the transcription and maps it to the study-type template:

 Whisper transcription
 → section segmentation (technique/findings/impression)
 → finding extraction with localization and characterization
 → SNOMED CT + RadLex mapping
 → DICOM SR with measurements
 → editable preview

The radiologist reviews and approves. Doesn’t write from scratch; corrects. The final report is DICOM SR, auditable and marks the difference between “Whisper + hand-edit” and “IA-native dictation to structured report”.

Whisper is 30% of the work. The 70% is what happens after: medical context, study awareness and mapping to DICOM SR structure. ::