Scope and approach
I investigated biomedical language and vision-language models for personalized treatment pathways, moving from medical question answering toward diagnosis and evidence-grounded recommendations. The work spans model comparison, parameter-efficient fine-tuning, retrieval over biomedical literature, multimodal information extraction, and a multi-agent diagnostic architecture.
The evaluation considered MedQA, PubMedQA, MedMCQA, and MMLU-Medical, alongside clinical cases and diagnostic-code prediction. Model families included MEDITRON, LLaVA-Med, MedAlpaca, OpenBioLLM, Llama, and DeepSeek. Standardized question answering and case-based clinical reasoning were treated as different views of model capability.
Parameter-efficient domain adaptation
Fine-tuned MEDITRON-7B with 8-bit quantization and LoRA on 200,000 artificially labeled PubMedQA examples. The configuration used rank 16, alpha 32, and dropout 0.1, targeting the attention projection layers. Training used AdamW, a learning rate of 2 × 10−5, three epochs, a 2,048-token maximum sequence length, mixed precision, gradient accumulation, and gradient checkpointing.
| Configuration | Accuracy |
|---|---|
| MEDITRON-7B baseline | 43.8% |
| Fine-tuned MEDITRON-7B | 73.8% |
The difference is 30.0 percentage points. LoRA-rank ablations explored ranks 4 through 64. Rank 16 offered a useful trade-off: rank 32 reached 74.1% but required 14.6 hours, compared with 73.8% in 8.4 hours at rank 16.
Retrieval over biomedical literature
Built a PubMed-based RAG pipeline with PubMedBERT embeddings, FAISS dense retrieval, BM25 sparse retrieval, and cross-encoder re-ranking. The documented setup uses 512-token chunks with 50-token overlap, 768-dimensional normalized embeddings, a 70/30 dense–sparse weighting, and five retrieved documents.
The thesis reports an average benchmark accuracy increase from 62.4% to 71.3% for fine-tuned MEDITRON-7B with retrieval: 8.9 percentage points. Component ablations report gains of 5.2 points for dense retrieval, 7.8 for hybrid retrieval, and 8.9 with re-ranking. Reported factual hallucination rates fell from 8.3% to 4.8% in the clinical evaluation, approximately a 42% relative reduction.
Retrieval quality, corpus recency, contradictory evidence, latency, and limited context windows remained important constraints. These are results from the thesis evaluation, rather than evidence of clinical deployment readiness.
Specialized diagnostic agents
Designed a system with three agents for symptoms, medical history, and clinical results, followed by a diagnostic integration agent. Agent-specific fine-tuning used DeepSeek R1-8B models. The integration stage receives structured summaries together with the original case and generates diagnoses with confidence scores.
Initial agents omitted safety details and psychiatric risk factors. I revised their inputs and output requirements to include urgency, severity, psychiatric history, medication safety, and structured differential diagnoses.
| Configuration | Peak F1 |
|---|---|
| Single diagnosis model | 0.475 |
| Original agents + case information | 0.455 |
| Original agents, without case information | 0.240 |
| Revised agents + case information | 0.490 |
| Revised agents, without case information | 0.460 |
| Revised agents + case information + ICD list | 0.420 |
| Revised agents + case information, 3-letter evaluation | 0.325 |
The best revised configuration improved on the single-model baseline by 0.015 absolute F1. The original agent system underperformed the baseline: specialization alone was insufficient. Retaining case information and revising the agent specifications materially affected the result.
Evaluation and robustness
Compared FrugalScore, BERTScore, GPT-4 judging, and BioBERT v2 against human assessment of diagnostic similarity. The thesis reports 82% agreement with human evaluation for BioBERT v2 F1, versus 71% for BERTScore and 76% for GPT-4 judging. This motivated closer attention to clinical meaning and partial correctness rather than relying on generic text overlap.
Error analysis examined diagnostic-code classes, patient age, confidence thresholds, missing case context, inaccurate clinical assumptions, and missed psychiatric risks. Strong question-answering scores did not guarantee appropriate recommendations in complex cases.
Additional directions explored
The thesis investigated extracting biomedical information from text, tables, and plots, including TabTransformer and DePlot-based components. It also explored fine-tuning Llama on DeepSeek-70B reasoning traces and an evidence-based differential diagnosis mechanism with five candidate diagnoses, supporting and opposing evidence, and a final weighting agent.
In the reported reasoning-distillation comparison, answer-only and reasoning-plus-answer Llama variants both reached 40% precision and 39% recall. That experiment did not demonstrate a gain from adding reasoning traces. Answer-conditioned teacher results were a diagnostic experiment and should not be read as independent diagnosis performance.
What the study establishes—and leaves open
The work supports domain adaptation, retrieval, and carefully designed agent interfaces as useful experimental directions. It also reveals a persistent gap between benchmark performance and clinical reasoning. The study uses curated cases, English-language sources, and a limited expert panel; reported improvements do not establish prospective patient benefit or autonomous clinical safety.
Future directions include clinically grounded evaluation, better uncertainty handling, more reliable evidence selection, and agents that preserve critical patient context.