The goal of this quasi-experimental educational study is to learn whether AI-generated post-consultation feedback in social robotic virtual patient interactions improves medical students' clinical performance in medical history-taking. The main question it aims to answer is: Can AI-generated feedback integrated in an AI-enhanced social robotic virtual patient platform improve medical students' clinical performance in medical history taking? Researchers will compare results from standardised examinations following the structure of an objective structured clinical examination (OSCE), of medical students performing virtual patient interactions with AI-generated post consultation feedback compared to medical students who have not received AI-generated feedback. Participants will perform five virtual patient cases in rheumatology using an established virtual patient platform: the Social AI-enhanced Robotic Interface (SARI). After completion of each case, students participate in follow-up seminars with consultant rheumatologists to discuss the cases. After completion of all nine cases, students take part in a OSCE based examination to evaluate medical-history taking.
The overarching aim of this project is to investigate if a system for direct feedback incorporated in an AI-empowered social robotic VP platform be more effective in the training of CR and communication skills for medical students than the same AI-empowered social robotic VP platform with no feedback system? Higher knowledge within this research field contributes to optimised usage of new technology alongside other educational resources to facilitate meaningful learning outcomes, including supervision and patient encounters. The short- and long-term aim is to improve the medical education environment to ensure better healthcare for patients living with rheumatic conditions, while being applicable to other medical disciplines as well. While it still undergoes refinements, this educational activity provides students with the opportunity to train in a safe and interactive environment. Background: In healthcare today, most services are provided at outpatient clinics and daycare wards, being also the case within rheumatology. However, a significant portion of undergraduate healthcare professions education (HPE) continues to take place in inpatient settings, where ensuring that hospitalised patients align with specific learning outcomes remains a challenge. The patients requiring hospital admission typically have longstanding conditions, often presenting with complex disease manifestations and complications, alongside substantial medical comorbidities. The COVID-19 pandemic highlighted additional barriers for students to engage with teachers, patients and peers. This situation, combined with previous proposals for alternative approaches toward theoretical and practical skills training in undergraduate HPE, emphasises the pressing need for innovative learning activities that facilitate meaningful interactions between students and patient cases that are represented in HPE curricula. Virtual patients (VP) are interactive digital simulations of clinical scenarios designed for educational purposes and are represented through a variety of systems and approaches. A fundamental component of VP simulations is their interactive interface, which enables users to engage with patients in a variety of ways: gathering medical history, conducting physical examinations, and implementing investigative strategies. These comprehensive possibilities of interaction allows users to collect essential information to perform diagnostics and decide upon appropriate management plans for their VPs. The rheumatology clinic at Karolinska University Hospital has complemented clinical placements for medical students over the past 15 years to ensure that students interact with representative patients by incorporating VP cases using computer software such as ReumaCase and Virtual Interactive Case simulator (VIC). These platforms exemplify semi-linear VP systems, enabling users to navigate freely between the predetermined points of initial patient presentation and concluding diagnostic and management decisions. HPE research has yielded novel learning approaches, including optimised VP implementation strategies and recommendations for developing clinical reasoning (CR) skills through VP practice. While undergraduate HPE students recognise VPs as valuable educational tools for CR training, they acknowledge that these simulations lack the complexity and interactivity of real patient encounters. While CR skills are important in medical education, some argue that empathy is one of the most important skills of healthcare practitioners engaged in patient care. Empathetic skills of healthcare providers have been shown to be associated with better health outcomes for patients and decrease the risk of hospitalisation. In undergraduate medical education, empathetic conduct is typically trained using role-playing with students, actors, or teachers portraying standardised patients, which are representative textbook medical conditions . However, standardised patients can be a costly option, and availability aspects can sometimes hurdle the practice of empathetic conduct. An alternative to this could be VPs that enable more authentic and interactive forms of communication. Recent technological advances have introduced enhanced interactive methods between users and VPs, particularly through artificial intelligence (AI) and social robotics. The investigators developed a novel platform which has been named Social AI-enhanced Robotic Interface (SARI), combining the social robot Furhat with the large language model (LLM) Chat-Generative Pretrained Transformer (Chat-GPT). This novel platform has been implemented for medical students at the Rheumatology clinic at Karolinska University Hospital to complement their training during clinical placements, alongside conventional computer-based VP encounters (VIC). Initial findings suggest that students perceive that SARI offers a more authentic and interactive VP experience in developing CR skills compared to a conventional computer-based platform. This project aims to evaluate the effectiveness of SARI as a VP platform for medical students, comparing whether SARI combined with an integrated AI-generated direct feedback system further enhances clinical performance in medical history-taking. Prior to this study the investigators have developed a total of ten VP cases, that have been implemented and are currently being used for clinical placements at the Division of Rheumatology at Karolinska University Hospital. Half of these cases have been developed in the computer-based platform VIC and half of them have been developed in the AI-enhanced social robotic platform SARI. The VP cases in both platforms have been developed according to distinct recommendations for VP platform development and are continuously being evaluated by medical students through direct (oral feedback and evaluation forms) and indirect (from analysis of research data) feedback. All the VP cases are now part of the educational activity "The Virtual Outpatient Clinic", which serves as a complement to clinical placements within rheumatology. Within this activity, all sixth semester medical students from Karolinska Institutet participate during a period of one and a half workdays during a one-week placement in rheumatology. "The Virtual Outpatient Clinic" has been ongoing at the clinic since the spring of 2024, hosting approximately 150 students every term during a period of six weeks. In this educational activity all students perform all unique VP cases and participate in follow-up seminars with consultant rheumatologists after each case to discuss questions regarding the case content and the represented rheumatic conditions. In the beginning of the activity, all students are asked to participate in the research project by evaluating the VP cases to further improve and investigate their added value using validated indices for VP platform development. Upon agreeing to participate and signing an informed consent form, every students' interaction with SARI is stored as an audio file and transcribed in real time. To this date, approximately 400 students have interacted with the five VP case in SARI which have resulted in almost 2000 unique VP interactions. The investigators have performed an initial evaluation of SARI and compared its performance in medical students' self-perceived acquirement of CR skills compared to VIC, utilising mixed quantitative and qualitative methodology during 2023. Fifteen medical students were included and completed the same VP case in both SARI and VIC. Students' self-perceived VP experience focusing on CR training was assessed using a previously validated index for VP platform development \[23\]. Furthermore, in-depth interviews were conducted with 8 medical students. This initial evaluation illustrated significant advantages in favour of SARI for the theme authenticity as well as the learning effect of the VP encounter according to predefined themes in the VP evaluation questionnaire. Along the analysis of those predefined themes, thematic analysis was performed of data from in-depth interviews, which resulted in four themes. Students experienced SARI as superior to VIC in training CR, communication, and emotional skills. During the spring of 2024, a total of 23 students from The Virtual Outpatient Clinic participated in in-depth interviews as a continuation project addressing their perceptions of the added value of SARI and VIC regarding acquirement of CR skills. For this study, the investigators created 4 additional VP cases in each platform. Furthermore, all students participated in seminars following completion of each case, to resemble supervision during real patient encounters. Thematic analysis of fully transcribed interviews resulted in similar findings: students found SARI to be more authentic and engaging compared to VIC, and experienced that SARI enabled highly interactive communication possibilities and could express emotions, collectively offering a realistic experience. SARI facilitated active learning, hypothesis generation, and adaptive thinking to a greater extent than VIC by allowing students to phrase their own questions instead of choosing between predetermined options for gathering medical history. Prior to VP interactions, students receive structured training in medical history-taking, covering both generic aspects (applicable across medical specialties) and rheumatology-specific components. This preparatory training ensures students have foundational knowledge before engaging with the VP cases. Students typically work in pairs or small groups of three during VP interactions, with one student taking the lead role in each patient encounter. This collaborative approach is consistent with evidence supporting peer learning in VP-based clinical reasoning training. The leading student conducts the primary interaction with the VP, while other group members observe and may contribute to subsequent discussion and diagnostic formulation. The Social AI-enhanced Robotic Interface (SARI): SARI represents a novel VP modality that integrates social robotics with large language model (LLM) artificial intelligence. The platform combines the Furhat social robot (developed at KTH Royal Institute of Technology, Stockholm) with GPT-4o-mini language models from OpenAI. The Furhat robot consists of a physical robotic head mounted on a stand, featuring a back-projected animated face that displays facial expressions and lip movements synchronized with speech. The robot includes a microphone array for speech recognition and speakers for audio output. The physical presence of the robot in the room creates what researchers term "situational interaction," wherein the robot occupies actual physical space rather than existing solely as a screen-based interface. The VP dialogue system operates through the following technical workflow: 1. Speech Recognition: Student speech is captured through the robot's microphone array and converted to text using automatic speech recognition (ASR) technology. 2. Dialogue Processing: The transcribed student input is combined with: * A detailed patient case description (including medical history, symptoms, examination findings, and personality characteristics) * The last 10 dialogue turns (to maintain conversational context while avoiding excessive token usage) * System instructions that prompt the LLM to respond from the patient's perspective rather than as a general assistant 3. Response Generation: The LLM generates contextually appropriate responses based on the patient case and conversation history. 4. Emotion Modeling: At specific anchor points during conversation, the LLM also generates instructions for appropriate facial expressions (e.g., concerned, relieved, anxious, confused) based on conversational context. The system selects from a predefined set of expressions available in the Furhat software. 5. Speech Synthesis: Generated text responses are converted to natural-sounding speech using text-to-speech (TTS) technology, with the animated face displaying synchronized lip movements and the selected emotional expression. 6. Transcription: All dialogue is automatically transcribed in real-time and stored securely for subsequent feedback generation and research analysis. Each VP case follows established principles for VP development, including: * Clear learning objectives aligned with curriculum requirements * Realistic patient presentations based on typical clinical scenarios * Appropriate level of complexity for sixth-semester medical students * Integration of both common conditions and diagnostically challenging presentations * Inclusion of relevant psychosocial factors and patient concerns All VP cases are presented in English to accommodate international exchange students participating in the program. Standard laboratory test panels with case-specific results are provided alongside each case. The VP cases represent diverse rheumatological conditions including inflammatory arthritides, connective tissue diseases, and musculoskeletal disorders. Each case requires students to conduct comprehensive medical history-taking, including symptom characterization, functional assessment, medication history, and relevant review of systems. The feedback system used as the intervention in this study employs a sophisticated two-stage algorithm developed iteratively in collaboration with consultant rheumatologists: Stage 1: Assessment Model The first stage analyzes the complete student-VP dialogue transcript using a predefined assessment rubric. This rubric evaluates multiple dimensions of clinical performance: Communication Skills: * Opening the consultation appropriately * Building rapport with the patient * Using clear, jargon-free language * Demonstrating active listening * Showing empathy and addressing patient concerns * Closing the consultation appropriately Generic Medical History-Taking: * Past medical history exploration * Current medications and allergies * Family history of relevant conditions * Social history (occupation, living situation, support systems) * Lifestyle factors (smoking, alcohol, exercise) * Review of systems Rheumatology-Specific History-Taking: * Systematic joint symptom assessment * Pain characterization (onset, location, duration, quality, radiation, aggravating/relieving factors) * Functional impact assessment (activities of daily living, work capacity) * Morning stiffness duration and pattern * Extra-articular manifestations * Pattern recognition for inflammatory versus mechanical symptoms Clinical Reasoning Components: * Systematic approach to information gathering * Recognition of symptom patterns and clinical significance * Appropriate breadth of differential diagnosis consideration * Logical progression through history-taking The assessment model uses the LLM to evaluate each dialogue against this rubric, generating structured assessments of strengths and areas for improvement across these domains. Stage 2: Feedback Generation The second stage transforms the structured assessment from Stage 1 into readable, constructive written feedback. The feedback generation uses carefully crafted prompts that instruct the LLM to: * Present feedback in a supportive, educational tone * Provide specific examples from the student's dialogue * Acknowledge strengths before addressing areas for improvement * Offer concrete suggestions for enhancement * Focus on learning rather than evaluation Feedback Content and Scope Students receive approximately one page (300-500 words) of structured written feedback immediately following each of five designated VP encounters. The feedback focuses specifically on medical history-taking skills and does not address diagnostic accuracy or management decisions, as these aspects are covered in subsequent seminars with consultant rheumatologists. The control group completes identical VP cases using the same SARI platform but does not receive AI-generated feedback. However, both groups receive equivalent exposure to the social robotic VP technology, participate in identical case-specific seminars after case completion with consultant rheumatologists, and have access to dialogue transcripts for self-review. The only systematic difference is the presence or absence of AI-generated feedback. Assessment is done using an objective structured clinical examination (OSCE)-based structure with standardized patients (SPs) following interaction with all available VPs in SARI. SPs receive written case information describing a patient presentation, symptom history, and relevant background. The SPs also receive training in consistent responses to common question formats, and instructions on appropriate emotional responses, non-verbal communication, and handling unexpected questions. The OSCE rubric used in this study was developed collaboratively with experienced consultant rheumatologists based on Entrustable Professional Activity (EPA) frameworks specific to medical history-taking competencies. The rubric underwent pilot testing and refinement to ensure inter-rater reliability. The assessment evaluates five distinct domains with specified point allocations (totaling 10 points): * Communication at consultation start (0-3 points): 30% of total score * Generic medical history (0-3.5 points): 35% of total score * Targeted medical history (0-1.5 points): 15% of total score * Diagnostics and management reasoning (0-1 point): 10% of total score * Communication at consultation end (0-1 point): 10% of total score A score of 6 points corresponds to a pass rate on the OSCE based examination. However, students receive information that this study had no examinational purposes and did not affect their grades in any way. Two consultant rheumatologist conducted all OSCE assessments, blinded to intervention or control arm allocation. The assessors were provided only with the assessment rubric and had no knowledge of which students had received AI-generated feedback. This blinding was maintained throughout the data collection period.
Study Type
INTERVENTIONAL
Allocation
NON_RANDOMIZED
Purpose
OTHER
Masking
DOUBLE
Enrollment
115
A feedback algorithm which follows a two-stage design was implemented to generate post consultation feedback using large language models (LLMs) from OpenAI. The first stage of the feedback algorithm is an assessment model that evaluates student-VP dialogues using a predefined rubric developed in collaboration with consultant rheumatologists. This assessment model was iteratievly refined and validated prior to the study. The second stage of the feedback algorithm involves generating a feedback output based on the stageone assessment Students received approximately one page of structured written feedback immediately after completing each VP encounter with SARI. This feedback focused on medical history-taking within the context of rheumatology and included constructive comments with examples covering general history-taking, specific symptom enquiries, and systematic assessment of the VPs.
Karolinska University Hospital
Stockholm, Solna, Sweden
Total OSCE score
Immediately after completing the virtual outpatient clinic, participating students underwent an OSCE-like evaluation designed to assess performance in medical history-taking and clinical reasoning. The eight-minute evaluation involved interaction with a standardised patient. Performance was assessed by a consultant rheumatologist using a validated rubric, with the assessor blinded to whether students had received feedback. The assessment used a ten-point scale across five domains: communication at consultation start (0-3 points), generic medical history (0-3.5 points), targeted medical history (0-1.5 points), diagnostics and management reasoning (0-1 point), and communication at consultation end (0-1 point).
Time frame: Enrollment and evaluation during the same week during a period of 1.5 working days. Evaluation was done following completion of all virtual patient cases at the end of Day 2 of the Virtual Outpatient Clinic.
Pass rate comparisons
The OSCE based examination (0-10 points) has a cutoff at 6 points that corresponds to a pass score of the examination. The proportion of students that achieve a pass score will be compared between the intervention and control arm.
Time frame: Enrollment and evaluation during the same week during a period of 1.5 working days. Evaluation was done following completion of all virtual patient cases at the end of Day 2 of the Virtual Outpatient Clinic.
Individual OSCE domain comparisons
The OSCE assessment used a ten-point scale across five domains: communication at consultation start (0-3 points), generic medical history (0-3.5 points), targeted medical history (0-1.5 points), diagnostics and management reasoning (0-1 point), and communication at consultation end (0-1 point). Scoring within these domains will be compared between the intervention and control arm.
Time frame: Enrollment and evaluation during the same week during a period of 1.5 working days. Evaluation was done following completion of all virtual patient cases at the end of Day 2 of the Virtual Outpatient Clinic.
This platform is for informational purposes only and does not constitute medical advice. Always consult a qualified healthcare professional.