This study aims to explore the feasibility of using medical large language models to assist in chronic disease management. By setting up experimental and control group interventions and having experts blindly evaluate anonymized cases, it compares different management approaches in chronic disease care, verifying the scientific basis, effectiveness, and potential for broader use of medical large language models in supporting chronic disease management.
Study Type
INTERVENTIONAL
Allocation
RANDOMIZED
Purpose
OTHER
Masking
SINGLE
Enrollment
20
The primary care physicians in the intervention group completed the management decisions for chronic disease cases, including disease assessment, examination suggestions, treatment plans, selection of management methods, and health education, with the assistance of the medical LLM.
The Affiliated Taizhou People's Hospital of Nanjing Medical University
Taizhou, Jiangsu, China
expert overall scores
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale on several aspects: accuracy of condition assessment, reasonableness of test recommendations, reasonableness of treatment plans, appropriateness of management choices, scientific nature of patient education content, feasibility at the primary care level, overall clinical quality. Experts were instructed to rank the seven dimensions from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.
Time frame: through study completion, an average of 3 months.
accuracy of condition assessment
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale on accuracy of condition assessment from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.
Time frame: through study completion, an average of 3 months.
reasonableness of test recommendations
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the reasonableness of test recommendations. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.
Time frame: through study completion, an average of 3 months.
reasonableness of treatment plans
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the reasonableness of treatment plans. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.
Time frame: through study completion, an average of 3 months.
appropriateness of management choices
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the ppropriateness of management choices. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.
Time frame: through study completion, an average of 3 months.
scientific nature of patient education content
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the scientific nature of patient education content. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.
Time frame: through study completion, an average of 3 months.
feasibility at the primary care level
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the feasibility at the primary care level. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.
Time frame: through study completion, an average of 3 months.
the overall clinical quality
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the overall clinical quality. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.
Time frame: through study completion, an average of 3 months.
risk advice
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. If there is no risk advice, the expert selects 'No'; otherwise, the expert selects 'Yes' and specifically lists the content of the risk advice.
Time frame: through study completion, an average of 3 months.
Any diagnostic or treatment information missing
A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. If there is no omission of diagnostic and treatment information, the expert selects "No"; otherwise, the expert selects "Yes" and specifically lists the omitted diagnostic and treatment content.
Time frame: through study completion, an average of 3 months.
feedback regarding the use of medical LLM by primary care physicians
After primary care physicians complete a case assessment assisted by medical LLM , they will fill out an evaluation of the medical LLM usage, including whether the medical LLM helped in disease assessment, assisted in formulating examination plans, aided in developing treatment plans, improved chronic disease management capabilities, supported chronic disease education, assisted in management approach selection, enhanced confidence, saved time, whether there were hallucinations or omissions, whether it provided risky suggestions, and any additional comments on the use of the medical LLM.
Time frame: through study completion, an average of 3 months.
This platform is for informational purposes only and does not constitute medical advice. Always consult a qualified healthcare professional.