Impact of Generative AI on University Students' Learning Processes and Critical Thinking

Overview

This study is a randomized, controlled trial with a pre-test/post-test design, conducted to determine the effects of Generative AI (GenAI)-supported learning on academic achievement, learning efficiency, and critical thinking within a university setting. The study population consists of students from the Dialysis Program at Bingöl University's Vocational School of Health Services; the sample comprises 100 volunteer students who met the inclusion criteria and were randomly assigned to either an experimental group (n=50) or a control group (n=50). The experimental group will participate in a "Generative AI-Supported Learning Program" over the course of one month-meeting two days a week for a total of eight 60-minute sessions-covering topics ranging from effective prompting techniques to case analysis and academic ethics; meanwhile, the control group will continue with their routine educational activities. Data will be collected using a Demographic Information Form, the Academic Achievement Inventory, the Critical Thinking Disposition Scale, and the Artificial Intelligence Dependency Scale, and will be analyzed using biostatistical methods (t-tests, ANCOVA, and effect size calculations).

Study Design and Methodology Type of Study: This study will be conducted as a randomized experimental design with a pretest-posttest control group to determine the impact of generative artificial intelligence-assisted learning on university students' academic achievement, learning efficiency, and critical thinking skills. Setting and Time of Study: The study will be carried out with students enrolled in the Dialysis Program at Bingöl University Vocational School of Health Services. It is planned to be completed within approximately one month following the receipt of institutional permissions and approval from the Ethics Committee. Population and Sample: The population of the study will consist of students studying in the Dialysis Program at Bingöl University Vocational School of Health Services. The sample will comprise a total of 100 students who voluntarily agree to participate in the study. Participants will be divided into two equal groups-50 in the experimental group and 50 in the control group-using a computer-assisted simple random randomization method. Data Collection Tools: Demographic Information Form: Developed by the researchers in line with the literature, this form will be used to determine the sociodemographic and educational characteristics of the students. It will consist of approximately 10-15 items covering age, gender, grade level, cumulative grade point average (CGPA), daily internet usage time, generative AI usage status, frequency of use, purpose of use, and prior AI training. Academic Success Inventory for College Students (ASICS): Originally developed by Prevatt, Roberson, and Peterson (2011) as the Academic Success Inventory for College Students (ASICS) to evaluate university students' behaviors, efforts, and performance perceptions related to academic success. The Turkish adaptation, validity, and reliability study was conducted by Orçanlı, Bekmezci, and Boztoprak (2021). In the adaptation study, confirmatory factor analysis for the first-order multi-factor model demonstrated good fit indices . The overall Cronbach's alpha internal consistency coefficient was reported as 0.937, indicating high internal reliability. Higher scores on this Likert-type scale indicate higher levels of academic achievement. Disposition Towards Critical Thinking Scale: Based on the "Disposition Towards Critical Thinking" model developed by Facione and Facione, this widely recognized international tool was adapted into Turkish by Hülya Ertaş Kılıç and Ahmet İlhan Şen. In the Turkish validity and reliability study, the five-factor structure (Analysis, Open-Mindedness, Inquisitiveness, Systematization, Cognitive Maturity/Self-Confidence) showed good fit indices, and the overall Cronbach's alpha coefficient ranged between 0.85 and 0.95. Responses are recorded on a 5-point Likert scale (1 = "Strongly Disagree" to 5 = "Strongly Agree"), where higher overall scores indicate a greater disposition toward critical thinking. Dependence on Artificial Intelligence Scale: Developed globally by Morales-García et al. (2024) as the Dependence on Artificial Intelligence Scale to measure AI addiction behaviors among college students, its Turkish adaptation was performed by Buğra Çağatay Savaş (2024). Confirmatory factor analysis for the single-factor structure revealed excellent fit indices and an overall Cronbach's alpha internal consistency coefficient of 0.937. The Likert-type scale will be administered as a pretest and posttest in both groups to evaluate changes in students' AI dependence levels resulting from generative AI-assisted learning applications. Study Implementation After obtaining written informed consent from participating students, randomization into experimental and control groups will be performed. At baseline, both groups will complete:Demographic Information FormAcademic Achievement Assessment ToolsMotivated Strategies for Learning Questionnaire (MSLQ)California Critical Thinking Disposition Inventory (CCTDI) / Disposition Towards Critical Thinking Scale. The experimental group will undergo a Generative AI-Assisted Learning Program developed by the researchers. Duration: 1 month Frequency: 2 days per week (Total of 8 sessions) Session Length: Approximately 60 minutes each Intervention Materials: Educational booklets, instructor presentations, implementation guides, prompt engineering handbooks, worksheets, and infographics. Session Outline: Session 1: Introduction to Generative AI Technologies Session 2: Effective Prompt Engineering Techniques Session 3: AI-Assisted Study Methods Session 4: Content Summarization and Knowledge Organization Session 5: Question Generation and Exam Preparation with AI Session 6: Case Analysis and Clinical Problem-Solving Applications Session 7: Critical Evaluation of AI Outputs and Academic Integrity Session 8: Practical Synthesis, Overall Evaluation, and Feedback Students in the control group will continue their standard curriculum without receiving any educational intervention. Upon completing the one-month period, all posttests will be re-administered to both groups. Data Analysis:Data will be analyzed using IBM SPSS Statistics software with support from a biostatistics consultant. Initial procedures will include data cleaning, missing value checks, outlier analysis, data coding, and assessing suitability for parametric testing.Statistical Methods to be Applied:Descriptive Statistics: Mean, standard deviation, median, minimum-maximum, frequencies, and percentages.Normality \& Homogeneity Tests: Shapiro-Wilk test and Levene's test for equality of variances.Comparative Inferential Tests: Chi-square test, Independent Samples t-test, Paired Samples t-test, Mann-Whitney U test, and Wilcoxon Signed-Rank test.Advanced \& Covariate Analysis: Analysis of Covariance (ANCOVA) for repeated measures and multiple linear regression models if indicated.Effect Size \& Reliability: Cohen's d, Partial Eta Squared , and Cronbach's alpha internal consistency analysis.All statistical tables, figures, and reporting will strictly follow scientific publication standards. The threshold for statistical significance will be set at p \< 0.05.