Retaining Artificial Intelligence Literacy: Evidence from a Project-Based Undergraduate Course
This paper examines whether a short project-based Artificial Intelligence (AI) literacy course is associated with larger gains in applying AI concepts, regulating AI-supported problem solving, and reasoning ethically among first-year undergraduates, and whether any advantages remain detectable at a four-week short-delay follow-up.
Artificial Intelligence literacy provision is expanding in higher education, but many short interventions are evaluated using single-group designs and only immediate posttests. Comparative evidence on short-delay retention in non-Computer Science (non-CS) or non-specialist cohorts remains limited, and existing evidence has not sufficiently separated applied concepts, strategy regulation, and ethical awareness as distinct literacy strands.
A quasi-experimental mixed-methods design compared an intervention group (n = 180) with a contemporaneous comparison group (n = 180) from the same university and study stage. Both groups completed measures at pretest, immediate posttest, and approximately four weeks after course completion. The quantitative strand was integrated with 155 reflective essays and focus-group interviews with 19 students from the intervention cohort.
The study contributes strand-specific evidence on the short-delay retention profile of a compact project-based AI literacy course. Its contribution is not simply that an AI literacy course was associated with improvement, but that applied concept use, strategy regulation, and ethical awareness exhibited distinct patterns of persistence following the same intervention. This distinction extends prior AI literacy intervention work by separating immediate gains from short-delay retention across literacy dimensions.
The intervention group showed larger differential gains in applied AI concepts and strategy regulation at both posttest and four-week follow-up. Ethical awareness showed a statistically significant immediate comparative advantage, but the follow-up contrast was smaller, included zero in its confidence interval, and was not statistically reliable.
Short AI literacy courses should embed concepts into project decisions and provide students with repeated opportunities to plan, monitor, revise, and justify AI-supported work. Ethical reasoning should be revisited after course completion through subsequent cases, critique tasks, or discipline-specific applications, rather than being treated as a one-time element.
Future studies should use comparison groups, delayed follow-ups, artifact-based evidence, and behavioral measures to test which course components sustain different strands of AI literacy across time.
The findings indicate that compact, project-based AI literacy courses are associated with applied conceptual gains and greater self-reported strategic awareness among non-specialist undergraduates, while also showing that responsible AI use is less likely to consolidate without continued reinforcement.
Further research should test longer follow-up intervals, randomized or matched designs, component-level reinforcement models, behavioral measures of regulation, and transfer across institutions, disciplines, and student populations.
Back