Skip to content

LLM Assessment Strategies#

See also: assessment

Place to think out loud and develop ideas. Goal being to be

helping to produce assessment design that is intellectually robust, manageable, and resistant to AI technologies.

What to do about assessments?#

What to do about assessments if we can't out-design or out-run AI? - points to more detailed pdf

  • help teachers ensure assessment changes have longevity
  • provide opportuntiies to reflect on human side of L&T
  • help students prepare for AI-augmented careers
  • encourage ethical, accountable and transparent use of AI

Ten myths#

Liu (2023) points out that some common approaches no longer apply

  • LLMs can access recent references (post 2021) - not true.
  • Can't get references right - not true at least for specific tools.
  • Can't write reflectively - Yes they can outperform students (Li et al 2023)
  • Can't do calculations or use contemporary class resources - different LLMs can - e.g. Finance
  • Authentic assessment is AI proof - no

Resources to examine#

General good advice#

  • Do students understand why/how this assessment task fits more broadly with their learning, careers etc.

LLM specific#

Michael Sankey presentation, includes some suggestions

  • Teaching students how to use generative LLMs
  • Enabling students to become creators with LLMs
  • provide feedback on their writing
  • simplify complex information
  • scaffold information on specific task
  • helpful for diverse students (neurodivergent/NESB)
  • In essence, create their own learning resources
  • Work around LLMs
  • Balance essays with other kinds of assessments
  • Design assignments where students demonstrate understanding independent of written works
  • "authentic assessments"
  • personalised or complex tasks

GU Workshop on Gen AI assessment#

Rene Hexel

  • Giving a longer history AI work as a sequence of peaks and troughs
  • Overview of LLM operations
  • Moving onto TEQSA discussion paper
  • Onto Griffith actions - mostly institutional
  • Apparently an L&T CoP??? - L&T site
  • Applications of ai

  • students and educators - powerpoint sample

  • Cogniti - Danny Liu's work on AI for Oz educators

References#

Li, Y., Sha, L., Yan, L., Lin, J., Raković, M., Galbraith, K., Lyons, K., Gašević, D., & Chen, G. (2023). Can large language models write reflectively. Computers and Education: Artificial Intelligence, 4, 100140. https://doi.org/10.1016/j.caeai.2023.100140

Liu, D. (2023). Ten myths about generative AI in education that are holding us back. Linkedin. https://www.linkedin.com/pulse/ten-myths-generative-ai-education-holding-us-back-danny-liu