1. Introduction

Having worked through the experiencing and examining sections, you started thinking about the various different approaches and elements of evaluation. The aim now is to help organise your thinking into a more formal and structured conceptualisation of evaluation. To provide you with a firm foundation upon which to plan and implement the valuation of your teaching practice.

The field of evaluation is diverse and deep, depending on the lens that you are using to view it. In this course, we concentrate on evaluation for learning and teaching contexts, particularly self-evaluation, peer evaluation, and student evaluation. However, in order to understand these areas, it is important to understand the goal of evaluation and the types of evidence in more detail.

This book aims to help you answer the following questions

• What is evaluation?

• What are the goals of evaluation?

• What types of evaluation are there?

• What is evidence?

2. What is evaluation?

While it is a commonly used word, ‘evaluation’ should be defined so that we have a mutual understanding of it.

Simply put, evaluation is the determination or fixing the value of something, or the determination of the significance, worth, or condition of something usually by careful appraisal or study (Merriam-Webster, 2017). In the context of learning and teaching, evaluation is more complex and includes notions of evidence, data, methods, and processes.

These components of evaluation should be clarified for higher education contexts in order to develop a common foundation from which to explore the different types of evaluation that can be undertaken in your learning and teaching context.

3. What are the goals of evaluation?

Berriet-Solliec, Labarthe, and Laurent (2014) describe three distinct goals of evaluation:

1. to measure (designed to assess the effects of a program),

2. to understand (designed to identify and analyse the mechanisms through which the program under evaluation can produce the expected outcomes or create diverse effects), and

3. to learn (designed as a collective learning process).

These three goals align with the common goals of evaluating teaching practice. That is, when we are evaluating our teaching practice, we are attempting to measure, understand, and/or learn about our teaching and its effects on student learning. The following are three examples of these goals in practice in higher education contexts.

• To measure.
  A teacher wants to measure the change in student understanding as a result of undertaking their course. They could design a questionnaire to measure student understanding of the content at the beginning, middle, and end of the course. Then, they would compare the results and determine if there are changes in student understanding of the course content. In this way, the evaluation of the student learning (i.e. the three questionnaires) are designed to assess the effects (or effectiveness) of the course.

• To understand.
  A teacher wants to understand if a specific technological tool can support greater student engagement in their course. So, they design a learning experience that can be undertaken in two ways - one that uses the tool and one that does not use the tool. The teacher then evaluates the level of student engagement in the activity as a way to understand if and how the tool can support this.

• To learn.
  A teacher wants to learn more about how students perceive their course, so they include course-specific questions (i.e. more than the core questions) in their student evaluation of course and teaching experience questionnaire that the institution administers at the end of the teaching period.

4. What types of evaluation are there?

In higher education, evaluations typically fall into three broad categories: self evaluation, peer evaluation, and student evaluation. Each of these types of evaluation have specific purposes and can be used in your learning and teaching context in order to more fully measure, understand, and learn about your teaching.

4.1. Self evaluation

Self evaluation is as it sounds - evaluating yourself. However, it’s not as easy as it sounds, either. That is, you need to consider a lot more than simple reflection of your teaching practice, such as different ways to understand and observe your teaching practice. Self evaluation can take many forms and involve a range of resources to help inform you. For example, you could refer to the AQF to help evaluate whether your course content and teaching meets the expectations for your AQF level. Your own professional learning goals can also form part of your self evaluation.

As a quick guide to self evaluation, The General Teaching Council for Scotland provides the following summary of self evaluation.

Self-evaluation should be a useful process that is rigorous and enables you to be critically reflective about yourself as a professional and your practice.

Self-evaluation should support you to:

• Reflect on what you have done

• Think about what you might do next

• Consider your own progress and development

• Deeply understand your professional practice, your professional learning and the impact of this on your thinking, professional actions, those you work with/support and the pupils and their learning

Self-evaluation will involve:

• Asking deep and searching questions about self and practice

• Using the GTCS Standards to inform and guide your reflections

• Using other influencing factors such as school or dept improvement plan; other standards or targets; issues relevant to your particular context

• Using your ongoing reflections and enquiry into practice

• Considering the needs of learners/colleagues in your context

• Using evidence from a range of sources to inform and support your self-evaluation

The self-evaluation process will enable you to:

• Plan for meaningful professional learning

• Engage in critically reflective dialogue as part of the PRD process

• Identify and focus on areas you wish to develop expertise or accomplishment

• Consider your career planning

For more information on how to conduct a self-evaluation of your teaching, you can also view the Ryerson Learning & Teaching Office’s guide to self evaluation or Warwick University’s Evaluate Your Own Teaching site.

4.2. Peer evaluation

Peer evaluation is the process of collegial feedback on the quality of your teaching. It is a purposeful process of gathering information and evidence about the effectiveness of your teaching processes and environments with a view to obtaining constructive feedback. You can approach a peer (colleague, industry professional, educational designer, etc.) to evaluate your teaching. This evaluation can occur in whichever manner you are teaching - it is not limited to an observation of your class. It can include evaluation of your online learning and teaching environment and activities, content, assessment items, student feedback, and so on. The typical purpose of peer evaluation is for either formative (for development) or summative (for personnel decision) purposes.

Flinders University provides some clear guidelines on the components of peer evaluation. These are described here.

Peers are a useful source of information about the whole learning and teaching cycle from curriculum design and development, through planning and preparation, and learning and teaching interactions, to critical review to improve and enhance teaching.

Peers are able to comment on:

• philosophy and approach to teaching

• presentation skills

• group work and facilitation skills

• course material and resources

• assessment methods

• curriculum development

• aims, objectives and content

• identifying opportunities for professional development

Peers are able to provide feedback on the elements of teaching and the relationships and alignment between them, before, during and after learning and teaching activities. Peer evaluation can be used in the following ways to enhance teaching practice:

1. gaining feedback on teaching from a colleague which adds another dimension to feedback collected from students and through self-reflection.

2. as a mechanism for collecting evidence to support both promotion and teaching awards applications.

3. as an input to discussion and reflection on:

1. teaching strategies

2. assignments

3. learning outcomes

4. assessment methods

5. monitoring of student learning

4. class and/or syllabus review

5. addressing problems in a topic

6. discussion and reflection on new ideas on teaching.

Peer evaluation is not:

• ‘novice learning from expert' - it should be collegial and directed towards honing skills not cloning skills

• for training people in a particular way of teaching.

For more information on how to conduct a peer-evaluation of your teaching, you can also view the Ryerson Learning & Teaching Office’s guide to peer evaluation or Flinders University’s Resources for Peer Evaluation of Teaching site.

4.3. Student evaluation

Student feedback and evaluation are rich sources of information about the delivered learning and teaching experiences. This information can be used formatively and summatively, and it is often used as evidence for staff promotion, probation and awards, and for internal and external quality assurance reporting requirements. Student evaluations are regularly sought at the institutional level through student evaluations of teaching (also known as course experience surveys, student evaluation of course and teaching, student feedback, etc.). At USQ, the institutional level student feedback survey is called MyOpinion and it is a set of core questions with the teacher having the option to add an additional customised questions from the approved question bank.

Student evaluations and feedback are used to ensure a student-centred approach whereby students are engaged and the student feedback informs course development and quality enhancement of courses. Student feedback also has the potential to provide valuable insights to the teacher through providing understanding of how the learning has been experienced. This feedback can augment self and peer evaluations to provide a richer and deeper evaluation of a learning episode, environment, event, and so on.

As Macquarie University’s Teaching Evaluation site states, student evaluation is important for:

• ensuring that students enjoy high quality learning experience,

• your continual improvement as a teacher,

• the continual improvement of the course and program of study that is being offered, and

• ensuring that the University is achieving the desired standard of quality in students' learning experiences.

Types of student evaluation sources of evidence

The Australian Awards for University Teaching (AAUT) divide student evaluation sources of evidence into two types: student reactions and student learning.

Student reactions focus on what students tell you about your teaching and their learning. The types of evidence can include

• student evaluations of teaching,

• student interviews and focus groups,

• informal class feedback,

• formal surveys and questionnaires,

• unsolicited student feedback,

• student logs and journals, and

• online feedback.

Student learning focuses on the indicators of student learning that attest to the success of your teaching strategies. The types of evidence can include

• students’ self-reported knowledge/skills gained,

• rates of attrition, failure, progression to honours/postgraduate,

• grade distribution,

• evaluation of generic skills/outcomes/attributes,

• student work - assessment, theses, projects,

• employer/workplace feedback, and

• approaches to study questionnaires, etc.

4.4. What types of evaluation for you?

5. What is "evidence"?

Evidence is the myriad of “things” that constitute information or demonstrate the object of the evaluation. These “things” can be quantitative or qualitative in nature.

For example, if the effectiveness of a traditional lecture is the goal of evaluation, then the evidence could include number of students attending, the presentation materials, a recording of the lecture, the number of questions asked during the lecture, the responses to set tasks (e.g. in the lecture, the teacher directs students to a forum - do they participate in the forum?), and so on.

Because there are so many types of evidence that can be collected, it is important to understand the goal of the evaluation in order to determine which types of evidence will best support the evaluation.

The following table (adapted from the "Evidence checklist for teaching awards", provides some idea of the different types of sources of evidence.

5.1. Types of evidence

In order to achieve the goal of your evaluation, you need to collect appropriate evidence that will support your understanding of the evaluation. In Berriet-Solliec et al.’s (2014) article, they identify three types of evidence, as follows:

• evidence of presence;
  Evidence that demonstrates that a phenomenon or object exists and is present, the most straightforward type of evidence.

• evidence of difference-making; and
  Sometimes known as evidence of effectiveness, refers to evidence that demonstrates impact (or lack thereof) of a phenomenon.

• evidence of a mechanism for a phenomenon.
  Refers to the evidence that confirms a relationship or phenomenon, e.g. confirming that a specific tool does increase student engagement.

5.2. What about data?

Data is a type of evidence, and one that can be more easily processed. For example, if the effectiveness of traditional lectures is the goal of the evaluation, the number of students attending the lecture is a type of data that can be collected and easily processed in terms of increased/decreased attendance over a period of time. Note that the number of students attending is also evidence, so there can be an overlap between data and evidence. Data can be classified as qualitative or quantitative in nature, and it can come in a variety of forms from a variety of sources, as some were identified in section 1.2. 

For some tertiary educators quantitative and qualitative data will be old friends. If you're uncertain about the differences between qualitative and quantitative data then resources such as this page (the comparison chart might be a good place to start) as might YouTube videos (e.g. this one)

6. What are methods and processes

Methods refers to the way in which you choose to evaluate your learning and teaching. There are a number of different options for approaching the evaluation of teaching and learning, and they primarily depend on the type of question or evaluation that you are undertaking. The evaluation methods are useful for understanding how you will collect data, analyse data, report data, and make decisions about the evaluation.

There are structured ways in which you can use the methods to undertake evaluations; they are sometimes also referred to as approaches. While there are some common processes that can be followed like a recipe, you can also define your own process that will better suit your context. In many cases, it can be good to start with a common process and then adapt it to your specific context considering your question, data, and so on.

There are various methods that you can use to collect evaluative data. In general, you can collect data from individuals and groups, and through observations, physical measurements, and reviewing existing records and data. Some options for collecting data in these ways can be found on BetterEvaluation.org, but keep in mind that some of these methods might not be appropriate to learning and teaching contexts. There is also a data collection matrix presented in Table 11.1 in Johnson & Turner’s (2003) chapter in the Handbook of Mixed Methods in SOcial & Behavioral Research; it could be worthwhile reading pages 297-299 if this chapter for a deeper explanation of how different data collection methods can align with qualitative and quantitative methods, or how they can be mixed. Combining the content in the two previous links with that provided in the Ryerson resource on self-evaluation, you can begin to see how different data collection methods can be used to obtain data and evidence for your evaluation. In particular, have a look at how Ryerson have presented different self-evaluation methods with different data collection methods - you could consider using some of these in your evaluation.

7. What are the criticisms and challenges of evaluation?

 By now, you should have an understanding of the components that make up an evaluation and the complex interplay between these components. When designed well with appropriate selection of tools (e.g. the evidence aligns with the question asked, which aligns with the purpose of the evaluation, etc.), evaluation can provide critical insight into a specific area under investigation and illuminate areas of competence and need.

However, it is important to be able to interpret the data and evidence gathered in context, rather than just as numbers or discrete artefacts. In an attempt to better understand this interpretation, please read this article by Franklin (2001).  As you read this article, reflect on the following questions (perhaps write a blog post):

• Considering your answers to the questions in the previous activities in this book, what are three critical contextual factors that need to be understood in order to interpret your proposed evaluation?

• What are two concerns that you must address regarding the validity and reliability of your evidence/data?

• What would be an acceptable response rate for your context, if you are using student data/student responses?

• What are the potential pitfalls of your broad conceptualisation of the evaluation? How could you potentially mitigate these?

• Are there any other considerations that you should make in order to make your evaluation more robust and useful?

1. Where are you?

Throughout this week, you have been learning about the components of evaluation in general and more specifically for learning and teaching evaluation. Now, it's time to think back to Assignment 1, Part B, and see how this information on evaluation can be adapted to your chosen teaching episode.

To refresh your memory, Part B from Assignment 1 asks you to think of a learning and teaching episode from your recent past, and design an evaluation plan for that episode. At this point, you should have a clearer understanding about evaluation, what it is, how it can be done, evidence, types of evidence, data, and data collection. So, it is time to apply it to your context.

2. Which episode?

3. Reflection