Monday, May 16, 2016

Why is it so Hard to Change our Assessment Practices?

My forte is talking about character education, behavioural challenges, student issues when it comes to addressing personal struggles that are interfering with learning, and integrating technology into the classroom effectively and meaningfully. Often as a young teacher, when I heard we were going to talk about assessment I felt like the person on the left. I'm by no means an assessment expert, although I have read more than my fair share of assessment experts' books, blogs, watched tons of videos, and attended conferences with Rick Wormeli, Doug Reeves, and Dylan Wiliam, and I understand and now practice a Standards Based or Outcomes Based philosophical approach. Before coming to Alberta last year, I thought I was just an ordinary educator from Saskatchewan, who was using and implementing outcomes-based assessment in the classroom largely because I thought most teachers were doing the same everywhere across Western Canada. However, I've come to realise over the past year and a half, that is not the case at all. Most teachers are very good at teaching to the outcomes. They know their material, they teach the concepts, students are learning, and we are seeing success. So it isn't so much in the teaching that we have our challenges. But aligning assessment practices to teaching outcomes is not happening in a consistent fashion across schools, divisions, and the province. I had my struggles with implementing outcomes-based assessment at first, but I made the change because it was the right and fair thing to students. 

Approximately eight years ago, I was put to the test about my assessment philosophy. I say philosophy because I was a district administrator at the time, and not directly in a K-12 classroom. But if I wanted to be an instructional leader, I needed to know what we expected from our teachers. I had just finished my Master's which had done little to redirect my pedagogical philosophy towards Outcomes-Based Assessment (OBA). The program was very traditional in its focus. It wasn't until I was quizzed, or rather challenged by a friend about my assessment views that I realised I needed to make the shift. He kept saying things like, "if we are teaching to outcomes, shouldn't we be only assessing the outcomes?" That was an important question because both of us were watching a lot of teachers assess the end product and thinking they were assessing the outcome.  For example, if a cover page for an essay or project was not one of the outcomes, then why is it being graded and included in the overall mark. Or if participation is not a listed outcome, why are teachers marking participation. That was the first challenge to my thinking about assessment. In addition, I think we really need to challenge the practice of including homework for grades, especially if that is not one of the outcomes.

The next challenge to my thinking was around how we penalise students for not knowing the outcomes at the time of instruction. If a student is assessed on one outcome in September and demonstrates only a basic understanding of it then, typically that was the grade they got at the time of testing.  But in November, I was testing another outcome that clearly required students to demonstrate the prior outcome in order to show they understood the new outcome, shouldn't we be going back and changing the grade of the first outcome to reflect that the student has a better understanding of a previous outcome? I really struggled with this, because I was trained that whatever a student got for a grade was what they carried with them into the final calculating of their grade. I was looking at assessment as defined by time, but if I really embraced lifelong learning, then why would I penalise students because they hadn't mastered the outcome when I taught it, only to demonstrate that they learned it later.

It took months of coaching, but I changed. Hence writing this blog. Maybe my story can be of help to others struggling with making the shift. So I would like to outline some of the challenges I see many teachers getting stuck with and preventing them from making the changes.

Take a few minutes to watch Doug Reeves video where he talks about changing our assessment practices to being more effective.

Although I didn't have the luxury to watch his video before I changed my assessment practices, he captures the essence of my journey. But let me highlight a few of the big changes I made.

1. Don't combine multiple outcomes into one grading category

The traditional breakdown in high school for course outlines usually includes a grading policy. I was trained to write out exactly how the students were to be graded so they knew exactly how I was coming up with their grade. That meant that I clumped all kinds of summative assessment together under one category. Usually, it looked something like this for an English course:

Written Assignments - 30%
Tests/Quizzes - 20%
Major Essay - 20%
Final Exam - 30%

Looks about right, doesn't it? To prove my point, I Googled course outlines and found hundreds of similar examples that are still being used like this today.

What I found was teachers working with categories of activities where they did exactly what I was doing years ago.

2. Don't test more than one general outcome per summative assessment.

Why? There are a few problems with this approach. Let's first of all assume that there are eight general outcomes to be covered in this course. If we are teaching to outcomes, there needs to be one summative assessment per general outcome. Summative doesn't mean only tests or exams, but it can be an essay, a project, or a quiz. Essentially, it's anything that determines what a student can demonstrate with varying degrees of understanding at the end of the learning outcome. The curriculum may have three or five specific outcomes or concepts under each general outcome, but for reporting purposes, there needs to be only one summative assessment for the general outcome. I need to use a significant number of formative assessments for each of the specific outcomes or concepts that guide my instruction to ensure that students understand the general outcome before that summative assessment happens. When we add more than one outcome into the summative process, we lose the efficacy of  the assessment process.  It needs to be clear to the student to what degree they have achieved the outcome. If we add other outcomes to our assessment, it has a negative impact on validity. Validity is how accurately a conclusion, measurement or concept corresponds to what is being tested.  How do we really know if a student has achieved mastery or not on an outcome if it's mixed with other outcomes? 

Take Saskatchewan ELA 30A Outcomes for example; there are 10 total general outcomes, therefore, we should only have one summative assessment per outcome. Like I said, I may use lots of formative assessments with the different novels, poems, short stories, non-fiction literature that I use in the classroom. But I don't grade the students' understanding of the literature, but their understanding of the outcomes. The literature is only the medium I'm using to demonstrate the students can show they have met the outcome. So when I'm teaching the first outcome: 

CR A 30.1 View, listen to, read, comprehend, and respond to a variety of grade-appropriate First Nations, Métis, Saskatchewan, and Canadian texts that address:
  • identity (e.g., Define the Individual, Negotiate the Community)
  • social responsibility (e.g., Shift Centres, Blur Margins), and
  • social action (agency) (e.g., Understand Beliefs, Initiate Action).
I may use multiple forms of literature to help students understand identity, social responsibility, and social action as it relates to First Nations, Metis, and Saskatchewan. But my focus needs to be how well students understand identity, social responsibility and social action, and not the literature itself. I don't need to have a chapter test or quiz on everything either. 

3. Don't average grades to come up with a final grade.

Then there is the issue of combining the grades over two or three outcomes and then averaging them. Rick Wormeli says,"Just because something is mathematically easy to calculate doesn’t mean it’s pedagogically sound." Averaging is bad pedagogy. So take the above grading policy illustration, class work is 25% of the final grade. But every outcome has been combined into one category and averaged out for 25% of the grade. In truth, a percentage also tells us very little about what a student knows or doesn't know. Wormeli adds, "we claim to be standards- (outcomes-) based. This means that assessments and grading are evidentiary, criterion-referenced." When we average the scores, it is no longer criterion-referenced.

I like what Todd Rose said about averaging, "The Myth of Average is a belief that’s been prominent in most sciences and in education. It’s the belief that we can use statistical averages to understand individuals. Scientists have come to realise that it’s a myth, and over the last 10 years have been moving from averages to individuals, so for example, we’re hearing a lot of things like ‘personalized medicine.’  Unfortunately, education has not quite realised the myth yet." Transfer that principle from the individual person to the individual outcome, once you average the outcomes, we lose sight of what the student knows about an outcome. It is simply wrong to average the outcomes to determine the score for the student. If we are teaching to the outcome, then we need to assess the outcome.

I'm a visual person and I need visuals to help illustrate for me. Look at this picture of a person on the right. It is multiple exposures over each other. What you are seeing is an average of all the pictures taken together. How clear is the picture? When we average our student's grades and combine everything into one picture or snapshot to give the students or pictures, this is exactly what we are giving them. It's not acceptable.

My preference is to use a 4 point system (Marzano) scale, where I list the outcomes using a bar graph color coded such as illustrated below. The summative results are used to determine the mark. If there is not enough summative information to determine levels of understanding, then formative results are taken into consideration.

Carey Lehner example

The general outcomes are listed on the left and the bar graph illustrates where the individual student is at for each outcome. We had a policy in Saskatchewan that "if it wasn't a 2, it was a redo." Teachers emphasised to students on getting 2 or higher. In order to move forward into the next grade students needed to get a minimum of 2 in 75% of the outcomes. If they didn't, that's when credit recovery kicked in requiring the students to only have to redo the outcomes they had lower than a 2. Credit recovery recognises all prior student learning and assessments and does not penalise students for where they were, but acknowledges where they were now. During the course, we encouraged our teachers to go back and reassess student understanding when they demonstrated their understanding of an outcome had grown from a previous assessment.

If it is necessary by the Ministry of Education or for scholarship purposes to have a percentage, Saskatchewan Rivers School Division developed  this modified Marzano conversion scale.

Personally, I don't think it's necessary to convert from the 4 point to the percentage, but some folks need that still. My question is, "How accurate is the percentage system anyways?" It's all fairly subjective in the end. I lean more towards the simple system that gives me a better picture of the learning, rather trying to tell a student they have an 85% understanding. How do I really know they have an 85% really?

If you think for a second that the change has been easy, you’re wrong. I went to school and was graded one way. I went to college and university and was trained to grade the same way. It was when I really looked more deeply into this issue and realised the error of my ways. I made the switch, I had to make the switch. I didn’t switch for me, I did it for the students, because it was the right thing to do. I hope you see it too.
Post a Comment