In this article, the author, George Manthey, describes how he analyzed the cognitive rigor of all 4,638 of California’s content standards. Using a cognitive scale (based on the Revised Bloom’s Taxonomy of Anderson and Krathwohl), he was able to determine the level of higher-order thinking of each standard, based on a the scale’s 6 levels (see below), which he refers to as “areas” (6 being the highest). Why did he do this? He wanted to see whether or not the cognitive rigor of student work aligned to the cognitive rigor of the standards. Thus, he compared 7^th grade student math samples to their standards, in the area of rigor. Based on his findings, 50% of the 7^th grade math standards require higher-order thinking at area 4, but only 4.7% of the student work he examined required higher-order thinking at area 4.

With the belief that “mastery of academic standards requires an alignment of both content and cognition,” Manthey concludes that data (standards and student work) can provide valuable feedback for teachers so that they can determine whether or not they are using strategies to help students master the cognitive rigor of the standards. He states, “The effective examination of data, including a critique of the cognitive rigor of student work and learning objectives, can have a positive impact on learning and teaching.”

I’m not surprised by Manthey’s findings as I have seen similar results when analyzing student work. For example, I have analyzed high school assignments in which the student is asked to name elements in a story: “Who is the main character,” “Describe the plot,” “Where does the story take place?” Typically, these are questions given on a quiz or test. They involve very little higher-order thinking since they are recall questions. The goal of these questions is reading comprehension, and they align to California’s Literary Response and Analysis standards. However, these particular recall questions align to a 1^st grade standard: “Identify and describe the elements of plot, setting, and character(s) in a story…” (CA 1^st Reading 3.1). On Mathey’s scale, the rigor of these questions aligns to area 1: “Remember or Understand Facts and/or Concepts.” Thus, when high schools students complete these questions, they are mastering a 1^st grade standard with the higher-order thinking at the lowest level on Manthey’s scale.

Obviously, it’s a problem when high school students are being given work that is not demanding in the area of rigor (and/or content). Most of the high school reading standards require much higher-order thinking than area 1. An example of a Literary Response and Analysis high school standard is the following: “Analyze interactions between main and subordinate characters in a literary text (e.g., internal and external conflicts, relationships, influences) and explain the way those interactions affect the plot” (CA 9^th/10^th Reading 3.3). This particular standard aligns to Manthey’s area 4: “Apply or Analyze Procedures or Metacognitive Knowledge.”

Clearly, the high school standard demands much higher-order thinking than the 1^st grade standard, and thus, requires teaching and learning strategies that will help students master the level of rigor of this standard. Recall questions won’t do that. An example of an assignment in which the cognitive level of the assignment aligns to the cognitive level of the standard would be the following: “Examine how Scrooge’s (main character) choices in life affect his relationships with the people (subordinate characters) in his life. Give an example of at least three relationships in Scrooge’s past or present life which were affected, either negatively or positively, by Scrooge’s life choices. Explain in detail how these relationships were affected. Explain how his interaction with these people (past and present) shapes the plot from beginning to end.”

In conclusion, Manthey’s research and findings support some of my own findings based on the results of analysis I have done at the Standards Company in which I have also analyzed and aligned both content and rigor of student work samples to their state standards. We do not use Manthey’s scale, but rather we use Norman Webb’s Depth-of-knowledge (DOK) tool as well as the Revised Bloom’s Taxonomy. Webb’s DOK has only four levels, compared to Manthey’s six levels (or “areas”), and while we do see assignments that align to the higher DOK and Bloom’s levels, the majority of assignments fall into the lower levels of DOK (levels 1 and 2) as well as the lower and middle levels of Bloom’s. So, I agree with Manthey that it would benefit teachers to examine data (the rigor of their assignments and teaching strategies and their grade-level content standards) in order to determine whether or not their teaching is truly requiring student mastery of the higher-order thinking required by the standards. The results would give valuable and eye-opening feedback which could then be used to make adjustments as needed.

Manthey’s Six Areas

• Area 1: Remember or Understand Facts and/or Concepts
• Area 2: Remember or Understand Procedures or Metacognitive Knowledge
• Area 3: Apply or Analyze Facts and/or Concepts
• Area 4: Apply or Analyze Procedures or Metacognitive Knowledge
• Area 5: Evaluate or Create Factual or Conceptual Knowledge
• Area 6: Evaluate or Create Procedures or Metacognitive Knowledge

REFERENCES

Anderson, L.W., Krathwohl, D.R., et al. (2001). A Taxonomy for Learning, Teaching, and Assessing. New York: Longman.

California Department of Education (1998). English–Language Arts Content Standards for California Public Schools Kindergarten Through Grade Twelve. Sacramento, CA.

Manthey, George (2005). Moving toward cognitive alignment: effective data provides feedback teachers can use to make adjustments in learning activities that result in standards alignment with content and cognitive rigor. Leadership Magazine.

To someone who is unfamiliar with DOK, it seems at first glance to be not much different than Bloom’s. And in some aspects they are similar. But DOK measures rigor, or complexity, written as a hierarchy but not as a taxonomy. Unlike Bloom’s, the verb is not the distinguishing factor.  Rather, the context of the verb is how one delineates between the different DOK levels, each of which basically describe and show the progression of the rigor of what is being taught and learned.

There are four DOK levels, created by Norman Webb, a Wisconsin research scientist and mathematics educator (http://www.wcer.wisc.edu/people/staff.php?sid=1342). Webb’s four levels of depth-of-knowledge are level 1 (recall), level 2 (skill/concept), level 3 (strategic thinking), and level 4 (extended thinking), and they are applicable to all subject areas and at all grade levels, including college.

Level 1 applies to items that involve simple knowledge. There is little comprehension involved at this level, no complexity, and no depth. It involves recall, rote response, and only a very basic, surface knowledge of material. It is not to say that Level 1 activities are unimportant because they are, in fact, very important. The levels of DOK progress in steps, one building on the other, and a student cannot reach a higher DOK level unless he has first mastered a lower DOK level.  Level 1 items are much like a foundation upon which deeper knowledge can be built.

Level 2 involves more comprehension of material than level 1. Students begin to apply skills, and they process concepts. At this level, students interpret material and make simple decisions about how to approach a problem. They do such things as infer, conclude, compare, summarize, and determine relationships (cause/effect, etc.). But, they are not yet dealing with much complexity. It is at level 3 that they “go beyond” and begin to wrestle with complex concepts, tasks, and material.

At level 3, students begin to do such things as analyze, evaluate, reason, and plan. They can also do these things at level 2, but they do them in much deeper and complex ways at level 3. For instance, at level 2, students might make inter-sentence inferences, but at level 3, students infer across an entire passage of text. At level 2, students compare/contrast ideas within one text; at level 3, they compare/contrast ideas between two separate texts. They begin to deal with abstractions and open-ended conclusions at this level, and they are also able to support their thinking.

Finally, after students have learned the basics at level 1, applied them at level 2, and strategically thought about them at level 3, they now extend their thinking by completing much deeper and complex tasks at level 4. At level 4, students’ grasp of knowledge is deep. According to Webb, higher-level thinking is absolutely central at this level, and students do such things as synthesize, hypothesize, evaluate, and analyze. Their interaction with material, whether it is language arts, math, science, or social studies, is in-depth, complete, and purposeful. Now, they are doing such things as writing in-depth multi-paragraph persuasive essays with thesis statements, evidence, examples, and complex reasoning. It involves much higher-level thought processes and skills than level 1 or 2.

Recognizing DOK levels is not always clear-cut or simple.  While some assignments, activities, and/or questions are easily identifiable as a certain DOK level, others are not.  Merely looking at the wording of a standard or assignment does not necessarily reveal the DOK level. Several things are involved, including the content, the activity and/or thinking processes, and the complexity of both the content and activity/thinking processes.

I hope to continue discussing Webb’s depth-of-knowledge in terms of its applicability and importance. But I’ll end with this, for now– here at The Standards Company, we discuss, research, and evaluate DOK on a regular basis. As a result, my own personal knowledge of Webb’s depth-of-knowledge levels has definitely grown from a DOK level 1 to a DOK level 4!

DOK LEVELS

Level 1: RECALL

Level 2: SKILL/CONCEPT

Level 3: STRATEGIC THINKING

Level 4:  EXTENDED THINKING

Differentiated Instruction : Bloom and Gardner