Kirkpatrick's 4 Levels of Evaluation

Captain Kirkpatrick, like Captain Kirk, went where nobody had gone before. He defined the way the training universe thinks about measuring outcomes.

This is number 8 on my list of Learning Theories That Actually Work.

You have a course, a workshop, a program and you need to measure the results in some way. You are probably using Kirkpatrick, maybe without knowing it. Donald Kirkpatrick has had such a heavy influence that in many training circles his levels are used as shorthand, almost a code. "We did a level one, but the level two isn't til next week."

The four levels are:

1. Reaction. Did they like it?
2. Learning. Did they learn it?
3. Behavior. Did they use it?
4. Results. Did it change anything?

The levels are usually diagramed as steps or as a pyramid. The idea is that each step is more valuable and harder to measure than the previous. The Level 1 "smile sheet" therefore is often regarded as an inferior set of data, almost a necessary evil, and its value downplayed.

But like "Captain Kirkpatrick" above, I think this is a mistake. The emotional component, the gut reaction, matters. In learning, motivation is everything. If learners simply don't like what you're providing, for whatever reason, all your other efforts may be for naught.

Note: people don't dislike training or education just because it's hard. This is a myth. Why do people sign up for difficult professions, play difficult games, solve difficult puzzles? People like difficulty if it is rewarding. So measuring Level 1 is important. If you get that wrong you probably are missing on some far deeper level than you think (see Social Learning Theory and Maslow with a twist for potential clues).

from Human Resource Managment https://goo.gl/aPuAt1

Here's a diagram I like far better, because it puts the learning in the middle where it belongs. You want to measure what you controlled most directly: did they learn it? Then the other three facets give you a handle on other outcomes that are not quite so directly in your control.

So here's Kirkpatrick, stacked against my criteria:

• It makes sense on the face of it

Mr. Spock likes it. It's logical.

• It has a solid history in research and practice

This has been around since 1959 and, as mentioned, is just assumed to be bedrock in many quarters. More on Kirkpatrick here. Are there other categories of assessment? Certainly. By all means get more sophisticated. CI 484 Learning Technologies offers a few more levels here, including Anthony Hamblin and others.

• It's easy to implement

The steps do tend to get more difficult to measure as you go up the presumptive ladder. However, the model itself just calls for you to address each of the different facets. In most cases it takes the extra effort of asking appropriate questions, checking appropriate data, and/or following up a few months down the road. It may be a pain. But it's not hard.

• I've tried it and it works

The critical thing in implementing Kirkpatrick is to plan ahead. When you define your learning objectives (see Bloom's), consider all four of Kirkpatrick's levels. Consider student's immediate response to be one of your objectives. Then measure direct learning outcomes, related behavior change, and longer-term business or other results. Figure out ahead of time what you will accept as measurements, as evidence, and make that work.

In higher education, we had weekly course-level "smile sheet" surveys (1), end-of-course grades (2), application assignments in which learning had to be applied and the results reported back (3), and we used course-to-course retention as a standard measure of business results (4). In sales training, we tracked engagement in training contests and activities week to week, month after month (1). We had quizzes (2), mystery-shopped our people (3), and of course measured improvement in sales (4).    Were these arbitrary measurements? Probably. Were they the best possible measures? Probably not. But we picked our measures and stuck with them, and each of them drove us to improve. It's easy to forget that at the end of the day, you're really measuring your own work.

Click here to go to back to the first learning theory that actually works

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Metacognition

Metacognition is an extremely powerful ally in learning, as necessary a tool for instructional designers as a hammer is for a carpenter. Building it into a learning experience is not hard. Leaving it out is not wise. 

This is number 7 on my list of Learning Theories That Actually Work.

Metacognition means thinking about thinking, and in learning environments that means getting students past thinking about what they are learning and into  thinking about how they are learning--what that learning means to them, how they can use it. .

The following is taken verbatim from the Teaching Excellence in Adult Literacy Fact Sheet (I added the italics):

____________________________________

Metacognitive knowledge refers to what individuals know about themselves as cognitive processors, about different approaches that can be used for learning and problem solving, and about the demands of a particular learning task. Metacognitive regulation refers to adjustments individuals make to their processes to help control their learning, such as planning, information management strategies, comprehension monitoring, de-bugging strategies, and evaluation of progress and goals. [John] Flavell (1979) further divides metacognitive knowledge into three categories:

• Person variables: What one recognizes about his or her strengths and weaknesses in learning and processing information.
• Task variables: What one knows or can figure out about the nature of a task and the processing demands required to complete the task—for example, knowledge that it will take more time to read, comprehend, and remember a technical article than it will a similar-length passage from a novel.
• Strategy variables: The strategies a person has “at the ready” to apply in a flexible way to successfully accomplish a task; for example, knowing how to activate prior knowledge before reading a technical article, using a glossary to look up unfamiliar words, or recognizing that sometimes one has to reread a paragraph several times before it makes sense.

[Jennifer A.] Livingston (1997) provides an example of all three variables: “I know that I (person variable) have difficulty with word problems (task variable), so I will answer the computational problems first and save the word problems for last (strategy variable).”

____________________________________

Here's how metacognition stacks up against my four criteria for making the list:

• It makes sense on the face of it

This is about taking responsibility my own learning. If I'm doing that, if I'm thinking about how I learn best, modifying my strategies, figuring out how to figure it out, I will learn better. So as a developer of learning, if you build in opportunities for this, or even require it, then you'll increase learning and retention. It's a no-brainer (so to speak).

• It has a solid history in research and practice

As the excerpt above demonstrates, this has been around a while and as far as I can tell, nobody is debunking anything about it. It's a very real phenomenon, and putting into practice yields good results.

• It's easy to implement

It is actually far simpler to put this into practice than you might guess. Add discussion questions that require it, assignments that require it, and review sessions that emphasize it... these are simple strategies that take just a little time and thought. One caution: never grade a metacognitive assignment! Explanation below.

• I've tried it, and it works

The standard higher-education online course has a weekly structure that goes something like this:

• Start a new week on Monday. 
• Read/listen/absorb new material by Wednesday. 
• Post in the discussion forum by Friday. 
• Turn in a paper by Sunday. 
• Lather, rinse, repeat.

This can be very a poor model if it's pretty much the whole model, but it can be very good if these Learning Theories That Work are incorporated. My teams have always made metacognition a significant component in higher education... and we call it, the Reflection Paper. (Quick aside: credit here to Barbara Schadlow, who really opened my eyes to the power of this.) We didn't require a Reflection Paper every week, but rather at the beginning, middle, and end of the course, just a few paragraphs describing what you're learning, how it's going, how you plan to use what you're learning, what you need to do differently in order to learn more.

It was never graded, other than a check mark for having completed it. This is important, because you want honesty. The value is in students actually reflecting on their own learning. We also made it private, between the student and the faculty only, which is also important. You don't want people comparing themselves to one another. It should always positioned for the benefit of the learner, which it is, in more than one way. In addition to helping in its own right, it gives teachers the opportunity to help learners with new learning strategies and approaches.

Adding metacognitive activities isn't hard. The concept is research-based. And it works. In my experience, students consistently write Reflection Papers about rededicating themselves to their own efforts, without any prompting, simply because they are taking the time to think about how things are going and how they might go better.

Click here to go to the next learning theory that actually works 

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Active Learning

Active learning at root is about learners doing things and thinking about the things they are doing.

This is number 6 on my list of Learning Theories That Actually Work.

The following is verbatim from "Active Learning: Creating Excitement in the Classroom" by Charles Bonwell (Italics below are added by me).

_______________________________

Some of the major characteristics associated with active learning strategies include:

Doing and thinking about doing

1. Students are involved in more than passive listening
2. Students are engaged in activities (e.g., reading, discussing, writing)
3. There is less emphasis placed on information transmission and greater emphasis placed on developing student skills
4. There is greater emphasis placed on the exploration of attitudes and values
5. Student motivation is increased (especially for adult learners)
6. Students can receive immediate feedback from their instructor
7. Students are involved in higher order thinking (analysis, synthesis, evaluation)

In summary, in the context of the college classroom, active learning involves students in doing things and thinking about the things they are doing

_______________________________

Bonwell wrote the above in the 1990s. Today, readers might find this a bit of a compendium of other learning theories. Just looking at my own list, there are clear parallels to Learning Styles (2 and 4), Maslow's Hierarchy (4), Gagne's 9 Events (6),  Bloom's Taxonomy (7), and Metacognition (thinking about the things they are doing).

But that's all to the good. Because Active Learning does include those things, plus it covers other very effective theories and models. Discovery Learning, in which the learners start with the problem and learn by solving it, is a type of Active Learning. Individualized Instruction is based on Active Learning; it's the logical byproduct of an emphasis on activities, skills, exploration of student's attitudes and values, immediate feedback, and higher-order thinking, which is by definition must be individualized. Take Active Learning to the next level and you get Personalized Instruction, which is the basis of Adaptive Learning. A lot of these methods and models make my list because they are just not that easy to implement yet. But all of them are types of Active Learning, and so all of them are worth exploring.

But let's take Active Learning through my criteria:

• It makes sense on the face of it

"Doing things and thinking about the things they are doing." How is that not better than sitting and listening?

• It has a solid history in research and practice

This has the longest history of practice of any learning model. Period. Call it apprenticeship for trades, call it discipleship for religious studies, the ancients wouldn't think of lecturing and then going home to dinner, thinking their work was done. Students did what the master did, that came first. If they could succeed at doing, then the explanation and nuance would come later. You learned this from this movie, which you will remember if you do something: click--> here (language warning)

• It's easy to implement

Get them to do things. Get them to talk about the things they do. Any questions?

• I've tried it, and it works

Flipping the classroom--doing the homework first--is active learning. Most online learning courses in higher education do this as part of the weekly strategy. We had a mantra when I had teams building many online courses simultaneously: "It's all about the assignments."

We had lots of pretty pictures, videos, great-looking user interfaces, and truly the best content we could find, sourced directly from the best experts. But our goal was learning, and none of those things mattered if the students didn't do something with them. So our "secret sauce" was in the assignments. We paid very close attention to what we asked the students to do, every week.

When they opened up their online classroom on a Monday, they were very likely already feeling the pressures of the coming week, feeling anything but motivated, maybe even dreading what they had to do this week for their degree program. That was the critical moment, the moment we wanted them to read the objectives and the activities and the assignment and think, "Hey, that's actually interesting. I can see how that will help me. I really want to do that."

So it's not what you tell them or show them, it's what you ask them to DO that matters. That's active learning in a nutshell.

Click here to go to the next learning theory that actually works:

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Bloom's Taxonomy

This is number 5 on my list of Learning Theories That Actually Work

Bloom's original pyramid

Bloom's does a lot of things, but mostly it is a very practical tool for writing learning objectives. If you want to create content and assignments and assessments that achieve the desired end, you need to carefully define the desired end. Enter Bloom's.

The image above is the original pyramid of critical thinking. From bottom to top:

• Knowledge: I can recite it 
• Comprehension: I can put in my own words
• Application: I can apply it to a new situation
• Analysis: I can break it down into component parts
• Synthesis: I can reconstruct those parts and combine them with other concepts
• Evaluation: I can make an informed critical judgment about it.

The graphic below illustrates the modified version by Anderson and Krathwohl. Click on the image to see the whole thing.

Click image to enlarge

This diagram is not nearly so complex as it may at first seem. The idea is to write your learning objectives starting with your content type on the left ("Knowledge Dimension") and then finding the appropriate level of critical thinking on the right ("Cognitive Process Dimension"). The Cognitive side is the old Bloom's pyramid, but with Synthesis dropped (actually folded into Analysis) and "Create" added as the highest level.

By my criteria:

• It makes sense on the face of it

Your course, lesson, elearning product, and/or assessment will naturally benefit from a little careful planning around learning objectives. Bloom's is a tool to help you get the right objective on a logical scale, from simple to complex.

• It has a solid history in research and practice

Nothing on my list is more tried and true. Benjamin Bloom chaired the committee that first developed the model, which was published in 1956. Pretty much everybody's been using it since. The updated version by Anderson and Krathwohl was created in 2000. The update is an improvement, and that is not controversial. Krathwohl actually worked with Bloom back in the 1960's.

• It's easy to implement

I think of Bloom's as a place to shop for active verbs. "At the end of the lesson, the learner will be able to VERB something." So in my practical way of thinking, Bloom's is a thesaurus for writing objectives. The lesson might require little or no critical thinking ("recite"), or it might require a lot ("analyze"). Get the right verb in the objective, use that verb when you design activities and assessments, and you can prove, and improve, learning outcomes.

Click image to enlarge

Above is one list of verbs, from the same site as the previous image. There are many other lists directly linked to a search engine near you. Find a verb, use it, done.

• I've tried it, and it works

I've found verbs, used them, and included Bloom's in development processes for courses, programs, seminars, workshops, videos, workbooks, you name it. Constructing learning objectives is actually a very small part of learning-product development, because it's so easily done and takes so very little time. But it's arguably the most critical part, because it's the seed from which learning success ultimately grows. So you need to do it well. And Bloom's is a critical asset for doing it well.

Click here to go to the next learning theory that actually works

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Maslow's Hierarchy and Identity-Level Outcomes

This is number 4 on my list of Learning Theories That Actually Work.

Maslow first proposed his hierarchy of needs in 1943, making this the graybeard learning theory of my list. And in fact, its age is showing. Much research done since then takes issue with some aspect or another. But here is the original, one and only:

The most basic needs are on the bottom, and the idea is that you have to fulfill them before moving up to address the next level. (More details on the model here.) The critics chirp about what goes where, do you really need this before that, or does self-actualization--being the best person you can be--even work for more collectivist-minded cultures. Maslow himself eventually added a spiritual level at the apex, which never got much play.

But there is a reason this has held a place in education for 70+ years. There is a basic reality expressed here that everyone understands: people have needs that either distract them from learning, or drive them to learning. And the highest level of all is the need to feel that what you do actually matters. And that's why it's on my list. I focus on one aspect, the top end. This is, in my experience, a theorem unto itself:

All learning events have identity-level outcomes

What does that mean? It means that learning always goes beyond the cog-and-gear realities of the world of work, beyond the need to learn this or that to be able to get this job to feed my family, or buy that particular roof I'd like over my head. Maslow's higher levels are always in play. Forget this at your own peril: Learning changes you. Always.

Commercial marketers will tell you that if you drink this beer or buy that car, you will be a different person. And of course, that's just not true. (I will make an exception if you want to argue for Harley-Davidson motorcycles. But only to avoid the argument). The point is that education actually does change your identity. It gives you different skills, puts you on different playing fields, adjusts your very self-image in permanent ways. You become someone different, if only a little bit, in your own eyes and those of others. Some education will actually change your name, by adding initials after it or an identifier in front of it. (Not even a Harley can do that.)

So how is this practical? What do I, as a learning developer, do with it? You ask yourself early in the development process, "How will learners see themselves differently when they're done?" Then you build content and activities that help that happen. You establish identity-level outcomes. I'm not talking about trite statements like, "You're a safer person when you wear your safety goggles!" I'm talking about a portrait of success.

Think about it. Why do people want to work in Silicon Valley? Why do they want to work on Wall Street, or join Greenpeace? These careers have a sense of identity that comes with them, one that everyone understands, and so they attract like-minded people. But the fact is, every job has that. Every field has a unique allure to those who sign up for it. You can tap into that allure, the one that draws those who actually find it irresistible... and you can use it.

I get it if you're creating degree programs, you might say, but how can this possibly be true if I'm training Walmart employees to do inventory? My answer would be that if that's what you're doing, you have even more reason to focus on identity. Remember my theorem. All learning events have identity-level outcomes. Those learners will find identity-level outcomes themselves. Here's one: "Wow, look at me. I have become the guy who does inventory at Walmart. How did this happen?" See how that works? What you are learning to do is what you are becoming. No exceptions.

But here's the flip side, actually the A side, the positive side: Walmart has a mission, too. Their mission is, "Saving people money so they can live better." Does that sound like a waste of time to you? Or does it sound like a worthwhile goal? Is it worth getting up in the morning and going to do inventory if that is a part of helping people live better? Your training should explain how doing inventory contributes to that end; not in happy talk, but with data. With actual stories. With reality, big picture and small.

And why can't your training highlight what success looks like in a Walmart employee (See Social Learning Theory for more on this). The point is, if you don't know what your identity-level outcomes are, you are the only one in the dark. You can be sure your students/trainees/learners do, if only intuitively.

So, speaking of doing inventory, let's go through my list of criteria for being a Learning Theory That Actually Works.

• It makes sense on the face of it

The whole reason Maslow has been around for 7 decades, and will likely outlive us all, is that it makes sense on the face of it. If I'm a learner, I'm not paying attention if I have to go to the bathroom. And I am motivated by a need to be accepted, to find my peer group. It's just logical, even if holes can be shot in in it here or there by researchers.

• It has a solid history in research and practice

Here's a 2011 report on research confirming Maslow. And here's one from 1976 questioning Maslow. You get the point. There's a lot out there. It's been studied, tested, used, abused... but it hasn't gone away.

• It's easy to implement

I've found it easier on the bottom and top levels than in the middle, but that's enough to qualify it for my list (the middle is very doable, however... think group dynamics). I've covered the top levels. At the bottom levels, learners need to feel safe and secure, with limited distractions. Take breaks, serve snacks, point out the bathrooms. If your training is online exclusively, keep their accounts and passwords and profile data secure!

• I've tried it, and it works

One story. When creating a Nursing master's degree for a well-known online university, we explored the motivators of nurses, and found that nurses in our target audience felt guilty about leaving the floor. Their work identity was wrapped up in taking care of patients. But after ten or fifteen years, they were worn out, burned out, thinking about money for their kids' college. So in the very first lesson of the very first course, we invested in a video that had a message from top nurse executives and conference speakers, which boiled down to: "All the things that you now do for six patients a day, you can do for 60. Or 600. Or 6,000. This is what it means to be a nurse executive or a nurse educator."

We created an identity-level objective, focused on an identity-level outcome, and we played that theme out in course after course. The result was amazing. We passed the initial nursing accreditation review with zero deficiencies. Our own faculty were blown away by that--they'd never heard of zero deficiencies for an on-the-ground Nursing master's degree, much less a fully-online one. And the ongoing results were outstanding as well, as reflected in very low attrition rates.

He may be old and slow, but Maslow makes the difference between good and great. He can play on my team any day.

Click here to go to the next learning theory that actually works 

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Felder-Silverman Learning Style Model

This is number 2 on my list of Learning Theories that Actually Work.

A lot of work has been done and lots more ink has been spilled on the subject of learning preferences
and learning styles. You've got multiple intelligences, right brain/left brain, social/solitary, and many other dichotomies and trichotomies from which to choose. And frankly, this whole area can become a mud pit for learning developers. But you can't just drive around it. People really do learn in very different ways, and if you gear everything to what you happen to think is effective, you're inviting everyone who isn't you, or a lot like you, to tune out.

Some are apples, some are oranges.
Deal with it.

So how do you invite them to tune in? The Felder-Silverman Learning Style Model is a good place to start, and in fact, if you're looking for practical applications, it's also a good place to end. The model covers four important learning differences:

1. Active vs. reflective learning (I want to try it right away vs. let me think about this first)
2. Sensing vs. intuitive learning (Fact vs. feeling)
3. Visual vs. verbal learning (Show me vs. tell me)
4. Sequential vs. global learning (Just give me the steps vs. give me the big picture first)

There are a number of ways to use these variances when building learning. Some adaptive technology platforms, for example, can suss out which way you learn more quickly and then send you more pictures and fewer words, or vice versa. But most of us are working on more ordinary learning platforms, so I will stick to my criteria for "Learning Theories that Actually Work." First criteria:

• It makes sense on the face of it

If you want to understand better how they apply to you, you can take the Solomon-Felder questionnaire to get a clear sense of your own preferences. But you probably know most of this from experience. Some people want to roll up their sleeves and just do something, anything, while others want to meet and plan and think and talk first. The latter read directions; the former don't. Some people want facts and data, while others want to discuss meaning, direction, impact. Some people use words like, "I see," while others say, "I hear you." The former prefer movies, the latter concerts. Some people can't listen to the story you're telling until you assure them everything turns out alright in the end. These are the same people who read the last page of a novel first. These are all learning preferences, and it just makes sense that appealing to each of them appropriately increases their satisfaction, which contributes to involvement, which helps learning and retention.

An example of how this works in real life, outside the classroom and training room... I was in a meeting where a vendor pitched to my CEO. The vendor started with his company's background, setting the usual context, but the CEO interrupted and said, "I don't need all that fluff, just show me how it works." The vendor was clearly put off, but obliged. When the demo ended, the CEO sat back and nodded and said, "So, how old is your company?" From there on he was very interested in "all that fluff."

Recognizing that the CEO was a strongly sequential learner might have kept the vendor from thinking his client's boss was self-absorbed, or just plain rude. And knowing this about himself might have prevented the CEO from acting that way. The point is, this is real and it matters and it just makes sense to address it when constructing learning experiences.

• It has a solid history in research and practice

Felder and Silverman go back to 1988, and Felder and Solomon created a readily-available survey that makes it personal and even more practical. So I've chosen this as safe, solid ground for addressing learning styles. Here's a link to a bit more research on the learning style model.

• It's easy to implement

Choosing to address different learning styles does not require any great investment of time or money. It's really about using what you have wisely.

You can address both active and reflective learners by giving them all something to do and something to think about, and then letting them do whichever they prefer in whichever order. You can address sensing and intuitive learners the same way, being sure to include both hard facts and human consequences. You can create a forum for those who need to talk. You can provide comment boxes. You can ask deeper-meaning questions. And you can let people explore concepts, never being satisfied because you can collect correct/incorrect answers.

Verbal and visual preferences can be addressed by including opportunities for learners to communicate back, or to do their assignments, using either words or images. More and more online and on-the-ground courses allow video or other forms of visual submissions. If that's not practical for you, at least be sure that visual learners have visuals.

The images above were free and clear from  freeimages.com and took about four minutes to find, a few more to format and upload using Keynote. Maybe they aren't the perfect visuals for making the point but, as they say, if you can't do everything at least do something.

If you can get a voice to read the on-screen text, all the better. But remember, writing is already verbal, much more closely linked to auditory learning than visual learning. So if your budget or deadlines require you to choose between recording a voice to read the words aloud and finding or creating images that illustrate the ideas, go with the images.

Lecture Video
(Standard Quality)

By the way, this is why talking head video lectures are tragically ineffective... Visual learners will actually be distracted watching a poorly-lit, badly-framed portrait speak, making it harder for them to absorb the words being said. And even verbal learners will be happier reading a well-written page in their own inner voice, at their own pace, instead of trying to work through the tinny audio and the uhs and ums and pauses and slurping sips from water bottles.

And no, Ted Talks are not "talking heads." They are slickly-produced, carefully designed experiences, created by video professionals. There's a reason you can't just make your own Ted Talk and upload it.

A simple way to address both sequential and global learners is to make sure there is a "next" button and a table of contents readily available, or preferably, both always visible on screen. You should also make the decision not to limit the learners' path by requiring only a single linear progression. Let people go where they will, jump ahead, move backward and forward as they would in a book, as much as the content allows. Global learners who aren't allowed this kind of freedom--at least adult learners--will feel like cattle in a corral. And unlike cattle, they can simply choose to wander off.

• I've tried it, and it works

As the above "easy-to-implement" examples suggest, I've found a number of ways to incorporate these learning styles. What I've found most helpful is to remember that your user interface, whatever it is, either does or does not address these learning differences. Whether it's custom-built or Blackboard, many of these can be addressed by settings on your LMS or instructions to developers.

Early in the online-learning era I was responsible for designing an interface from scratch, for online live video lectures with slides, and used Felder-Silverman. We put a button on the primary screen called "Overview," and made the decision to let learners page ahead to look at upcoming slides (Global/Sequential). We had a button for texting with online experts who were not presenters, so learners could discuss whatever they wanted at any time (Active/ Reflective and Sensing/Intuitive). And we used graphic images and pre-produced videos liberally (Verbal/Visual). We also built in Gagne with buttons labeled "Objectives," "Prior Knowledge," and "Feedback." Of course there is not much call these days to build your own interface, but you can look for the same functionality in whatever you are using. And you can use Felder-Silverman as part of your checklist if you are looking to change your LMS.

In truth, I've gotten to the point that when I'm in an online learning environment, most of these are second-nature. I'm a global learner and need that bird's eye view, but if I don't see that "next" button... I get nervous.

And that may help explain why I present in the following as part of the structure of these posts:

Click here to go to the next theory

Or...

Click in any order you prefer:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

What an interdependent learning ecosystem looks like

A survey conducted by Human Capital Media for CLO Magazine found that over ninety percent of responding organizations partner with a vendor for some area of learning and development. It's an ecosystem out there. And what it looks like is...

...If you're not part of Pac Man, you're part of what's getting chewed up.

Learning Theories That Actually Work

"The development of effective online learning materials should be based on proven and sound learning theories." --Anderson and Elloumi, Athabasca University

Sounds good. Couldn't agree more. But what exactly are the "proven theories" I should be using? Learning science can be strange... after all, a theory by definition is "an idea that is suggested or presented as possibly true but that is not known or proven to be true." (Merriam Webster). So how do you get a "proven theory"? Well, theories are also structures, ways to think about and exploit certain bodies of knowledge. And in some cases, in learning for example, they can be highly practical tools for simplifying what otherwise can be highly complex. So they get used, in the real world, and in that way are "proven to be sound."

I'm not a researcher. I'm not even an academic. I am, however, an educator. Learning products and programs and courses and services I've built, or built the guidelines and teams for building, have taught literally hundreds of thousands of people on all levels of education in all sorts of different fields of content. I've worked in corporate, higher ed, continuing ed, K-12. I've worked internationally and domestically, and in content areas as diverse as sales training and medical education, social work and computer science. I've done a lot of teaching teachers to teach. All of it has been technology-delivered, from web to video to satellite to CD-ROM to videoconference. A lot has been hybrid. So I'm in a bit of a unique position to talk about what works... everywhere.

I also operate in the business world, which is to say I build things that people buy. And people who buy things expect them to work. So I don't and won't waste time and energy and resources trying out the latest, greatest, hottest new learning theory. I have to go with what I know. I've made some mistakes. But there are some things that just never let me down. And that's what I'd like to share here.

So in response to Athabasca's Theory and Practice of Online Learning quoted above, and to the Deans for Impact report on the Science of Learning, I offer my own brief compendium of what I call Learning Theories That Actually Work.

Here's what a learning theory needs in order to qualify for my list:

• It makes sense on the face of it
• It has a solid history in research and practice
• It's easy to implement
• I've tried it, and it works

Below are my top 8. They are in order from "Start Here" to "End There," with "Include This By All Means" in between.

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Individualized Instruction)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Gagne's 9 Events of Instruction

This is number 1 on my list of "Learning Theories that Actually Work."

Gagne's Nine Events of Instruction are a research-based learning model, a proven path for learners that you can use as you design learning experiences for others. Gagne is the granddaddy of learning models, the model for many other learning models.

If you are unfamiliar with the concept of a learning model, you should recognize that every learning event has one, whether the model is any good or not, or whether the instructor even knows it exists or not. The only one that is widely known is, "Tell them what you're going to tell them, tell them, then tell them what you told them." It's not terribly sophisticated, but it manages to hit three of Gagne's nine events (1, 4, and 9). In medicine, it's "Watch one, do one, teach one," (4, 6, and 9). In too many corporate training rooms it's "Death by Powerpoint" (4).

Here are Gagne's Nine:

1. Gain attention
2. Inform the learner of objectives
3. Stimulate the recall of prior learning
4. Present stimulus material (new content)
5. Provide learner guidance
6. Elicit performance
7. Provide feedback
8. Assess performance
9. Enhance retention and transfer

Let's look at Gagne in light of my criteria for being included on a list of Learning Theories That Actually Work:

• It makes sense on the face of it

First, it is a repeatable pattern, a format. That alone provides structure, which means it gives you the ability to assure quality. It can be used as a checklist as you design any learning experience. Second, each step makes sense in its own right. Here are the 9 steps in one readily-identifiable context: 

1. Okay, class, I've got a question for you... (gain attention)

2. When we're done, here's what you'll be able to do... (state objectives)

3. Remember how you did [the previous lesson]? Well, now we're going one better... (prior learning)

4. Here's what you need to know... (new information)

5. Here are some examples... Everyone got it? Any questions? (provide guidance) 

6. Okay, now you give it a try. (elicit performance)

7. That's a good effort, but it will work better if you do this instead. (provide feedback)

8. And yes, this will be on the quiz tomorrow. (assess performance) 

9. And yes, it will also be on the final exam at the end of the term. (enhance retention)

That's an easy classroom example, but you get the point. It will work as a checklist in any learning environment, technical or otherwise.

• It has a solid history in research and practice

Gagne started creating his approaches to teaching and learning for the military in World War II. He's written and been written about extensively since then, and his events, or steps of instruction have been a staple everywhere. Just Google "Gagne's nine events" and you'll find a long list of people, schools, and organizations that are using it, from Northern Illinois University to IEEE to InstructionalDesign.org to the US Embassy in Hungary. If you look hard enough you'll find that research has been done on each step, and on the steps in combination, but it's the sheer weight of its diverse usage over such a long period time that is more than enough to pass this criteria.

• It's easy to implement

This is as easy to use as any other checklist. In order to generate a learning experience for others, you have to devise events and activities and put them in order anyway. Each lesson, each unique module, each seminar, each whatever, should follow a carefully considered pattern, and for the sake of learners, that pattern shouldn't vary dramatically lesson to lesson. That pattern is your learning model. Gagne is a great default. Vary where you want, but vary because you decided to vary not because you forgot something. It costs nothing (except maybe the extra time it always takes to do something right!)

• I've tried it and it works

Early in my career I was more a media producer than a learning developer, and research-backed learning models were an absolute revelation to me. If the steps didn't work, I never would have kept using them--I'd have just added more video, more animation, more of that powerful media stuff everyone likes. But in fact, a good learning model is everything. I once took a job where literally millions of dollars were being spent on high-gloss "TV shows" that were purported to be training. I instituted Gagne, created a development process around it, and dropped a lot of costs (not quality) while improving the learning outcomes dramatically. Gagne improved both my learning and business results, and I was hooked. Now I take him everywhere I go.

Click here to go to the next learning theory that actually works.

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post