Kirkpatrick's 4 Levels of Evaluation

Captain Kirkpatrick, like Captain Kirk, went where nobody had gone before. He defined the way the training universe thinks about measuring outcomes.

This is number 8 on my list of Learning Theories That Actually Work.

You have a course, a workshop, a program and you need to measure the results in some way. You are probably using Kirkpatrick, maybe without knowing it. Donald Kirkpatrick has had such a heavy influence that in many training circles his levels are used as shorthand, almost a code. "We did a level one, but the level two isn't til next week."

The four levels are:

1. Reaction. Did they like it?
2. Learning. Did they learn it?
3. Behavior. Did they use it?
4. Results. Did it change anything?

The levels are usually diagramed as steps or as a pyramid. The idea is that each step is more valuable and harder to measure than the previous. The Level 1 "smile sheet" therefore is often regarded as an inferior set of data, almost a necessary evil, and its value downplayed.

But like "Captain Kirkpatrick" above, I think this is a mistake. The emotional component, the gut reaction, matters. In learning, motivation is everything. If learners simply don't like what you're providing, for whatever reason, all your other efforts may be for naught.

Note: people don't dislike training or education just because it's hard. This is a myth. Why do people sign up for difficult professions, play difficult games, solve difficult puzzles? People like difficulty if it is rewarding. So measuring Level 1 is important. If you get that wrong you probably are missing on some far deeper level than you think (see Social Learning Theory and Maslow with a twist for potential clues).

from Human Resource Managment https://goo.gl/aPuAt1

Here's a diagram I like far better, because it puts the learning in the middle where it belongs. You want to measure what you controlled most directly: did they learn it? Then the other three facets give you a handle on other outcomes that are not quite so directly in your control.

So here's Kirkpatrick, stacked against my criteria:

• It makes sense on the face of it

Mr. Spock likes it. It's logical.

• It has a solid history in research and practice

This has been around since 1959 and, as mentioned, is just assumed to be bedrock in many quarters. More on Kirkpatrick here. Are there other categories of assessment? Certainly. By all means get more sophisticated. CI 484 Learning Technologies offers a few more levels here, including Anthony Hamblin and others.

• It's easy to implement

The steps do tend to get more difficult to measure as you go up the presumptive ladder. However, the model itself just calls for you to address each of the different facets. In most cases it takes the extra effort of asking appropriate questions, checking appropriate data, and/or following up a few months down the road. It may be a pain. But it's not hard.

• I've tried it and it works

The critical thing in implementing Kirkpatrick is to plan ahead. When you define your learning objectives (see Bloom's), consider all four of Kirkpatrick's levels. Consider student's immediate response to be one of your objectives. Then measure direct learning outcomes, related behavior change, and longer-term business or other results. Figure out ahead of time what you will accept as measurements, as evidence, and make that work.

In higher education, we had weekly course-level "smile sheet" surveys (1), end-of-course grades (2), application assignments in which learning had to be applied and the results reported back (3), and we used course-to-course retention as a standard measure of business results (4). In sales training, we tracked engagement in training contests and activities week to week, month after month (1). We had quizzes (2), mystery-shopped our people (3), and of course measured improvement in sales (4).    Were these arbitrary measurements? Probably. Were they the best possible measures? Probably not. But we picked our measures and stuck with them, and each of them drove us to improve. It's easy to forget that at the end of the day, you're really measuring your own work.

Click here to go to back to the first learning theory that actually works

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Metacognition

Metacognition is an extremely powerful ally in learning, as necessary a tool for instructional designers as a hammer is for a carpenter. Building it into a learning experience is not hard. Leaving it out is not wise. 

This is number 7 on my list of Learning Theories That Actually Work.

Metacognition means thinking about thinking, and in learning environments that means getting students past thinking about what they are learning and into  thinking about how they are learning--what that learning means to them, how they can use it. .

The following is taken verbatim from the Teaching Excellence in Adult Literacy Fact Sheet (I added the italics):

____________________________________

Metacognitive knowledge refers to what individuals know about themselves as cognitive processors, about different approaches that can be used for learning and problem solving, and about the demands of a particular learning task. Metacognitive regulation refers to adjustments individuals make to their processes to help control their learning, such as planning, information management strategies, comprehension monitoring, de-bugging strategies, and evaluation of progress and goals. [John] Flavell (1979) further divides metacognitive knowledge into three categories:

• Person variables: What one recognizes about his or her strengths and weaknesses in learning and processing information.
• Task variables: What one knows or can figure out about the nature of a task and the processing demands required to complete the task—for example, knowledge that it will take more time to read, comprehend, and remember a technical article than it will a similar-length passage from a novel.
• Strategy variables: The strategies a person has “at the ready” to apply in a flexible way to successfully accomplish a task; for example, knowing how to activate prior knowledge before reading a technical article, using a glossary to look up unfamiliar words, or recognizing that sometimes one has to reread a paragraph several times before it makes sense.

[Jennifer A.] Livingston (1997) provides an example of all three variables: “I know that I (person variable) have difficulty with word problems (task variable), so I will answer the computational problems first and save the word problems for last (strategy variable).”

____________________________________

Here's how metacognition stacks up against my four criteria for making the list:

• It makes sense on the face of it

This is about taking responsibility my own learning. If I'm doing that, if I'm thinking about how I learn best, modifying my strategies, figuring out how to figure it out, I will learn better. So as a developer of learning, if you build in opportunities for this, or even require it, then you'll increase learning and retention. It's a no-brainer (so to speak).

• It has a solid history in research and practice

As the excerpt above demonstrates, this has been around a while and as far as I can tell, nobody is debunking anything about it. It's a very real phenomenon, and putting into practice yields good results.

• It's easy to implement

It is actually far simpler to put this into practice than you might guess. Add discussion questions that require it, assignments that require it, and review sessions that emphasize it... these are simple strategies that take just a little time and thought. One caution: never grade a metacognitive assignment! Explanation below.

• I've tried it, and it works

The standard higher-education online course has a weekly structure that goes something like this:

• Start a new week on Monday. 
• Read/listen/absorb new material by Wednesday. 
• Post in the discussion forum by Friday. 
• Turn in a paper by Sunday. 
• Lather, rinse, repeat.

This can be very a poor model if it's pretty much the whole model, but it can be very good if these Learning Theories That Work are incorporated. My teams have always made metacognition a significant component in higher education... and we call it, the Reflection Paper. (Quick aside: credit here to Barbara Schadlow, who really opened my eyes to the power of this.) We didn't require a Reflection Paper every week, but rather at the beginning, middle, and end of the course, just a few paragraphs describing what you're learning, how it's going, how you plan to use what you're learning, what you need to do differently in order to learn more.

It was never graded, other than a check mark for having completed it. This is important, because you want honesty. The value is in students actually reflecting on their own learning. We also made it private, between the student and the faculty only, which is also important. You don't want people comparing themselves to one another. It should always positioned for the benefit of the learner, which it is, in more than one way. In addition to helping in its own right, it gives teachers the opportunity to help learners with new learning strategies and approaches.

Adding metacognitive activities isn't hard. The concept is research-based. And it works. In my experience, students consistently write Reflection Papers about rededicating themselves to their own efforts, without any prompting, simply because they are taking the time to think about how things are going and how they might go better.

Click here to go to the next learning theory that actually works 

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Active Learning

Active learning at root is about learners doing things and thinking about the things they are doing.

This is number 6 on my list of Learning Theories That Actually Work.

The following is verbatim from "Active Learning: Creating Excitement in the Classroom" by Charles Bonwell (Italics below are added by me).

_______________________________

Some of the major characteristics associated with active learning strategies include:

Doing and thinking about doing

1. Students are involved in more than passive listening
2. Students are engaged in activities (e.g., reading, discussing, writing)
3. There is less emphasis placed on information transmission and greater emphasis placed on developing student skills
4. There is greater emphasis placed on the exploration of attitudes and values
5. Student motivation is increased (especially for adult learners)
6. Students can receive immediate feedback from their instructor
7. Students are involved in higher order thinking (analysis, synthesis, evaluation)

In summary, in the context of the college classroom, active learning involves students in doing things and thinking about the things they are doing

_______________________________

Bonwell wrote the above in the 1990s. Today, readers might find this a bit of a compendium of other learning theories. Just looking at my own list, there are clear parallels to Learning Styles (2 and 4), Maslow's Hierarchy (4), Gagne's 9 Events (6),  Bloom's Taxonomy (7), and Metacognition (thinking about the things they are doing).

But that's all to the good. Because Active Learning does include those things, plus it covers other very effective theories and models. Discovery Learning, in which the learners start with the problem and learn by solving it, is a type of Active Learning. Individualized Instruction is based on Active Learning; it's the logical byproduct of an emphasis on activities, skills, exploration of student's attitudes and values, immediate feedback, and higher-order thinking, which is by definition must be individualized. Take Active Learning to the next level and you get Personalized Instruction, which is the basis of Adaptive Learning. A lot of these methods and models make my list because they are just not that easy to implement yet. But all of them are types of Active Learning, and so all of them are worth exploring.

But let's take Active Learning through my criteria:

• It makes sense on the face of it

"Doing things and thinking about the things they are doing." How is that not better than sitting and listening?

• It has a solid history in research and practice

This has the longest history of practice of any learning model. Period. Call it apprenticeship for trades, call it discipleship for religious studies, the ancients wouldn't think of lecturing and then going home to dinner, thinking their work was done. Students did what the master did, that came first. If they could succeed at doing, then the explanation and nuance would come later. You learned this from this movie, which you will remember if you do something: click--> here (language warning)

• It's easy to implement

Get them to do things. Get them to talk about the things they do. Any questions?

• I've tried it, and it works

Flipping the classroom--doing the homework first--is active learning. Most online learning courses in higher education do this as part of the weekly strategy. We had a mantra when I had teams building many online courses simultaneously: "It's all about the assignments."

We had lots of pretty pictures, videos, great-looking user interfaces, and truly the best content we could find, sourced directly from the best experts. But our goal was learning, and none of those things mattered if the students didn't do something with them. So our "secret sauce" was in the assignments. We paid very close attention to what we asked the students to do, every week.

When they opened up their online classroom on a Monday, they were very likely already feeling the pressures of the coming week, feeling anything but motivated, maybe even dreading what they had to do this week for their degree program. That was the critical moment, the moment we wanted them to read the objectives and the activities and the assignment and think, "Hey, that's actually interesting. I can see how that will help me. I really want to do that."

So it's not what you tell them or show them, it's what you ask them to DO that matters. That's active learning in a nutshell.

Click here to go to the next learning theory that actually works:

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Bloom's Taxonomy

This is number 5 on my list of Learning Theories That Actually Work

Bloom's original pyramid

Bloom's does a lot of things, but mostly it is a very practical tool for writing learning objectives. If you want to create content and assignments and assessments that achieve the desired end, you need to carefully define the desired end. Enter Bloom's.

The image above is the original pyramid of critical thinking. From bottom to top:

• Knowledge: I can recite it 
• Comprehension: I can put in my own words
• Application: I can apply it to a new situation
• Analysis: I can break it down into component parts
• Synthesis: I can reconstruct those parts and combine them with other concepts
• Evaluation: I can make an informed critical judgment about it.

The graphic below illustrates the modified version by Anderson and Krathwohl. Click on the image to see the whole thing.

Click image to enlarge

This diagram is not nearly so complex as it may at first seem. The idea is to write your learning objectives starting with your content type on the left ("Knowledge Dimension") and then finding the appropriate level of critical thinking on the right ("Cognitive Process Dimension"). The Cognitive side is the old Bloom's pyramid, but with Synthesis dropped (actually folded into Analysis) and "Create" added as the highest level.

By my criteria:

• It makes sense on the face of it

Your course, lesson, elearning product, and/or assessment will naturally benefit from a little careful planning around learning objectives. Bloom's is a tool to help you get the right objective on a logical scale, from simple to complex.

• It has a solid history in research and practice

Nothing on my list is more tried and true. Benjamin Bloom chaired the committee that first developed the model, which was published in 1956. Pretty much everybody's been using it since. The updated version by Anderson and Krathwohl was created in 2000. The update is an improvement, and that is not controversial. Krathwohl actually worked with Bloom back in the 1960's.

• It's easy to implement

I think of Bloom's as a place to shop for active verbs. "At the end of the lesson, the learner will be able to VERB something." So in my practical way of thinking, Bloom's is a thesaurus for writing objectives. The lesson might require little or no critical thinking ("recite"), or it might require a lot ("analyze"). Get the right verb in the objective, use that verb when you design activities and assessments, and you can prove, and improve, learning outcomes.

Click image to enlarge

Above is one list of verbs, from the same site as the previous image. There are many other lists directly linked to a search engine near you. Find a verb, use it, done.

• I've tried it, and it works

I've found verbs, used them, and included Bloom's in development processes for courses, programs, seminars, workshops, videos, workbooks, you name it. Constructing learning objectives is actually a very small part of learning-product development, because it's so easily done and takes so very little time. But it's arguably the most critical part, because it's the seed from which learning success ultimately grows. So you need to do it well. And Bloom's is a critical asset for doing it well.

Click here to go to the next learning theory that actually works

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Maslow's Hierarchy and Identity-Level Outcomes

This is number 4 on my list of Learning Theories That Actually Work.

Maslow first proposed his hierarchy of needs in 1943, making this the graybeard learning theory of my list. And in fact, its age is showing. Much research done since then takes issue with some aspect or another. But here is the original, one and only:

The most basic needs are on the bottom, and the idea is that you have to fulfill them before moving up to address the next level. (More details on the model here.) The critics chirp about what goes where, do you really need this before that, or does self-actualization--being the best person you can be--even work for more collectivist-minded cultures. Maslow himself eventually added a spiritual level at the apex, which never got much play.

But there is a reason this has held a place in education for 70+ years. There is a basic reality expressed here that everyone understands: people have needs that either distract them from learning, or drive them to learning. And the highest level of all is the need to feel that what you do actually matters. And that's why it's on my list. I focus on one aspect, the top end. This is, in my experience, a theorem unto itself:

All learning events have identity-level outcomes

What does that mean? It means that learning always goes beyond the cog-and-gear realities of the world of work, beyond the need to learn this or that to be able to get this job to feed my family, or buy that particular roof I'd like over my head. Maslow's higher levels are always in play. Forget this at your own peril: Learning changes you. Always.

Commercial marketers will tell you that if you drink this beer or buy that car, you will be a different person. And of course, that's just not true. (I will make an exception if you want to argue for Harley-Davidson motorcycles. But only to avoid the argument). The point is that education actually does change your identity. It gives you different skills, puts you on different playing fields, adjusts your very self-image in permanent ways. You become someone different, if only a little bit, in your own eyes and those of others. Some education will actually change your name, by adding initials after it or an identifier in front of it. (Not even a Harley can do that.)

So how is this practical? What do I, as a learning developer, do with it? You ask yourself early in the development process, "How will learners see themselves differently when they're done?" Then you build content and activities that help that happen. You establish identity-level outcomes. I'm not talking about trite statements like, "You're a safer person when you wear your safety goggles!" I'm talking about a portrait of success.

Think about it. Why do people want to work in Silicon Valley? Why do they want to work on Wall Street, or join Greenpeace? These careers have a sense of identity that comes with them, one that everyone understands, and so they attract like-minded people. But the fact is, every job has that. Every field has a unique allure to those who sign up for it. You can tap into that allure, the one that draws those who actually find it irresistible... and you can use it.

I get it if you're creating degree programs, you might say, but how can this possibly be true if I'm training Walmart employees to do inventory? My answer would be that if that's what you're doing, you have even more reason to focus on identity. Remember my theorem. All learning events have identity-level outcomes. Those learners will find identity-level outcomes themselves. Here's one: "Wow, look at me. I have become the guy who does inventory at Walmart. How did this happen?" See how that works? What you are learning to do is what you are becoming. No exceptions.

But here's the flip side, actually the A side, the positive side: Walmart has a mission, too. Their mission is, "Saving people money so they can live better." Does that sound like a waste of time to you? Or does it sound like a worthwhile goal? Is it worth getting up in the morning and going to do inventory if that is a part of helping people live better? Your training should explain how doing inventory contributes to that end; not in happy talk, but with data. With actual stories. With reality, big picture and small.

And why can't your training highlight what success looks like in a Walmart employee (See Social Learning Theory for more on this). The point is, if you don't know what your identity-level outcomes are, you are the only one in the dark. You can be sure your students/trainees/learners do, if only intuitively.

So, speaking of doing inventory, let's go through my list of criteria for being a Learning Theory That Actually Works.

• It makes sense on the face of it

The whole reason Maslow has been around for 7 decades, and will likely outlive us all, is that it makes sense on the face of it. If I'm a learner, I'm not paying attention if I have to go to the bathroom. And I am motivated by a need to be accepted, to find my peer group. It's just logical, even if holes can be shot in in it here or there by researchers.

• It has a solid history in research and practice

Here's a 2011 report on research confirming Maslow. And here's one from 1976 questioning Maslow. You get the point. There's a lot out there. It's been studied, tested, used, abused... but it hasn't gone away.

• It's easy to implement

I've found it easier on the bottom and top levels than in the middle, but that's enough to qualify it for my list (the middle is very doable, however... think group dynamics). I've covered the top levels. At the bottom levels, learners need to feel safe and secure, with limited distractions. Take breaks, serve snacks, point out the bathrooms. If your training is online exclusively, keep their accounts and passwords and profile data secure!

• I've tried it, and it works

One story. When creating a Nursing master's degree for a well-known online university, we explored the motivators of nurses, and found that nurses in our target audience felt guilty about leaving the floor. Their work identity was wrapped up in taking care of patients. But after ten or fifteen years, they were worn out, burned out, thinking about money for their kids' college. So in the very first lesson of the very first course, we invested in a video that had a message from top nurse executives and conference speakers, which boiled down to: "All the things that you now do for six patients a day, you can do for 60. Or 600. Or 6,000. This is what it means to be a nurse executive or a nurse educator."

We created an identity-level objective, focused on an identity-level outcome, and we played that theme out in course after course. The result was amazing. We passed the initial nursing accreditation review with zero deficiencies. Our own faculty were blown away by that--they'd never heard of zero deficiencies for an on-the-ground Nursing master's degree, much less a fully-online one. And the ongoing results were outstanding as well, as reflected in very low attrition rates.

He may be old and slow, but Maslow makes the difference between good and great. He can play on my team any day.

Click here to go to the next learning theory that actually works 

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

Felder-Silverman Learning Style Model

This is number 2 on my list of Learning Theories that Actually Work.

A lot of work has been done and lots more ink has been spilled on the subject of learning preferences
and learning styles. You've got multiple intelligences, right brain/left brain, social/solitary, and many other dichotomies and trichotomies from which to choose. And frankly, this whole area can become a mud pit for learning developers. But you can't just drive around it. People really do learn in very different ways, and if you gear everything to what you happen to think is effective, you're inviting everyone who isn't you, or a lot like you, to tune out.

Some are apples, some are oranges.
Deal with it.

So how do you invite them to tune in? The Felder-Silverman Learning Style Model is a good place to start, and in fact, if you're looking for practical applications, it's also a good place to end. The model covers four important learning differences:

1. Active vs. reflective learning (I want to try it right away vs. let me think about this first)
2. Sensing vs. intuitive learning (Fact vs. feeling)
3. Visual vs. verbal learning (Show me vs. tell me)
4. Sequential vs. global learning (Just give me the steps vs. give me the big picture first)

There are a number of ways to use these variances when building learning. Some adaptive technology platforms, for example, can suss out which way you learn more quickly and then send you more pictures and fewer words, or vice versa. But most of us are working on more ordinary learning platforms, so I will stick to my criteria for "Learning Theories that Actually Work." First criteria:

• It makes sense on the face of it

If you want to understand better how they apply to you, you can take the Solomon-Felder questionnaire to get a clear sense of your own preferences. But you probably know most of this from experience. Some people want to roll up their sleeves and just do something, anything, while others want to meet and plan and think and talk first. The latter read directions; the former don't. Some people want facts and data, while others want to discuss meaning, direction, impact. Some people use words like, "I see," while others say, "I hear you." The former prefer movies, the latter concerts. Some people can't listen to the story you're telling until you assure them everything turns out alright in the end. These are the same people who read the last page of a novel first. These are all learning preferences, and it just makes sense that appealing to each of them appropriately increases their satisfaction, which contributes to involvement, which helps learning and retention.

An example of how this works in real life, outside the classroom and training room... I was in a meeting where a vendor pitched to my CEO. The vendor started with his company's background, setting the usual context, but the CEO interrupted and said, "I don't need all that fluff, just show me how it works." The vendor was clearly put off, but obliged. When the demo ended, the CEO sat back and nodded and said, "So, how old is your company?" From there on he was very interested in "all that fluff."

Recognizing that the CEO was a strongly sequential learner might have kept the vendor from thinking his client's boss was self-absorbed, or just plain rude. And knowing this about himself might have prevented the CEO from acting that way. The point is, this is real and it matters and it just makes sense to address it when constructing learning experiences.

• It has a solid history in research and practice

Felder and Silverman go back to 1988, and Felder and Solomon created a readily-available survey that makes it personal and even more practical. So I've chosen this as safe, solid ground for addressing learning styles. Here's a link to a bit more research on the learning style model.

• It's easy to implement

Choosing to address different learning styles does not require any great investment of time or money. It's really about using what you have wisely.

You can address both active and reflective learners by giving them all something to do and something to think about, and then letting them do whichever they prefer in whichever order. You can address sensing and intuitive learners the same way, being sure to include both hard facts and human consequences. You can create a forum for those who need to talk. You can provide comment boxes. You can ask deeper-meaning questions. And you can let people explore concepts, never being satisfied because you can collect correct/incorrect answers.

Verbal and visual preferences can be addressed by including opportunities for learners to communicate back, or to do their assignments, using either words or images. More and more online and on-the-ground courses allow video or other forms of visual submissions. If that's not practical for you, at least be sure that visual learners have visuals.

The images above were free and clear from  freeimages.com and took about four minutes to find, a few more to format and upload using Keynote. Maybe they aren't the perfect visuals for making the point but, as they say, if you can't do everything at least do something.

If you can get a voice to read the on-screen text, all the better. But remember, writing is already verbal, much more closely linked to auditory learning than visual learning. So if your budget or deadlines require you to choose between recording a voice to read the words aloud and finding or creating images that illustrate the ideas, go with the images.

Lecture Video
(Standard Quality)

By the way, this is why talking head video lectures are tragically ineffective... Visual learners will actually be distracted watching a poorly-lit, badly-framed portrait speak, making it harder for them to absorb the words being said. And even verbal learners will be happier reading a well-written page in their own inner voice, at their own pace, instead of trying to work through the tinny audio and the uhs and ums and pauses and slurping sips from water bottles.

And no, Ted Talks are not "talking heads." They are slickly-produced, carefully designed experiences, created by video professionals. There's a reason you can't just make your own Ted Talk and upload it.

A simple way to address both sequential and global learners is to make sure there is a "next" button and a table of contents readily available, or preferably, both always visible on screen. You should also make the decision not to limit the learners' path by requiring only a single linear progression. Let people go where they will, jump ahead, move backward and forward as they would in a book, as much as the content allows. Global learners who aren't allowed this kind of freedom--at least adult learners--will feel like cattle in a corral. And unlike cattle, they can simply choose to wander off.

• I've tried it, and it works

As the above "easy-to-implement" examples suggest, I've found a number of ways to incorporate these learning styles. What I've found most helpful is to remember that your user interface, whatever it is, either does or does not address these learning differences. Whether it's custom-built or Blackboard, many of these can be addressed by settings on your LMS or instructions to developers.

Early in the online-learning era I was responsible for designing an interface from scratch, for online live video lectures with slides, and used Felder-Silverman. We put a button on the primary screen called "Overview," and made the decision to let learners page ahead to look at upcoming slides (Global/Sequential). We had a button for texting with online experts who were not presenters, so learners could discuss whatever they wanted at any time (Active/ Reflective and Sensing/Intuitive). And we used graphic images and pre-produced videos liberally (Verbal/Visual). We also built in Gagne with buttons labeled "Objectives," "Prior Knowledge," and "Feedback." Of course there is not much call these days to build your own interface, but you can look for the same functionality in whatever you are using. And you can use Felder-Silverman as part of your checklist if you are looking to change your LMS.

In truth, I've gotten to the point that when I'm in an online learning environment, most of these are second-nature. I'm a global learner and need that bird's eye view, but if I don't see that "next" button... I get nervous.

And that may help explain why I present in the following as part of the structure of these posts:

Click here to go to the next theory

Or...

Click in any order you prefer:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

What an interdependent learning ecosystem looks like

A survey conducted by Human Capital Media for CLO Magazine found that over ninety percent of responding organizations partner with a vendor for some area of learning and development. It's an ecosystem out there. And what it looks like is...

...If you're not part of Pac Man, you're part of what's getting chewed up.

Learning Theories That Actually Work

"The development of effective online learning materials should be based on proven and sound learning theories." --Anderson and Elloumi, Athabasca University

Sounds good. Couldn't agree more. But what exactly are the "proven theories" I should be using? Learning science can be strange... after all, a theory by definition is "an idea that is suggested or presented as possibly true but that is not known or proven to be true." (Merriam Webster). So how do you get a "proven theory"? Well, theories are also structures, ways to think about and exploit certain bodies of knowledge. And in some cases, in learning for example, they can be highly practical tools for simplifying what otherwise can be highly complex. So they get used, in the real world, and in that way are "proven to be sound."

I'm not a researcher. I'm not even an academic. I am, however, an educator. Learning products and programs and courses and services I've built, or built the guidelines and teams for building, have taught literally hundreds of thousands of people on all levels of education in all sorts of different fields of content. I've worked in corporate, higher ed, continuing ed, K-12. I've worked internationally and domestically, and in content areas as diverse as sales training and medical education, social work and computer science. I've done a lot of teaching teachers to teach. All of it has been technology-delivered, from web to video to satellite to CD-ROM to videoconference. A lot has been hybrid. So I'm in a bit of a unique position to talk about what works... everywhere.

I also operate in the business world, which is to say I build things that people buy. And people who buy things expect them to work. So I don't and won't waste time and energy and resources trying out the latest, greatest, hottest new learning theory. I have to go with what I know. I've made some mistakes. But there are some things that just never let me down. And that's what I'd like to share here.

So in response to Athabasca's Theory and Practice of Online Learning quoted above, and to the Deans for Impact report on the Science of Learning, I offer my own brief compendium of what I call Learning Theories That Actually Work.

Here's what a learning theory needs in order to qualify for my list:

• It makes sense on the face of it
• It has a solid history in research and practice
• It's easy to implement
• I've tried it, and it works

Below are my top 8. They are in order from "Start Here" to "End There," with "Include This By All Means" in between.

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Individualized Instruction)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Gagne's 9 Events of Instruction

This is number 1 on my list of "Learning Theories that Actually Work."

Gagne's Nine Events of Instruction are a research-based learning model, a proven path for learners that you can use as you design learning experiences for others. Gagne is the granddaddy of learning models, the model for many other learning models.

If you are unfamiliar with the concept of a learning model, you should recognize that every learning event has one, whether the model is any good or not, or whether the instructor even knows it exists or not. The only one that is widely known is, "Tell them what you're going to tell them, tell them, then tell them what you told them." It's not terribly sophisticated, but it manages to hit three of Gagne's nine events (1, 4, and 9). In medicine, it's "Watch one, do one, teach one," (4, 6, and 9). In too many corporate training rooms it's "Death by Powerpoint" (4).

Here are Gagne's Nine:

1. Gain attention
2. Inform the learner of objectives
3. Stimulate the recall of prior learning
4. Present stimulus material (new content)
5. Provide learner guidance
6. Elicit performance
7. Provide feedback
8. Assess performance
9. Enhance retention and transfer

Let's look at Gagne in light of my criteria for being included on a list of Learning Theories That Actually Work:

• It makes sense on the face of it

First, it is a repeatable pattern, a format. That alone provides structure, which means it gives you the ability to assure quality. It can be used as a checklist as you design any learning experience. Second, each step makes sense in its own right. Here are the 9 steps in one readily-identifiable context: 

1. Okay, class, I've got a question for you... (gain attention)

2. When we're done, here's what you'll be able to do... (state objectives)

3. Remember how you did [the previous lesson]? Well, now we're going one better... (prior learning)

4. Here's what you need to know... (new information)

5. Here are some examples... Everyone got it? Any questions? (provide guidance) 

6. Okay, now you give it a try. (elicit performance)

7. That's a good effort, but it will work better if you do this instead. (provide feedback)

8. And yes, this will be on the quiz tomorrow. (assess performance) 

9. And yes, it will also be on the final exam at the end of the term. (enhance retention)

That's an easy classroom example, but you get the point. It will work as a checklist in any learning environment, technical or otherwise.

• It has a solid history in research and practice

Gagne started creating his approaches to teaching and learning for the military in World War II. He's written and been written about extensively since then, and his events, or steps of instruction have been a staple everywhere. Just Google "Gagne's nine events" and you'll find a long list of people, schools, and organizations that are using it, from Northern Illinois University to IEEE to InstructionalDesign.org to the US Embassy in Hungary. If you look hard enough you'll find that research has been done on each step, and on the steps in combination, but it's the sheer weight of its diverse usage over such a long period time that is more than enough to pass this criteria.

• It's easy to implement

This is as easy to use as any other checklist. In order to generate a learning experience for others, you have to devise events and activities and put them in order anyway. Each lesson, each unique module, each seminar, each whatever, should follow a carefully considered pattern, and for the sake of learners, that pattern shouldn't vary dramatically lesson to lesson. That pattern is your learning model. Gagne is a great default. Vary where you want, but vary because you decided to vary not because you forgot something. It costs nothing (except maybe the extra time it always takes to do something right!)

• I've tried it and it works

Early in my career I was more a media producer than a learning developer, and research-backed learning models were an absolute revelation to me. If the steps didn't work, I never would have kept using them--I'd have just added more video, more animation, more of that powerful media stuff everyone likes. But in fact, a good learning model is everything. I once took a job where literally millions of dollars were being spent on high-gloss "TV shows" that were purported to be training. I instituted Gagne, created a development process around it, and dropped a lot of costs (not quality) while improving the learning outcomes dramatically. Gagne improved both my learning and business results, and I was hooked. Now I take him everywhere I go.

Click here to go to the next learning theory that actually works.

Or...
Click in any order:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post

eLearning for external stakeholders? You need product development.

This is enormous.

The 2015 Virtual Learning Trends study from eLearning! Media Group includes this remarkable statistic: Up to 83% of respondents report using virtual methodologies to train external audiences. This is a tectonic shift. There was little to no warning in their 2014 report. eLearning Industry cited their Top Ten stats from last year without mentioned it. This seems like an earthquake already happened, with the tsunami yet to hit.

Okay, that analogy may be over the top, because with a tsunami you just grab the kids and the laptop and run. Here, there are actually new opportunities. This wave can be surfed. But if you're training externals, and you're building your training the way you've always done it, consider this post a friendly alarm. Surf's up!

Here are five consequences of this little statistic, and all of them require action:

1. eLearning has become digital product development

Treating an external audience like an internal one is anathema to product developers. Once you've got people outside the organization--channel partners, dealers, and (gasp!) customers--as an audience, your eLearning is product. It doesn't matter whether they are paying for the learning directly or not, or whether it's bundled in with other fees or services. These audiences have an entirely different definition of success: Their own. And that means...

2. User experience matters

From Utility to Engagement – Beyond Usability (Accenture) 

UX design is a critical component of any product, and has its own world of expertise. Take a look at this article by Alexandra Quevedo, and ask yourself how your training product stacks up. Or doesn't. It's important because...

3. eLearning must now reflect brand promise

If you've got a bit of compliance training to do for an internal audience, you can buy something of reasonable quality off the shelf and tick the box. But if your external stakeholders are logging on to anything whatever, then this is you, your organization, your mission, your values, your promises, out there for the world to judge. Training has become an extension of your brand, and it sends its own messages. The flip side for internal audiences is this: they need the same thing. There are very few topics that don't have an inside/outside component, and so what you train employees to do needs to align with what you're saying externally. All your training therefore needs to be steeped in, to come from, and to clearly reflect, your mission, vision, values, and brand. Then there's this...

4. Legal liability issues now change 

Of course you already have a set of legal do's and don'ts when you're training employees or contractors, but when externals--particularly customers--are involved, you are making product promises. These promises have a different legal weight. And if your training goes to an international audience, the issues become even bigger. Every country has laws that protect customers from false claims or misleading promises. And how many times does your training say or suggest that "if you do this, this good thing will happen"? There are legal protections on education and training, of course, but do you know what they are, and how they apply, where? Your legal costs are going to go up. But that's not necessarily a bad thing, because...

5. Training now needs a measurable Return On Investment. 

Once you've ventured into the realm of learning product, you are firmly in ROI territory. And this is true even if you're "throwing in the training for free." Because in reality, your training just became a part of your organization's COGS, or at least its COS. This in turn means that you should have a seat at the cost/benefit table. You can and should make a case for this investment, and set up both ROI and RAI (Reach And Impact) targets. You're in business. Your eLearning just hit the big time.

These are some of the consequences. There are others, of course. But those five are a very good starting point. And here is one (admittedly catch-all) action required:

Develop a digital product development discipline. 

Add PD expertise. Now. Product development means market research, product planning, UX design, branding, prototyping, legal protection, a business case, ongoing user data, customer service, continuous improvement. Add it now or add it later, but later means you're behind the curve, waiting out the tsunami. Add it now, and catch that wave of opportunity.

Every minute of every day

Presented without comment...

From DOMO.com, with sources listed: bits.blogs.nytimes.com, intel.com, apple.com, dailymail.co.uk, skype.com, statisticbrain.com

Digital product development in the learning space

So, what do you do for a living?

It's the simplest, safest of questions, but every time someone asks it, I nod and smile and buy a little time. It's not that I don't have a nice, concise answer... it's just that the answer doesn't answer, not like "I'm an accountant." I do sometimes go with, "I'm a consultant," but that just postpones the inevitable. The tightest answer is this: "I do digital product development in the learning space." But to most people that's meaningless, like saying I'm a creativity analyst for nonlinear investment properties. Huh? And so I pause a moment in order to assess... just how interested are you, really?

Assuming they are in fact interested, I elaborate a little: See, I create web sites and courses and videos, all kinds of products and services for clients in higher education, schools, continuing ed, publishing, corporate training. For teachers, students, nurses, doctors, lawyers, accountants, car salesmen... you name it. The audiences differ, the clients differ, the content differs, and the products differ, but what I do is always the same... I build digital learning experiences that the target audience needs and wants, while meeting the requirements of educators, technologists, and business people. Oh, right. Polite nod. So... you write software?

Sigh. What I often end up doing is attempting to describe orally this visual. I work in the middle. I bring together all these different tribes... experts from business, education, technology, design. They have their own cultures, language, values, and attitudes toward one another (not always friendly), and someone has to bring all this together into something that works for each of them. This someone has to know how to develop a product through a process. That's me. I don't know the depths of each color, but the product that emerges from all the colors.

The lines get drawn pretty quickly:

• "Look, I need to have final say, because I'm the one who understands the content. And that's what this is all about." 
• "You can't have final say over the technology. I'll tell you how it works, and you pump your content into my structure. We'll be fine." 
• "Pump in the content? Without a good design, they won't like it, and if they don't like it, they won't care about the content or the technology. The design is everything." 
• "Well and good, but what happens when the thing starts losing money? Then nothing else matters. I've got the bottom line, so I've got the final say." 

Without a product developer, one of those people wins that argument. And that is a very bad thing. It leads to a skewed, limping thing in the middle.

So how do you keep someone from winning the argument? The secret is to flip the model, focus it not on the production process or the inputs, but on the audience and their needs. They are the highest authority. Each tribe is ultimately responsible to make sure that their own area of expertise is applied in the service of the product's audience. And that's something they can all agree on.

Here's the same venn diagram, but with the audience at the center. The educator has to make it works for the learner. The technologist needs to be sure the end-user is happy. The designer has to think about the unique audience, whether that's working adults, college seniors, or fourth-graders. And the business person has to make it work for the customer. And in most cases, these are not different people, but different faces of the same person. An online student taking an executive education course, for example, is at the same time customer, learner, and end-user, while being a unique person within a targeted demographic.

As a digital product developer in the learning space, I don't need to know everything about the technology, or the content, or be responsible for the entire business model. But I do need to know the target audience inside and out. I need to know the needs and habits and hopes and dreams of the learner, the end-user, and the customer. My authority, my ability to get all those tribes to work together, comes directly from the audience. I need the data, the research, the proof... because at the end of the day, my work is where audience and product meet.

Not so complex, right? Not at all. Quite interesting. Knowing nod. Have you tried the crab dip? 

Three words that can make your training incredibly powerful

Before I tell you what they are, let me back up a step. Ever been in an organization that struggled with the value of training? Ever seen eyes roll at the mention of investing in media for courses? Ever dropped a bundle on video that fell flat, or had a wow factor but no other obvious impact? If you answered "yes" to any of those questions, you've come to the right post.

Here's the problem at its nub: You know the value of what you do, every day, to contribute to the success of your learners, but the folks upstairs in the C-suite can't see it. And that's where the budget decisions are made. They look at all these other factors--ROI, sales, cost of sales, market share, operational efficiency, strategy, growth. Sure, you can get approval if you can show how you'll keep them from a lawsuit or a social media circus, but otherwise, good luck and make do.

But what if your training had an obvious, positive impact on all the stuff that the C-suits are most concerned about? What if you found yourself standing on the accelerator that drives organizational results? Think it's impossible? It's not. You just have to speak the language of leadership, and take their challenges as your own. You do that by imbuing every aspect of your training with the organizational mission.

Organizational mission? Whoa, you say... mission isn't ROI. Mission is as fluffy as it comes. But think a minute... in your organization, who spends the most time on mission, vision, values? The front line? Middle management? No. It's those same C-suits that cut your last budget. The same C-crew that hammers away on margin also hammers home the mission. Why? Is there some rule that every organization needs a mission statement emblazoned on a poorly-designed poster, strategically positioned to collect coffee spatter in the break room? I submit there is not. So why do they care about that cottony, feathery fluff? The answer is, it's not cottony or feathery to them. It's steel and iron and asphalt. They care about mission because they are wracked by existential terror.

The C-suits care about mission because they are wracked by existential terror.

Why do bean counters count their beans so carefully? Fear. Organizational fear of becoming irrelevant, of being overtaken by the competition, of getting creamed by new technology, of making that strategic misstep that cooks their gooses and opens their books to those bad chapters... 7, 11, and 13. Mission drives vision. Vision drives 5-year plans and strategic direction. And strategy drives budget. Mission is survival.

But let's flip it around to the positive. Mission is also hope, dreams, growth, a united front, all of us singing from the same song sheet, pulling together on the oars, marching to the same drum, a force to be reckoned with. Mission is us, it's our identity. It's who we are as an organization. It's who we are as leaders, managers, and individual contributors. With few exceptions, CEOs live in an either-or world, with screaming success rising up on one side, and crashing failure falling away on the other.

Starting to get the picture? I hope so, because that picture is what I'm talking about... a moving, talking picture that's painting a full-color portrait of screaming success, if writ small. It's being painted in every training, every learning opportunity. It's driving mission. Or at least, it's driving mission if your your training includes "identity-level" objectives. Who we are individually, and who we are together, and what we are becoming... make these a part of every training, woven into the how-to, no matter how rote or menial the training task may seem, and you're making the best possible case for budget. Take this to the bank, or at least to the budget meeting: If you can speak powerfully to mission, you will get funded.

Take this to the bank, or at least the budget meeting: if you can speak powerfully to mission, you will get your funding.

But how do you speak powerfully to mission? How do you move the mission needle in all your training? Now it's time for those three little words. They are: Social Learning Theory. Don't let that word "theory" distract you... everything in learning, no matter how proven or practical, is called a theory. Just the nature of the beast. But the fact is, it has been shown that even one single, powerful example, a role model, will change behaviors. That kind of change doesn't even require reinforcement. One good exposure will do it. You can look it up.

But you won't need a lot of scholarly convincing if you think about moments in your own life when you said, "I want to be like that." If you're like most people, you were watching someone when you said those words. It was a parent, an uncle, a character in a movie or a TV show, or a teacher in front of a classroom. It may even have been a commercial (don't worry, you don't have to admit it), because TV commercials are often designed with a full understanding of Social Learning Theory. "The Most Interesting Man in the World" is an obvious, if tongue-in-cheek example. Marketers do it because it works.

You don't have to be a marketer. You don't have to devise cute and creative videos. You just need to capture good people doing good work in ways that are real, that matter, that tell you they are competent and they care. Role models. People doing the work, captured in ways that let others say, "I want that." These ways can be simple interviews with b-roll. It's not complex. It just has to be authentic. And it must be planned and executed, incorporated as part of your training every day.

Pitch the power of video to demonstrate a thousand mission-moving behaviors.

And it has to be part of your training budget. Next time you pitch your budget, pitch identity. Pitch mission. Pitch the power of video to demonstrate mission-moving behaviors. If every training effort is an opportunity to hammer home who we are, what we are becoming, the kind of company we are... how is this abstract? What is the ROI on the flywheel effect? This is not only mission, vision, and values. It's also passion. And that's the flip side of fear... this is the very passion that drives those C-level execs.

Social Learning Theory

This is number 3 on my list of Learning Theories That Actually Work

The following is authored by Mary Miller at the University of Georgia, and can be found verbatim here (italics added by me):

• This theory suggests that an individual learns attitudes by observing the behaviors of others and modeling or imitating them (McDonald and Kielsmeier, 1970). 

• An observed behavior does not have to be reinforced to be learned (Zimbardo and Leippe, 1991)

• The model "can be presented on film or by television, in a novel, or by other vicarious means" (Martin and Briggs, 1986, p. 28). 

From Questia Blog post: Role Models
http://goo.gl/JnkwWL

• The model must be credible to the target audience (Bednar and Levie, 1993). Credibility is largely a function of expertise and trustworthiness. 

• Observational learning is greater when models are perceived as powerful and/or warm and supportive, and "imitative behavior is more likely when there are multiple models doing the same thing" (Zimbardo and Leippe, 1991, p. 51). 

• While "attitudes formed through direct experience with the attitude object or issue are more predictive of behavior than those formed more indirectly" (Zimbardo and Leippe, 1991, p. 193), "media can be substitutes for many live experiences" (Wetzel et al., 1994, p. 26). Thus, observing a model via video is a viable method of learning a new attitude. 

• For passive learners, instruction delivered by media may facilitate the rapid acquisition of complex affective behaviors more effectively than live demonstrations (McDonald and Kielsmeier, 1970). 

• However, receivers may attend mediated messages less closely than those presented directly, thereby diminishing their effectiveness (Bednar and Levie, 1993).

This easily meets my 4 criteria to be included on a short list of "Learning Theories that Actually Work":

• It makes sense on the face of it

Role models. We get it.

• It has a solid history in research and practice

The theory has weight and a track record. Plus, a large number of the ads you'll see on commercial television use this learning theory, trying to get someone to react in exactly this way. Here's one example. If you watch it and you say, "I want to be like that," you've just experienced it.

• It's easy to implement

Assuming you're doing video already, it involves making the choice to show someone who is really good at something, doing exactly that something. It involves shooting the right activities, and asking the right questions in interviews. It also involves choosing a person to be on-camera whom others will want emulate: Someone who is cool, maybe, or delightful in some way, or fun, or funny, or just powerfully confident. None of this costs an extra dime. Well, maybe a little extra time.

• I've tried it, and it works

People love likeable experts. This has been particularly successful for me in the early courses within degree programs, when we consciously created the image of a successful graduate, and continued to reinforce that image throughout. It improved retention by double digits.

(More on this theory here.)

Click here to go to the next theory.

Or...

Click in any order you prefer:

1. Gagne's 9 Events (Learning Model)
2. Felder-Silverman Learning Styles Model (Global/Sequential, Visual/Verbal)
3. Social Learning Theory (Role Models)
4. Maslow's Hierarchy (Identity-Level Outcomes)
5. Bloom's Taxonomy (Critical Thinking)
6. Active Learning (Discovery, Flipping the Classroom)
7. Metacognition (Self-Awareness)
8. Kirkpatrick's 4 Levels of Evaluation (Outcome Measurement)

Click here to return to the original post