Monday, November 23, 2009

Focus Groups


Our latest class assignment (4.1) was to use the Level 3 Evaluation Method handout from class to discuss focus group strengths, weaknesses, and general comments associated with using focus groups.


I led a focus group for computer hardware, software, and firmware issues that the engineering community needed to resolve. The group was made up of service managers, field engineers, remote engineers, technical writers, graphics artists, researchers, supply chain, and logistics people.


The focus group was virtual with people attending from all over the world meeting once per month for one hour. The amount of information gathered at these meetings was tremendous and it led to product quality improvements and customer satisfaction resolutions that no other team inside HP was able to identify because we were close to the customer and worked with our customer base weekly listening to their concerns and implementing solutions.


Strengths

Weaknesses

Comments

Can be specifically focused on a particular topic.

Can be too focused on a topic and thus lose the “forest for the trees”. The danger is in getting too in depth to the point of having useful, meaningful information.

Ensure that the topics picked lend themselves to open discussion so that others are willing to share their expertise.

Used to create troubleshooting content for a product after product release.

Can take several months before enough useful data is collected to create troubleshooting scenarios that are realistic and useful to the student.

Should ensure some type of troubleshooting material is in place upon project release so that the service technicians have something to worth with in the field.

Can obtain methodologies used by others in the field. Quickly discover what others are encountering when solving system-wide problems.

It is possible that certain geographies experience different problems than another region. Make note of such anomalies and define if they are geographic, cultural, economic, fiscal, or other concerns.

It can be too easy to take the first instance of a problem and assume that all other problems behave the same way.

Provides subject matter experts who can help you write your material. This makes the content relevant and applicable to actual problems with the product.

SMEs time can be difficult to maintain on a consistent basis. Must have a backup plan in place for such a concern.

Overall, focus groups can be a rich source of information BUT they can also lead a team in a direction completely out of character for the problem that needs solving.

Sunday, November 15, 2009

Kirkpatrick Levels of Evaluation

I'm back in class this month taking "Evaluating Training Interventions" facilitated by Don Jackson. Don asked us post our comments to the classroom discussion board (assignment 1.3) relating our experience with evaluation instruments to which I wrote:

I am framing my response to this question from the viewpoint of the levels of evaluation (Kirkpatrick’s levels 1 – 4). Early in my career I placed way too much emphasis on the level 1 evaluation. I allowed the negative comment of one or two students to override the 12 – 14 positive comments from the rest of the students. Another funny thing about level 1 evaluations, we would ask about the comfort level in the room including chairs, tables, temperature and other silly items but never corrected any complaints that had about the environmental conditions. So why ask questions you don’t intend to use to correct a situation that needs correction? I was told because it’s always been done that way.

We rarely did a pre-test evaluation and when we did, even if the student flunked the pre-test they were still allowed in the class. Again, why conduct the test if you aren’t going to enforce it? There never was a post-test for any of the classes I taught so I created my own. I considered it a level 2 evaluation since I taught computer hardware, firmware, and software topics and would ask the student to demonstrate their new-found skills by troubleshooting the systems I bugged.

The real evaluation takes place at level 3 where you check back with the student a few months later to see if they are using the skills attained from the classroom and building upon those skills in the work environment. Sadly, few companies ever reach level 3 or level 4 evaluations. Statistically speaking (this comes from a previous post I wrote):

Level 1 - between 72 to 89 percent of organizations use Level 1 (reaction)
Level 2 - between 29 to 32 percent of organizations use Level 2 (learning)
Level 3 - between 11 to 12 percent of organizations use Level 3 (behavior)
Level 4 - between 0 to 3 percent of organizations use Level 4 (results)

Use the NIOSH Training Evaluation Tips for further explanation of this assignment:
http://www.cdc.gov/niosh/mining/topics/training/trainingevaluationtips.htm

By the way, Kirkpatrick has revised the four levels and now has an updated white paper, April 2009, available for download from the internet. If you are unable to find it, let me know and I can make a copy available to you.

Thanks for reading,

Kevin Love, MBA (e-Business)
Training and Development Specialist
Dallas / Fort Worth Area
Google Voice: 817-778-8540
Personal Email: kevindlove@hotmail.com
LinkedIn: http://www.linkedin.com/in/kevinlove