USABILITY TESTING SUMMARY

IMPROVING THE EFFECTIVENESS OF CORPORATE INTERNAL COMMUNICATIONS

Researched & Conducted by: Mina C. Hami

Assistance and Guidance from: Karen Weldon

INTRODUCTION

The purpose of this research is to determine if the existing publication. The 9 Environmental Regulations You Should Know About pamphlet, can be improved. This document was initially created to provide leadership with key information regarding nine of the top environmental regulations the utility should be aware of. The pamphlet was small, pocketbook and served as a quick “cheat sheet” for leadership in high level meetings. Since initial use, the pamphlet had been recently updated and re-printed for distribution a second time, and an added request for distribution to all employees in addition to using it as a source for video production. With all the production costs involved, for the purpose of this product I was interested in determining how helpful it was at conveying information to employees. For this research I used the technique of usability testing, which focuses around user-centered interactions to evaluate a product. Although usability testing is typically used for software, application, device and web testing, it can also be applied to gauge the effectiveness of written communications and design.

 

 

PUBLICATIONS TESTED

Two Documents Tested

Document #1: The 9 Environmental Regulations You Should Know About Pamphlet

Was originally created for managers, directors, and executives to use use as a reference guide about environmental regulations that affect the utility, Consumers Energy. The pamphlet was initially distributed February 4th, 2016 at the Managers and Directors Meeting, as well as, emailed to roughly 200 managers and directors at Consumers Energy. This publication has received some attention with executives because of its small, accordion fold design that allows for easy carrying. But the design also had issues, including small fonts that hindered readability, and complex content covering scientific information that might not be understood by all employees. In addition, the reading level was too high at nearly a college senior level (a good target is about 7th or 8th grade reading level).


Document #2: We’re Committed to Environment Compliance & Sustainability Comic

The second document titled, We’re Committed to Environment Compliance & Sustainability Comic, is a revised version of the first document, presented in a comic book inspired format. This document omits the heavy, scientific text that the pamphlet contained. Summary information was also added in the header area to provide the reader with a quick overview (example section  of comic presented to the right. For this summary the image is black & white - the test poster was in full color)


 

METHODOLOGY

The following approaches were used during this project:

  • Baseline survey

  • Focus groups

  • Knowledge testing

  • Reading comprehension

Baseline Survey

A baseline survey was conducted to collect feedback from the initial recipients of the pamphlet in three areas: (1) Did they remember receiving it?, (2) Did they read it?, (3) Did they share it with others on their team (as originally instructed to do when they received the document).

Surveys were sent out to 131 managers and directors using Survey Monkey. Results concluded that 69% of individuals remembered receiving the pamphlet. Of those only 58% said they read it and only 35% shared it with others. Recipients were also asked to provide initial feedback and suggestions for improvement. Some responses stated that they “appreciated the concise explanation of the regulations” and that is was “very useful,” other suggestions called for the pamphlet to be simpler and use layman’s language in order to understand the text heavy information. Some thought the pamphlet also did not present a reason why an employee would need to know the information to function in their day to day job role.


 

Focus Groups

Focus groups are a great way  to gain general feeling about, and reactions with communication materials from your audience; it is a great tool in acquiring information and honest feedback.

Participants were selected randomly, from a cross section of the company. Each session had up to  6 people in attendance and sessions were held at Consumers Energy Headquarters (OEP), in various conference rooms. The focus groups were scheduled for about 30 minutes each. We wanted to gain feedback on both documents, including: what the audience liked and disliked, which document would they most likely share with their teams, and how might each document be improved. Participants were encouraged to share their honest points of view; both negative and positive.

During each session, participants were given each document (separately) for consideration. Participants were asked to read through each document and provide various levels of feedback--from first impressions, to general preferences, to specific answers to questions about the content (i.e., was information easy to find in each document?). There was an observable difference in reaction of the participants  given the pamphlet as compared to the comic. The pamphlet brought many looks of confusion, while the comic generally brought smiles. Participants thought the pamphlet was “wall of black text” and it “would take about 20-30 minutes to digest and understand.”

The comic was more favored with the focus group participants —84% preferred the comic as compared to the pamphlet. It did, however, have some negative feedback. Some of the participants felt it was too graphic to the point of being distracting. One individual stated, “I still would not be able to catch this in a drive by, even if it was on digital signage.” Participants who favored the poster thought it was positive, reactionary and innovative. While both pieces had significant visual differences, both still did not clearly state the “so what factor,” “how does it relate to my job,” nor did each piece have a call to action


 

Knowledge Testing

Beyond gathering audience opinions, we wanted to determine if one approach to the content would prove better at helping readers understand and retain the information they read. Under the usability umbrella, knowledge testing is another great tool in acquiring this information.

 

QuizMaker software was used to create two 10-question quizzes and then uploaded the quizzes to the Learning Management System (LMS) to enabling the research team to see test scores. Test subjects were provided either the pamphlet or the comic to read and refer to while taking the quiz online, examinations took about 10-15 minutes to complete.

A majority of participants were video recorded using TechSmith’s Morae Recording software. Moare Recorder is a usability software that allows the researcher to gain powerful insight into the user’s experience, the recording feature allows researchers to observe user interactions, analyze results, and share findings.

Participants who were given the comic to read and complete the test scored an average of 96%, while those given the pamphlet scored an average of 85%.

Reading Comprehension

Just because something is easy to read doesn’t necessarily mean your audience will understand it. To gauge reading comprehension, we used a simplified version of the Cloze Test, which helps determine whether our target audience actually understands the content’s meaning. This exercise helps gauge whether sentences are written simply enough for readers to “guess” at missing words.

Participants were given sentences from other publications. Each sentence had 2 to 3 words missing, and blank spaces for participants to fill in words that made the most sense to complete the sentence. Results concluded higher accuracy rate with participants that were given the comic than questions associated with the pamphlet.

CONCLUSION

A successful communication piece would effectively inform readers how to answer, locate and understand given information. I conducted this research to determine if the existing publication, The 9 Environmental Regulations You Should Know About Pamphlet, can be improved and how helpful it was in conveying information to employees. Based on the data the pamphlet can be improved. A different style demonstrated that employees were able to effectively read and comprehend the material by 11%. Although the new style fared better with participants, many questioned why the information should be communicated. Some participants asked, “Is this supposed to make me feel smarter?” “Do I need to know this information to write a letter to my State Representative” “How does it help me in my job?” Therefore, it can be concluded that although the pamphlet can be improved, it should not be mass produced for all employees.

© 2020 by Mina C. Hami