Test Administration: Strategies and Scenarios
Do you have what it takes to manage every possible situation and minimize testing irregularities?
Audience: K-12 teachers, administrators, and other school staff.
Responsibilities: Instructional Design, Action Mapping (Needs Analysis), Storyboarding, Visual Design, Mockups, eLearning Development, Subject Matter Expert (SME)
Tools Used: Articulate Storyline 360, Freepik.com, Google Docs and Slides, Mindmeister, Vyond, Articulate Rise.
KPI: 1. The total percentage of tested students will be between 95%-100% of total student enrollment. 2. After completion of testing, misadministration and testing irregularities will be reported and minimized. 3. Teachers will follow the appropriate procedures for any testing irregularities or misadministration with 100% accuracy.
The problem: Test administration training is an annual requirement set by the NC Department of Public Instruction. At the end of each school year and before testing begins, all school staff are required to attend training sessions on testing procedures, accommodations, and the code of ethics. NCDPI utilizes a train the trainer method of sharing the information to each public, private, or charter school (K-12). Each school based test coordinator is tasked with the responsibility of creating the content and facilitating the instruction.
One issue with the process is that the information is identical every year. There is a lack of consistent resources to use from school to school, besides the test administrator and proctor manuals. Consequently, the redundant information and traditional lecture approach to covering policies and procedures leads to low staff motivation and engagement with the content.
The solution: After careful analysis with the district testing coordinator (SME), I determined that the problem was instructional design related, and caused by the redundancy of required annual test training, regardless of past experience or performance. To solve the engagement and motivation problem I proposed an immersive scenario-based eLearning experience to help teachers and staff practice using low-pressure decision making during testing situations in a real-world setting.
Keeping testing policies and procedures in mind, the learning experience would help make a connection between using decision making strategies and establishing a positive and productive testing environment. Developing a new approach to the content while leveraging adult learning theory will aid in the improvement of teacher and staff motivation and engagement with the content.
Analysis
(Articulate Storyline, Vyond)
(Articulate Rise 360)
My Design Process
Guided by the ADDIE and SAM instructional design models, I iterated on each step of my process until I was confident with the look, feel, and content of the learning experience. Before reaching full development, my process included creating an action map, drafting a storyboard, designing visual mockups, and building out an interactive prototype.
Action Map
Using Cathy Moore’s strategic action mapping process, I acted as the subject matter expert (SME) and drew from my experience as a Testing Coordinator.
After creating the business goal, I worked as the SME to brainstorm the observable tasks and behaviors the learners would need to engage in and avoid to help meet the performance goal. Next, I analyzed all brainstormed tasks and behaviors and categorized them into actions and sub-actions.
The next step in my process was to carefully evaluate each action and consider which would help contribute most to the business goal. This evaluation revealed five high-priority actions that became the focus of the content in the learning experience and had a heavy influence on my design and development process. These actions can be seen branching out from the possible testing situations in the action map below.
Action mapping is a crucial part of my design process because it allows me to ensure I'm meeting the client's needs while also keeping the content in the learning experience meaningful and relevant to the learner.
By setting a measurable performance goal, I can keep all stakeholders concentrated on business performance and success rather than just the “information” they need to know. Prioritizing actions allows me to reduce cognitive load and help the learner encode and retain the most crucial information concerning the performance goal.
Text Based Storyboard
Once I identified all high-priority observable tasks and behaviors through action mapping, I drafted and iterated on a text-based storyboard to serve as a blueprint for my final product.
To best help the learners meet the established performance goal, I prioritized the main features I wanted to include in the experience. It was important that the final product was scenario-driven and based on selected actions in the action map to avoid information overload.
Another priority was to include learner feedback based on selected choices. Through a mix of text and visual design, the learner would receive positive reinforcement for correct decisions. The learner would watch the consequence scenario play out for incorrect choices, learn from it, and try again.
Once I prioritized the critical assets of the course, I had a clear path for completing the storyboard in a way that would best help engage the learner in the scenario. I also included a mentor to help the audience access helpful information when needed.
The storyboard was an essential part of my design process because it’s where I embedded substantial instructional practice and learning theory to make the experience ideal for the learners.
Interactive Prototype
Creating the interactive prototype fully brought the experience to life. Using Articulate Storyline 360 with Vyond integration, I developed a prototype to collect feedback on functionality and development. I included several interactive slides up until the first scenario-based question. I added in triggers, animation, and a success meter. Overall, the interactive prototype was and will continue to be an essential step of my development process as it allows me to improve the experience based on user testing and feedback from the clients and community.
Once I was fully confident with the look and functionality of the prototype, I developed the rest of the scenario-based slides and iterated on them until all animations, visuals, and triggers flowed perfectly together. I also made a few changes based on user testing and feedback to enhance the learning experience. This learning experience helps engage the learner through visual scenes that immerse them in the scenario. Instead of just being told what might happen next, they can experience it by being shown what happens due to their correct or incorrect choices.
Implementation and Evaluation Feedback
"I think it helps that you are simulating the test taking environment and having us react to different scenarios and make choices, it really is a good warm up for the actual test."
"I think that the multiple choice options you have given are good because they represent relevant issues that may arise. It familiarizes us with them and teaches us what to look out for, and it highlights the bad decisions administrators might actually make."
"Really no negative feedback to give because learning about it this way is a lot more useful and engaging than listening to someone explain it to us while we all just sit and listen."
"I like that when you introduce the Ask Samantha button, the user has to test it and use it to move to the next slide."
"This exercise forces us to learn how to respond, and highlights that the key factor is that we need to alert the testing coordinators if there is a possible misadministration."
"I like that you made it into a game (you have to win 5 trophies), it makes it more interesting."
Subject Matter Expert (SME) Collaboration
"I really liked this activity. My one suggestion is to make the options less obvious. IE during the cell phone example add a choice such as stop testing, call the student out, etc. Then, in the description, you could explain this is not correct because it could unnecessarily escalate the situation. Just a suggestion, but overall this is great. Awesome job!"
Samantha Villegas
Director of Curriculum and Instruction
Kestrel Heights Charter School
