1. Knowledge Fluency
2. PQR – The components of our training and assessment items
3. Adaptive Learning Method
4. Curriculum Building
5. Research Methods
7. Software-as-a-Service Environment and Security
8. Technology Background
DeckChair Learning Systems Inc. is a privately owned corporation that was formed in 2007 to commercialize a revolutionary software assessment and training tool developed at the University of Toronto during the beginning of this century.
Our innovative technology is for Teaching and Training professionals who want to assess, track, and train knowledge fluency and time-critical skills.
Our Product is a Software as a Service (SaaS) online learning system that trains and assesses learners based on their personal learning profiles. At the same time, the system analyses curriculum and content effectiveness.
Our new online technology and automatic adaptive learning methods improve training success and certification rates in all situations.
Unlike other Survey and Test applications, we have assembled an online learning engine that incorporates thinking time, decision time, and reaction times into the learning experience.
We refer to our application as the deckchairtutor© system and it’s all about timing!
1. Knowledge Fluency
“Behavioral fluency is accuracy plus speed, what we recognize in experts as
nearly automatic or ‘second nature’ performance.” 1
Learning is the process by which we not only obtain knowledge but also the process that dictates its retention and the ease with which we can apply it to meaningful and important aspects of our careers and daily lives.
One of the best examples of practical knowledge fluency is that we do not get a driver’s license by simply passing a written test. If you cannot apply the ‘rules of the road’ in a timely manner when you are behind the wheel, the consequences can be fatal. We therefore must pass the road test – a classic knowledge fluent assessment environment!
There is an intricate relationship between knowledge and an individual’s ability to apply it in real world scenarios. Knowledge makes up the building blocks of our decision-making and our decisions shape our behaviors. In turn the results or consequences of those behaviors create the very essence of our lives.
Whether the situation is playing in a jazz combo, officiating a hockey game, speaking a foreign language, answering questions pertaining to financial products and services, performing triage, or simply writing a test, masterful performance is never slow and hesitant.
As we forge ahead in the second decade of the new millennium, with the adjunct of powerful personal computers, we have the opportunity to readily assess and train each individual in a timed environment enabling us to promote knowledge fluency in a real way. We must see this as a new tool in our teachers’ toolkit to help every learner develop confidence in applying the skills and knowledge they acquire.
Carl Binder’s extensive research over the past 4 decades points to 3 broad categories where training knowledge fluency has a major impact:
• Fluency is linked to improved retention or maintenance of skills and knowledge.
• Fluency improves attention span or resistance to distraction.
• Fluency in prerequisite skills or knowledge supports application or transfer of new learning to more advanced or complex performance. 1
The primary functionality of the deckchairtutor© system is its unique software design that tracks all browser behaviors and collects these measurements right down to each and every mouseclick. The program is your online coach helping you attain your best possible outcomes in the least amount of time.
So what makes the deckchairtutor© system so different?
The problem for educators when trying to assess knowledge fluency is based on the way knowledge assessment has been traditionally presented in our industrial/historical education system. Paper and pencil tests give a test taker all the information, questions, and possible answers at the same time. Remember the instruction to, “Read the entire test before answering any of the questions.”
In this scenario we have two critical issues:
i) there is no way of discovering how much time an individual spends on each of the questions let alone on each of the elements of a question and,
ii) there is no way of parsing review time from the initial exposure to each question or its elements which typically interferes with assessing the real reaction time to any given problem or scenario.
Knowledge fluency in the past has been decried to mean, “How much can you get right in X amount of time.”
Current online questionnaires, surveys and test programs have followed suit and have merely changed the venue, not the science of evaluation.
We thought better of that and we have deconstructed the problem,
or question, depending on your viewpoint.
2. PQR – The components of our training and assessment items
It seemed so simple to us and is one of the tenets of our patent pending learning method.
All questions in the deckchairtutor© system are constructed to have a series of individual elements comprised of Problem/Presentation(P) fields, Question(Q) fields, and Response(R) fields. These three elements (PQR) can have different types of content and be combined in a variety of ways to create and deliver learning experiences in a truly revolutionary fashion.
By manipulating how these components are presented (we call it the DISPLAY MODE*) we are able to track how much time each learner spends on each of the elements brought into a learning or test assignment.
This gives the system the ability to measure and take into account;
• reading speed,
• information retrieval speed, and
• decision-making speed.
By combining these critical speeds with accuracy measurements we have discovered a really flexible way of scoring knowledge fluency. We call it Skillscore* and Skillscores are used in all aspects of the deckchairtutor© system to determine how each and every individual is progressing and accomplishing learning objectives.
There is a rich pool of information to be gathered when we can measure the response time for each of the question’s elements and this information provides a robust research and authoring opportunity for the world of education that heretofore has not been addressed in the prior art of our industry.
Measuring a Learner’s Skill Profile
Here’s a brief description of how we measure Knowledge Fluency. Skillscores are made up of the following,
Time Scores (TS)
• Authors and Instructors choose the Critical Time to be measured.
• Range Values (in seconds): Any Mouseclick interim to Total Question Time
Accuracy Scores (AS)
• Right or wrong but the system can also handle partial marks and penalties for
• Range Values: 0 to 1
These 2 scores are then “weighted” before calculating Skillscores with proprietary algorithms. TS and AS score weighting can range from 0% to 100% but their combined weighting must always equal 100%. This adjustment gives Authors and Researchers the flexibility to determine which of the 2 fluency components is most important in any given question item.
By default the system weights time and accuracy equally at 50% – 50%. These measurements are then entered into a historical performance algorithm and Skillscores are created ranging from 0 to 1000. The deckchairtutor© system is entirely based on these Skillscores and the ability to manipulate their components.
“Understanding fluency, measuring it and applying methods to produce fluent skills and knowledge can be the crucial difference in making training more cost-effective.” 2
We have also built in a powerful method for applying Skillscores to accelerate learning on an individual basis.
3. Adaptive Learning Method
The next step in the deckchairtutor© system is to apply our inventive Adaptive Learning Method* which is based on our Knowledge Fluency calculations, not just accuracy scores. Student performance measures (Skillscores) are stored in the deckchairtutor© system database to allow the software to track performance, model the learner’s preferences as they interact with the system, and guide students through their assignments providing help along the way.
The adaptive learning features customize each student’s curriculum depending on performance minimums and other criteria set by the course instructor and the learning goals selected by the student. By employing adaptive learning features the software can respond to the individual needs of the student therefore streamlining valuable study time.
What is adaptive learning?
Adaptive Learning has many meanings in the computer-assisted learning world, and essentially captures the notion that the software automatically presents educational or testing material in a way that is sensitive to the immediate and/or long term needs of the learner. It takes away the “one to many” broadcast model of the past century.
This can be based on the student’s response to the current question or scenario (i.e., on incorrect responses the system provides feedback or provides a branch to a review page), and on an analysis of the student’s previous performance history (i.e., scores on previous assignments), on their learning styles (i.e., visual vs. verbal, risk tolerance, goal setting) and the performance criteria set by the course designers. All of these considerations have been built into the deckchairtutor© system. In a world where the biggest barrier to learning is “fear” the system provides an online opportunity where rehearsal and practice are guided in a non-threatening environment always available when learners are ready to learn.
The adaptive features can apply at the question level (how difficult should the next item be for that student?), or at the task level (have they learned enough to move on?). As more data is collected for that student, the software refines a model of their current skill level and adjusts the itinerary on a per use basis.
At the heart of the deckchairtutor© system there are proprietary algorithms
that automatically guide individuals at their own pace.
In many courses, whether they are in corporate or academic settings, the curriculum seeks to cover a set of pre-defined, organized content that in some cases must be mastered in a particular order.
For example, one strategy would be to run skill assessments before teaching or training to measure the range of skills for an incoming group of students. Weak topics can then be identified;
• for the whole group impacting classroom activities, or
• in each student’s skill profile impacting the intelligent algorithm guidance.
It is then possible to measure the same skill sets after training to see if those topics have improved or been mastered. This is an important method for understanding the ROI of training programs. It is also important to note that particular skills sets can be identified and independently improved when skill assessments are properly utilized and the results can be collected and summarized in a matter of moments.
Adaptive Learning Algorithms and Data Mining
The training and reporting algorithms onboard intelligently adjust sets of exercises for each student to target their skill deficiencies. The system:
• tracks performance on a detailed skill profile per user, and per class and generates performance reports pointing to areas that need more study
• guides students through their assignments to complete the curriculum in an optimal fashion and
• provides guided feedback when students make mistakes or are in need of fluency improvement
The deckchairtutor© system’s Curriculum Pacing interface allows all materials in the database to be organized into a learning taxonomy as a set of content areas that can coincide with the Chapters and Sections of text books. The content can then be further divided into subtopics, and difficulty levels (easy, middle, hard etc). The instructor or author uses this structure to set up Curriculum Pacing by defining;
• mastery criteria (speed and accuracy benchmarks),
• maintenance criteria (practice known facts), and
• quick release criteria (advance students when performance predicts success
on the whole set).
In summary, Skillscores, representing knowledge fluency, are compiled into a distribution curve that is used to; advance curriculum, maintain our on-the-fly custom learning training techniques, and create real-time feedback of learning progress.
4. Curriculum Building
When working with clients our staff, or the course authors, upload, edit, and create their questions and training materials in the deckchairtutor© system’s database. Online courses will vary in the size and complexity depending on the knowledge domain being taught and the level and reliability of the required performance measure outcomes. Stakeholders and their Subject Matter Experts will also determine what measurements need to be made and how quickly skills need to be developed.
Authors organize curriculum content in question banks and media libraries to design, test, and deliver curriculum content in a scheduled itinerary of diagnostic, teaching, and assessment tasks. Additional tags are employed to;
• map specific concepts or topic areas that may overlap traditional Chapters and
• define the difficulty levels for each learning item as determined by the author or empirically from performance histories.
Thus, the curriculum has a set of rules for how content items are grouped into task assignments, and how assignments are grouped into possible itineraries to complete the learning objectives.
In general we suggest following the principles of the ADDIE method of curriculum building. Assess, Design, Deploy, Implement, and Evaluate.
When a research driven instructional design is required for the attainment of more complex skills learning we suggest using the 4C/ID Model (vanMerrienBoer and deCroock, 2002). For more detailed information on these models please request our paper on the subject entitled: “DCLS Curriculum Building”.
There are many different question types incorporated in the deckchairtutor© system that are used for evaluations.
• Built in Standards – Multiple Choice, Fill-in-the-Blank (exact string match), Multiple Blanks, Paragraph (to be marked with an instructor account)
• Event Detection – analyze anticipatory and reactive behaviors
• Interactive – On screen object manipulation and matching. Target Identification.
• If you can imagine it, we can put it on the screen!
The following is a very nice summary of the basics of building effective assignments in a Knowledge Fluency Environment;
1. Provide sufficient practice opportunities. Fluency only comes with practice. Few training programs in any field (besides music or athletics) provide sufficient practice opportunities to attain fluency. It’s important that your training design includes enough examples and exercises on critical skills and functions for trainees to practice to the point of fluency.
2. Build fluency in small chunks. Until one is fluent, it’s better to work in brief practice sessions than for extended, tiring periods. Likewise, it’s far easier to build fluency on a small set of commands or functions than to try to become fluent on a larger number at once. Therefore, it’s advisable to design brief repeated practice activities (one to five minutes each) with short breaks in between. Try to define small sets of skills that can by themselves accomplish real tasks or subtasks; then let users become good at them before adding more. Dysfluent skills or information create weak foundations for further learning.
3. Establish fluency goals. Because fluency is accuracy plus speed, you can set time criteria for any skill or information process. For each practice task, measure how long it takes five or six experts to complete and provide trainees with practice goals based on that information (e.g., duplicating a simple spreadsheet or memo format in less than four minutes).
4. Encourage learners to measure their performance. It’s easy to time brief practice activities and count errors afterwards—users can do it for themselves. Self-monitoring provides continuous feedback and lets learners compete with themselves. Lack of measured progress signals them to seek more help.
5. Let trainees grow at their own pace. We can define fluency objectively, in terms of time limits plus accuracy levels. But we can’t predict how long it will take a given individual to reach fluency. Invariably, some attain fluency faster than others, so individualized practice is best. Self-paced learning with explicit fluency goals at each step gives users the best shot at motivated, satisfying growth.3
There’s so much learning that goes on after a 100%.
5. Research Methods
Using the deckchairtutor© system’s performance database, researchers are able to apply a variety of data mining strategies to implement and test evolving algorithms to improve their approaches to predicting student success, to create more efficient itineraries, and to build better training and learning content that will save even more time for instructors and students.
The system also supplies the means for researchers to design and deploy this type of educational research automatically over the internet by providing tools that systematically;
• evaluate the effectiveness of adaptive learning interventions,
• optimize training performance – evaluate help, branching, and hint
• fine-tune the content materials as data is collected.
Some common research design principals that are easily implemented in the deckchairtutor© system include;
• PreTest/PostTest - measure skills before and after training, evaluate training
• Test/Teach/Test cycles - track skill development from start to final exam or
certification, tailor specific teaching interventions based on diagnostic results both online and in the classroom
• Rehearse & Drill - training to improve speed and accuracy of decisions.
• Practice, Practice, Practice.
Do you know what your students or workforce don’t know?
A variety of reports are available for students in a course, for instructors or course managers, and for the authors of the curriculum.
Student User Reports and Performance Data Export
When a task has been completed students can immediately view their results online. This option can be turned on and off by the Instructor. The Results page shows
• the overall grade in the assignment,
• the average time spent on each question,
• the grade and average time in each TOPIC concept covered in the assignment.
The Instructor also has the option to allow students to export raw data from any assignment they have completed. This feature can be used in a variety of ways in a blended curriculum to further enhance learning activities.
Instructor Gradebooks (Task, Module, and Courses)
Assignment report summaries for all students in a course, showing the average speed and accuracy in each of the target topics or learning objectives. The report is sorted by default from best to worst so that we can track the averages of the top, middle, and bottom sections of the class. (Other custom sorting is available on screen) These reports quickly identify areas where more instruction and/or study would be most effective.
Content and Curriculum Analytics
Question Statistical Reports are available to both instructors and authors so that question quality, difficulty, and classification are verified over time. These are the results that help analyse and build equivalent test and learning assignments.
Other tools allow the raw data from the click-by-click event logs to be exported for research analyses and data mining purposes.
More detailed descriptions and examples of reporting are available in our ““DCLS Users Manual”.
7. Software-as-a-Service Environment and Security
In a learner centered environment we need not concern ourselves with “cheating” per se. The learner has chosen to study a curriculum and the onus is on them to be honest in taking exams and certificate tests. Truly secure testing, as always, would have to be achieved by students attending a test centre where identification and proctoring are in place. However, 2 important testing methods are built into the system to help with providing a unique test environment for every individual, whether they are at a personal computer or in a classroom setting;
• random question delivery
- this helps prevent students from sharing information by question number
- the actual answer options in multiple choice questions can also be
presented randomly which further impedes info sharing.
• question equivalency
- when question items have a robust history of use they actually have a
Skillscore of their own.
- When enough data is available to validate Skillscores in the same Topic,
Sub-Topic, Stage or Set, it is possible to build equivalent tests made
from totally different sets of questions.
8. Technology Background
Dr. Jeffrey Graham has managed and taught at the psychology department’s computer lab since 1992 at the University of Toronto at Mississauga (UTM) where approximately 1600 first year students attend - every year. The challenges of dealing with these large student classes has led to the invention of this innovative testing and teaching database system.
With a PHD in cognition, learning has always been a major focus of Dr. Graham’s career. With a vast array of Apple computers available in the UTM Introductory Psychology Laboratory, the environment is ripe for developing effective computer based learning tools.
The first highly successful product developed within the psychology department was a standalone application known as Sniffy the Virtual Rat4. This program is designed to teach the most basic aspects of learning, classical and operant conditioning, and has been heralded as an ethical alternative to using live animals in the laboratory. Sniffy, published by Wadsworth / Cengage Learning, has been one of the hottest selling psychology learning tools since it was first released in the early nineties.
During the same time frame, Mr. Allan Sura, began learning creative computer applications, internet technologies, and developing standalone testing applications. Allan joined the Sniffy team and was able to add realistic video capture to the Sniffy project creating a new and fun interface for version 2.0 and up.
Their collaboration quickly led to other computer based learning discussion and, with the advent of incredibly powerful personal computers, they began developing an online assessment and training tool. Drawing on many aspects of the psychology of learning, particularly knowledge fluency and adaptive learning, the deckchairtutor project was born.
At the heart of the deckchairtutor© system, almost every mouseclick
in your browser application is tracked and timed.
With keen persistence, a few grants, and a ton of help from undergrads, the deckchairtutor project took shape and was showcased at a couple of University of Toronto symposiums. The deckchairtutor Intellectual Property was selected by the University of Toronto’s Innovations and Partnerships Office to assist in its commercial development.
With monetary support from the Ontario Centres of Excellence Market Readiness program and guidance from the Mars Discovery District, commercialization of the DeckChairTutor software began at the beginning of 2008.
The deckchairtutor© system is a true measurement tool for Knowledge Fluency and “Skill” acquisition.
“Building fluency in product knowledge can produce immediate effects in the performance of sales personnel and on the bottom line.” 1
Automatic adaptive training, maintenance and remediation on a “per user/student” basis are impossible to deploy to large numbers of learners without a computational database driven system. The deckchairtutor© system has been designed to handle these situations and perform automatic tutoring with a minimal amount of instructor settings. Teaching & Training curriculum analyses are also built into the system with the ability to validate equivalent training and assessment content objects.
deckchairtutor© helps train people to “Think Fast”
DeckChair Learning Systems Inc.
1. Binder, C., (1989) Fluent Product Knowledge: Application in the Financial Services Industry. Performance and Instruction (Issue - February 1989)
2. Binder, C., (1990) Closing the Confidence Gap. Training (Issue – September 1990)
3. Binder, C., (Sept 1987) Computing "Fluency" and Productivity. Managing End-User Computing
4. Alloway, T., Wilson, G., & Graham, D.J. (2011). Sniffy: the virtual rat lite version 3.0 (with CD Rom). Belmont, Wadsworth-Cengage Learning.