Contents
1.
Knowledge Fluency
2. PQR – The components of our training
and assessment items
3.
Adaptive Learning Method
4.
Curriculum Building
5.
Research Methods
6. Reporting
7.
Software-as-a-Service Environment and Security
8.
Technology Background
9.
Summary
DeckChair Learning Systems Inc.
is a privately owned corporation that was formed in 2007 to commercialize a
revolutionary software assessment and training tool developed at the University of Toronto during the
beginning of this century.
Our
innovative technology is for Teaching
and Training professionals who want to assess, track, and train knowledge
fluency and time-critical skills.
Our
Product is a Software as a Service (SaaS) online
learning system that trains and assesses learners based on their personal learning
profiles. At the same time, the system analyses curriculum and content
effectiveness.
Our
new online technology and automatic adaptive learning methods improve training
success and certification rates in all situations.
Unlike
other Survey and Test applications, we have assembled an online learning engine
that incorporates thinking time, decision time, and reaction times into the
learning experience.
We
refer to our application as the deckchairtutor©
system and it’s all about timing!
1. Knowledge Fluency
“Behavioral fluency is accuracy
plus speed, what we recognize in experts as
nearly automatic or
‘second nature’ performance.” 1
Learning is the process by which we not
only obtain knowledge but also the process that dictates its retention and the
ease with which we can apply it to meaningful and important aspects of our
careers and daily lives.
One of the best examples of practical
knowledge fluency is that we do not get a driver’s license by simply passing a
written test. If you cannot apply the
‘rules of the road’ in a timely manner when you are behind the wheel, the
consequences can be fatal. We therefore
must pass the road test – a classic knowledge fluent assessment environment!
There is an intricate relationship
between knowledge and an individual’s ability to apply it in real world
scenarios. Knowledge makes up the
building blocks of our decision-making and our decisions shape our behaviors.
In turn the results or consequences of those behaviors create the very essence
of our lives.
Whether the situation is playing in a
jazz combo, officiating a hockey game, speaking a
foreign language, answering questions pertaining to financial products and
services, performing triage, or simply writing a test, masterful performance is
never slow and hesitant.
As we forge ahead in the second decade
of the new millennium, with the adjunct of powerful personal computers, we have
the opportunity to readily assess and train each individual in a timed
environment enabling us to promote knowledge fluency in a real way. We must see this as a new tool in our
teachers’ toolkit to help every learner develop confidence in applying the
skills and knowledge they acquire.
Carl Binder’s extensive research over
the past 4 decades points to 3 broad categories where training knowledge
fluency has a major impact:
•
Fluency is linked to improved retention or maintenance of skills and knowledge.
•
Fluency improves attention span or resistance to distraction.
•
Fluency in prerequisite skills or knowledge supports application or transfer of
new learning to more advanced or complex performance.
1
The primary functionality of the deckchairtutor© system is its unique software design
that tracks all browser behaviors and collects these measurements right down to
each and every mouseclick. The program is your online coach helping you
attain your best possible outcomes in the least amount of time.
So what makes the deckchairtutor© system so different?
The problem for educators when trying
to assess knowledge fluency is based on the way knowledge assessment has been
traditionally presented in our industrial/historical education system. Paper and pencil tests give a test taker all
the information, questions, and possible answers at the same time. Remember the instruction to, “Read the
entire test before answering any of the questions.”
In this scenario we have two critical
issues:
i)
there is no way of discovering how much time an individual spends on each of
the questions let alone on each of the elements of a question and,
ii) there is no way of parsing review
time from the initial exposure to each question or its elements which typically
interferes with assessing the real reaction time to any given problem or
scenario.
Knowledge fluency in the past has been
decried to mean, “How much can you get right in X amount of time.”
Current online questionnaires, surveys
and test programs have followed suit and have merely changed the venue, not the
science of evaluation.
We thought better of that and we have
deconstructed the problem,
or
question, depending on your viewpoint.
2.
PQR – The components of our training and assessment items
It seemed so simple to us and
is one of the tenets of our patent pending learning method.
All questions in the deckchairtutor© system
are constructed to have a series of individual
elements comprised of Problem/Presentation(P) fields, Question(Q) fields, and
Response(R) fields. These three elements (PQR) can have different types of
content and be combined in a variety of ways to create and deliver learning
experiences in a truly revolutionary fashion.
By manipulating how these
components are presented (we call it the DISPLAY MODE*) we are able to track
how much time each learner spends on each of the elements brought into a
learning or test assignment.
This gives the system the
ability to measure and take into account;
• reading speed,
• information
retrieval speed, and
• decision-making
speed.
By combining these critical
speeds with accuracy measurements we have discovered a really flexible way of
scoring knowledge fluency. We call it Skillscore* and Skillscores are
used in all aspects of the deckchairtutor© system to determine how each and every
individual is progressing and accomplishing learning objectives.
There is a rich pool of information to
be gathered when we can measure the response time for each of the question’s
elements and this information provides a robust research and authoring
opportunity for the world of education that heretofore has not been addressed in
the prior art of our industry.
Measuring
a Learner’s Skill Profile
Here’s a brief description of how we
measure Knowledge Fluency. Skillscores are made up of the following,
Time Scores (TS)
• Authors and Instructors choose the
Critical Time to be measured.
• Range Values (in seconds): Any Mouseclick
interim to Total Question Time
Accuracy Scores (AS)
• Right or wrong but the system can
also handle partial marks and penalties for
wrong answers
• Range Values: 0 to 1
These 2 scores are then “weighted”
before calculating Skillscores with proprietary
algorithms. TS and AS
score weighting can range from 0% to 100% but their combined weighting must
always equal 100%. This
adjustment gives Authors and Researchers the flexibility to determine which of
the 2 fluency components is most important in any given question item.
By default the system weights time and
accuracy equally at 50% – 50%. These
measurements are then entered into a historical performance algorithm and Skillscores are created ranging from 0 to 1000. The deckchairtutor© system is entirely based on these Skillscores and the ability to manipulate their components.
“Understanding fluency,
measuring it and applying methods to produce fluent skills and knowledge can be
the crucial difference in making training more cost-effective.” 2
We have also built in a powerful method
for applying Skillscores to accelerate learning on an
individual basis.
3.
Adaptive Learning Method
The next step in the deckchairtutor©
system is to apply our inventive Adaptive Learning Method* which is based
on our Knowledge Fluency calculations, not just accuracy scores. Student performance measures (Skillscores) are stored in the deckchairtutor© system database to allow the software
to track performance, model the learner’s preferences as they interact with the
system, and guide students through their assignments providing help along the
way.
The adaptive learning features
customize each student’s curriculum depending on performance minimums and other
criteria set by the course instructor and the learning goals selected by the
student. By employing adaptive learning features the software can respond to
the individual needs of the student therefore streamlining valuable study time.
What
is adaptive learning?
Adaptive Learning has many meanings in
the computer-assisted learning world, and essentially captures the notion that
the software automatically presents educational or testing material in a way
that is sensitive to the immediate and/or long term needs of the learner. It takes away the “one to many” broadcast
model of the past century.
This can be based on the student’s
response to the current question or scenario (i.e., on incorrect responses the
system provides feedback or provides a branch to a review page), and on an
analysis of the student’s previous performance history (i.e., scores on
previous assignments), on their learning styles (i.e., visual vs. verbal, risk
tolerance, goal setting) and the performance criteria set by the course
designers. All of these considerations
have been built into the deckchairtutor©
system. In a world where the biggest
barrier to learning is “fear” the system
provides an online opportunity where rehearsal and practice are guided in a
non-threatening environment always available when learners are ready to learn.
The adaptive features can apply at the
question level (how difficult should the next item be for that student?), or at
the task level (have they learned enough to move on?). As more data is collected for that student,
the software refines a model of their current skill level and adjusts the
itinerary on a per use basis.
At the heart of the deckchairtutor© system there are proprietary algorithms
that automatically
guide individuals at their own pace.
In many courses, whether they are in
corporate or academic settings, the curriculum seeks to cover a set of
pre-defined, organized content that in some cases must be mastered in a
particular order.
For example, one strategy would be to
run skill assessments before teaching or training to measure the range of
skills for an incoming group of students.
Weak topics can then be identified;
• for the
whole group impacting classroom activities, or
• in each
student’s skill profile impacting the intelligent algorithm guidance.
It is then possible to measure the same
skill sets after training to see if those topics have improved or been
mastered. This is an important method
for understanding the ROI of training programs.
It is also important to note that particular skills sets can be
identified and independently improved when skill assessments are properly
utilized and the results can be collected and summarized in a matter of
moments.
Adaptive
Learning Algorithms and Data Mining
The training and reporting algorithms
onboard intelligently adjust sets of exercises for each student to target their
skill deficiencies. The system:
• tracks performance on a detailed
skill profile per user, and per class and generates performance reports
pointing to areas that need more study
• guides students through their
assignments to complete the curriculum in an optimal fashion and
• provides guided feedback when
students make mistakes or are in need of fluency improvement
The deckchairtutor© system’s Curriculum Pacing interface
allows all materials in the database to be organized into a learning taxonomy
as a set of content areas that can coincide with the Chapters and Sections of
text books. The content can then be
further divided into subtopics, and difficulty levels (easy, middle, hard etc). The instructor
or author uses this structure to set up Curriculum Pacing by defining;
• mastery
criteria (speed and accuracy benchmarks),
• maintenance
criteria (practice known facts), and
• quick
release criteria (advance students when performance predicts success
on the whole set).
In summary, Skillscores,
representing knowledge fluency, are compiled into a distribution curve that is
used to; advance curriculum, maintain our on-the-fly custom learning training
techniques, and create real-time feedback of learning progress.
4. Curriculum
Building
When working with clients
our staff, or the course authors, upload, edit, and create their questions and
training materials in the deckchairtutor© system’s database. Online courses will vary in the size and
complexity depending on the knowledge domain being taught and the level and
reliability of the required performance measure outcomes. Stakeholders and their Subject Matter Experts
will also determine what measurements need to be made and how quickly skills
need to be developed.
Authors organize curriculum content in question banks and media
libraries to design, test, and deliver curriculum content in a scheduled
itinerary of diagnostic, teaching, and assessment tasks. Additional tags are
employed to;
• map specific concepts or topic areas that may overlap
traditional Chapters and
Sections classification,
• define the difficulty levels for each learning item as
determined by the author or empirically from performance histories.
Thus, the curriculum has a set of rules for how content items are
grouped into task assignments, and how assignments are grouped into possible
itineraries to complete the learning objectives.
In
general we suggest following the principles of the ADDIE method of curriculum building. Assess,
Design, Deploy, Implement, and Evaluate.
When
a research driven instructional design is required for the attainment of more
complex skills learning we suggest using the 4C/ID Model (vanMerrienBoer and deCroock, 2002). For
more detailed information on these models please request our paper on the
subject entitled: “DCLS Curriculum Building”.
There are many different question types
incorporated in the deckchairtutor© system that are used for evaluations.
Question
Types
• Built in Standards – Multiple Choice,
Fill-in-the-Blank (exact string match), Multiple Blanks, Paragraph
(to be marked with an instructor account)
• Event Detection – analyze
anticipatory and reactive behaviors
•
Interactive – On screen object manipulation and matching. Target
Identification.
•
If you can imagine it, we can put it on the screen!
The
following is a very nice summary of the basics of building effective
assignments in a Knowledge Fluency Environment;
1.
Provide sufficient practice opportunities. Fluency only comes with practice.
Few training programs in any field (besides music or athletics) provide
sufficient practice opportunities to attain fluency. It’s important that your
training design includes enough examples and exercises on critical skills and
functions for trainees to practice to the point of fluency.
2.
Build fluency in small chunks. Until one is fluent, it’s better to work in
brief practice sessions than for extended, tiring periods. Likewise, it’s far
easier to build fluency on a small set of commands or functions than to try to
become fluent on a larger number at once. Therefore, it’s advisable to design
brief repeated practice activities (one to five minutes each) with short breaks
in between. Try to define small sets of skills that can by themselves
accomplish real tasks or subtasks; then let users become good at them before
adding more. Dysfluent skills or information create
weak foundations for further learning.
3.
Establish fluency goals. Because fluency is accuracy plus speed, you can set
time criteria for any skill or information process. For each practice task,
measure how long it takes five or six experts to complete and provide trainees
with practice goals based on that information (e.g., duplicating a simple
spreadsheet or memo format in less than four minutes).
4.
Encourage learners to measure their performance. It’s easy to time brief
practice activities and count errors afterwards—users can do it for themselves.
Self-monitoring provides continuous feedback and lets learners compete with
themselves. Lack of measured progress signals them to seek more help.
5.
Let trainees grow at their own pace. We can define
fluency objectively, in terms of time limits plus accuracy levels. But we can’t
predict how long it will take a given individual to reach fluency. Invariably,
some attain fluency faster than others, so individualized practice is best.
Self-paced learning with explicit fluency goals at each step gives users the
best shot at motivated, satisfying growth.3
There’s
so much learning that goes on after a 100%.
5. Research Methods
Using
the deckchairtutor© system’s performance database,
researchers are able to apply a variety of data mining strategies to implement
and test evolving algorithms to improve their approaches to predicting student
success, to create more efficient itineraries, and to build better training and
learning content that will save even more time for instructors and students.
The
system also supplies the means for researchers to design and deploy this type
of educational research automatically over the internet by providing tools that systematically;
• evaluate the effectiveness of adaptive learning interventions,
• optimize training performance – evaluate help, branching, and hint
effectiveness
• fine-tune the content materials as data is collected.
Some common research design principals that are easily implemented in the deckchairtutor© system include;
• PreTest/PostTest - measure skills before and after
training, evaluate training
content effectiveness
• Test/Teach/Test cycles - track skill
development from start to final exam or
certification, tailor specific teaching
interventions based on diagnostic results both online and in the classroom
• Rehearse & Drill - training to
improve speed and accuracy of decisions.
• Practice, Practice, Practice.
Do
you know what your students or workforce don’t know?
6. Reporting
A
variety of reports are available for students in a course, for instructors or
course managers, and for the authors of the curriculum.
Student User Reports and Performance
Data Export
When
a task has been completed students can immediately view their results online.
This option can be turned on and off by the Instructor. The Results page shows
•
the overall grade in the assignment,
•
the average time spent on each question,
•
the grade and average time in each TOPIC concept covered in the assignment.
The
Instructor also has the option to allow students to export raw data from any
assignment they have completed. This
feature can be used in a variety of ways in a blended curriculum to further
enhance learning activities.
Instructor Gradebooks (Task, Module,
and Courses)
Assignment
report summaries for all students in a course, showing the average speed and
accuracy in each of the target topics or learning objectives. The report is sorted by default from best to
worst so that we can track the averages of the top, middle, and bottom sections
of the class. (Other custom sorting is
available on screen) These reports
quickly identify areas where more instruction and/or study would be most
effective.
Content
and Curriculum Analytics
Question
Statistical Reports are available to both instructors and authors so that
question quality, difficulty, and classification are verified over time. These are the results that help analyse and
build equivalent test and learning assignments.
Other
tools allow the raw data from the click-by-click event logs to be exported for
research analyses and data mining purposes.
More
detailed descriptions and examples of reporting are available in our ““DCLS Users Manual”.
7.
Software-as-a-Service Environment and Security
In a learner centered environment we
need not concern ourselves with “cheating” per se. The learner has chosen to study a curriculum
and the onus is on them to be honest in taking exams and certificate
tests. Truly secure testing, as always,
would have to be achieved by students attending a test centre
where identification and proctoring are in place. However, 2 important testing methods are
built into the system to help with providing a unique test environment for
every individual, whether they are at a personal computer or in a classroom
setting;
• random
question delivery
-
this helps prevent students from sharing information
by question number
-
the actual answer options in multiple choice questions
can also be
presented randomly
which further impedes info sharing.
• question
equivalency
-
when question items have a robust history of use they
actually have a
Skillscore of their own.
-
When enough data is available to validate Skillscores
in the same Topic,
Sub-Topic, Stage or Set, it is possible to build equivalent
tests made
from totally
different sets of questions.
Data
Protection
The deckchairtutor© system
is delivered online through an SSL Certificate environment from our secure
servers located in Toronto, Canada. DeckChair Learning Systems Inc. has a strict privacy policy
and under no circumstances shares any information about individuals using the
system. Our Privacy Policy is posted on
our website at:
http://deckchairlearning.com/PrivacyPolicy.html
8. Technology
Background
Dr. Jeffrey Graham has managed and
taught at the psychology department’s computer lab since 1992 at the University
of Toronto at Mississauga (UTM) where approximately 1600 first year students
attend - every year. The challenges of
dealing with these large student classes has led to the invention of this
innovative testing and teaching database system.
With a PHD in cognition, learning has
always been a major focus of Dr. Graham’s career. With a vast array of Apple computers
available in the UTM Introductory Psychology Laboratory, the environment is
ripe for developing effective computer based learning tools.
The first highly successful product
developed within the psychology department was a standalone application known
as Sniffy the Virtual Rat4. This program is designed to teach the most
basic aspects of learning, classical and operant conditioning, and has been
heralded as an ethical alternative to using live animals in the
laboratory. Sniffy, published by
Wadsworth / Cengage Learning, has been one of the hottest selling psychology
learning tools since it was first released in the early nineties.
During the same time frame, Mr. Allan Sura, began learning creative computer applications,
internet technologies, and developing standalone testing applications. Allan joined the Sniffy team and was able to
add realistic video capture to the Sniffy project creating a new and fun
interface for version 2.0 and up.
Their collaboration quickly led to
other computer based learning discussion and, with the advent of incredibly
powerful personal computers, they began developing an online assessment and
training tool. Drawing on many aspects
of the psychology of learning, particularly knowledge fluency and adaptive
learning, the deckchairtutor project was born.
At
the heart of the deckchairtutor© system, almost every
mouseclick
in
your browser application is tracked and timed.
With keen persistence, a few grants,
and a ton of help from undergrads, the deckchairtutor
project took shape and was showcased at a couple of University of Toronto
symposiums. The deckchairtutor
Intellectual Property was selected by the University of Toronto’s Innovations and Partnerships Office to
assist in its commercial development.
With monetary support from the Ontario Centres
of Excellence Market Readiness program and guidance from the Mars Discovery District,
commercialization of the DeckChairTutor software
began at the beginning of 2008.
9. Summary
The deckchairtutor© system is a true measurement tool for
Knowledge Fluency and “Skill” acquisition.
“Building
fluency in product knowledge can produce immediate effects in the performance
of sales personnel and on the bottom line.” 1
Automatic adaptive training,
maintenance and remediation on a “per user/student” basis are impossible to
deploy to large numbers of learners without a computational database driven
system. The deckchairtutor©
system has been designed to handle
these situations and perform automatic tutoring with a minimal amount of
instructor settings. Teaching & Training curriculum analyses are also built
into the system with the ability to validate equivalent training and assessment
content objects.
deckchairtutor©
helps train people to
“Think Fast”
DeckChair
Learning Systems Inc.
*patent
pending
References
1.
Binder, C., (1989)
Fluent Product Knowledge: Application in the Financial Services
Industry. Performance and Instruction (Issue - February 1989)
2.
Binder, C., (1990)
Closing the Confidence Gap.
Training (Issue – September
1990)
3.
Binder, C., (Sept 1987) Computing "Fluency" and Productivity. Managing End-User Computing
4. Alloway, T.,
Wilson, G., & Graham, D.J. (2011). Sniffy: the virtual rat lite version
3.0 (with CD Rom). Belmont, Wadsworth-Cengage Learning.