Recent Publications

1. Barbara Wise, Ron Cole, Sarel van Vuuren, Scott Schwartz, Lynn Snyder, Nattawut Ngampatipatpong, Jariya Tuantranont, and Bryan Pellom (in press), "Learning to Read with a Virtual Tutor: Foundations to Literacy", In Kinzer, C. & Verhoeven, L. (Eds) Interactive Literacy Education: Facilitating literacy learning environments through technology.

2. Ronald Cole, Sarel van Vuuren, Bryan Pellom, Kadri Hacioglu, Jiyong Ma, Javier Movellan, Scott Schwartz, David Wade-Stein, Wayne Ward, Jie Yan, "Perceptive Animated Interfaces: First Steps Toward a New Paradigm for Human Computer Interaction", Proceedings of the IEEE: Special Issue on Multimodal Human Computer Interface, Aug. 2003. 

Project Overview

The CSLR Reading Project is a component of the Colorado Literacy Tutor, a collaboration between universities and public schools that aims to improve student achievement through development of educational software that helps students learn to read and comprehend text. The Colorado Literacy Tutor consists of two main projects, Foundations to Literacy™ , or FtL™ , a comprehensive, scientifically-based and individualized reading program that uses a virtual tutor to teach students to read and learn from text, and Summary Street, a program that uses Latent Semantic Analysis to grade students' summaries of text and provides feedback that helps them revise and improve their summaries.

A key objective of the CSLR Reading Project is to improve human communication technologies through basic research, leading to the invention of perceptive animated agents- lifelike computer characters that speak, emote and gesture, and engage learners in natural face-to-face conversations in learning tasks, much like effective and sensitive teachers. By embedding perceptive animated agents in immersive, multimedia learning environments, we hope to improve student achievement by teaching foundation reading skills (e.g., phonological awareness, sounding out words), fluent reading and comprehension. Inventing perceptive animated agents involves integrating research advances in areas of speech recognition, natural language understanding, computer vision and computer animation. If we are successful, future learning tools will incorporate animated agents that behave much like sensitive and effective teachers in specific application domains.

This web site is intended to provide an overview of activities conducted by CSLR and its collaborators within the Colorado Literacy Tutor. For additional information on the Colorado Literacy Tutor, including the Summary Street comprehension training program , please visit the Colorado Literacy Tutor and Summary Street Web sites.

Sponsors

The CSLR Reading Project is funded by grants to the University of Colorado from the National Science Foundation's Information Technology Research (ITR) Program, the Interagency Educational Research Initiative (a joint program of the National Science Foundation, Department of Education and National Institutes of Health) and the Coleman Institute for Cognitive Disabilities.

The following research grants provide direct support for the CSLR Reading Project:

NSF/ITR: REC-0115419 - Kintsch, W., Landauer T., Caccamise, D., Cole, R., "ITR/PE: Latent Semantic Analysis Theory and Technology," $2,400,000, NSF, 09/01/01 - 08/31/06.

NSF/IERI: EIA-0121201 - Kintsch, W., Caccamise, D., Cole, R., Olson, R., Snyder, L., "IERI: Scalable and Sustainable Technologies for Reading Instruction and Assessment," $5,997,404, NSF, 07/01/01 - 06/26/06.

NSF/ITR: IIS-0086107 - Cole, R., van Santen, J., Movellan, J., "ITR: Creating the Next Generation of Intelligent Animated Conversational Agents," $4,000,000, NSF, 09/01/00 - 08/31/05.

NSF/IERI : 1R01HD-44276.01Cole, R., Barker, L., Schwartz S., Snyder, L., Wise, B., "IERI: Scaling up Reading Tutors," $1,000,000.00, NIH. 9/27/02 - 9/30/04.

Coleman Institute for Cognitive Disabilities: Schwartz, S., Cole, R., Wise, B., Doxas, I., "Coleman Foundation Grant: Participatory Design for Creating Computer Based Learning Tools," $8184, Coleman.

NSF ITR Supplement Research Experience for Undergrads: REU/ITR: Creating the Next Generation of Intelligent Animated Conversational Agents (supplement to NSF 0086107), $45,500.00, NSF, 06/01/02 - 8/31/03.

NSF ITR Supplement Research Experience for Teachers: RET/ITR: Creating the Next Generation of Intelligent Animated Conversational Agents (supplement to NSF 0086107) $40,000.00, NSF, 09/01/02 - 08/31/03.

Institutions, Project Staff and Project Activities

University of Colorado:

Project staff at the University of Colorado include Ron Cole (PI), Lecia Barker, Kathy Bunch, Donna Caccamise, Kathy Garvin-Doxas, Ma Jiyong, Nattawut Ngampatipatpong, Scott Schwartz, Lynn Snyder, Taylor Struemph, Sarel van Vuuren, Jariya Tuantranont, Barb Wise and Jessica Yan.

The Center for Spoken Language Research (CSLR) contributes to the Colorado Literacy Tutor by:

• Developing and testing digital environments for authoring, deploying and using reading tutors, and developing and testing toolkits for designing learning applications within these environments.
• Researching and developing human communication technologies in areas of speech recognition, language understanding, computer animation and dialogue modeling.
• Designing and evaluating specific learning tools (in collaboration with co-PI Lynn Snyder, Professor in the Department of Speech, Language and Hearing Sciences at CU).
• Conducting research to identify barriers to scaling up the education software developed during the project, and proposing solutions to overcome these barriers (in collaboration with Lecia Barker and Kathy Garvin-Doxas at ATLAS, the Alliance for Technology Learning and Society at CU).

Boulder Valley School District:

Jean Riordan, Jana McMillan and Scott Schwartz work closely with CSLR and the Institute for Cognitive Science to plan, coordinate and facilitate interactions with public schools. Activities include arranging meetings with administrators, facilitating data collection and participatory design activities, planning and facilitating software deployment and assessment activities, and conducting research on scaling educational software to different education environments.

OGI School of Engineering Center for Spoken Language Understanding (Oregon Health & Sciences University):

Co-PI Jan van Santen and Jacques de Villiers are incorporating new technologies, such as the CU Animate system into the CSLU Toolkit, for use in language training applications with exceptional children.

Introduction

We envision a new generation of intelligent and embodied animated agents that engage users in natural face-to-face conversational interaction in learning and language training tasks. An intelligent agent is one that mimics the actions of real persons and behaves intelligently in the context of a specific application or task domain. An embodied agent is one that resembles a real person. The goal of our work is to invent intelligent animated agents that engage children in face-to-face conversational interaction to help them learn to speak, understand, read and write language. In our software applications we are creating, the animated agents will interact with both English- and Spanish-speaking children with speech and/or reading difficulties.

Perceptive intelligent animated agents "live" in Interactive Books and Tutors. Together, Books and Tutors are designed to provide a comprehensive reading program. Interactive Books will help children learn to recognize words, read fluently and understand what they read. They provide an environment for learning from pre-readers (who can have stories narrated to them by animated characters, and then engage in dialogues with the characters to assess and train comprehension) to advanced readers who can read stories and then receive comprehension training (work that will be conducted under a recently awarded IERI grant). Interactive books will help indicate what foundational reading skills are lacking or weak, and engage learners in an individualized sequence of exercises that will assess and teach these skills. Our Tutors integrate fully with our Interactive Books. Children learn and practice foundational skills in completely individualized ways using these tutors. A full Assessment of the educational efficacy of the interactive books and tutors is ongoing. An on-line assessment of a child's reading abilities is also available that can place students into the appropriate instructional levels of tutors and books as well as assess their progress. Teachers, students, and parents all help in a Participatory Design process, to ensure that our Books and Tutors meet the needs of the people who use them. Researchers continue to collect auditory and visual data for our Speech Corpus Development, so that all our programs will accurately recognize and interpret the speech of children of all ages.

Interactive Books

Our goal in designing interactive books is to provide a powerful and immersive environment for learning new knowledge and acquiring new skills. In this environment, students interact with intelligent animated agents using natural communication skills. Advanced human communication technologies—including speech recognition, natural language processing, speech generation, character animation, computer vision and dialogue modeling—enable students to engage in natural face-to-face spoken dialogue interaction with intelligent, perceptive animated characters that will behave much like sensitive, caring and effective teachers and mentors. This software will be freely available to educators and university researchers worldwide.

Interactive books are designed to serve several functions. These are:

• Learning tools designed to help children learn to read fluently and effortlessly, to develop effective strategies to comprehend what they read, and to apply knowledge gained from reading to real world situations.
• Authoring tools for designing effective and immersive learning experiences. These experiences include dialogue interaction with animated characters in a wide variety of contexts (e.g., comprehension training, sounding out words), having students reading aloud with real time feedback on the pronunciation of each word (provided by an automatic speech recognition system) narration of stories by animated characters, and the ability to create or revise change their own animated stories.
• Test beds
  • Researching and developing perceptive animated interfaces—systems that engage users in natural face-to-face conversational interaction with intelligent animated characters;
  • Researching, developing and evaluating core technologies in learning tasks; including speech recognition, speech generation, natural language processing, computer vision, computer animation, and dialogue modeling and management;
  • Conducting education research in reading instruction, second language learning, knowledge acquisition in content areas, comprehension training, etc.

Features of Interactive Books

Interactive books incorporate leading edge speech, language, computer vision and character animation technologies to provide engaging and immersive learning experiences. We describe the main features of interactive books in terms of the learning experiences they enable and the technologies that enable them.

        Animated Speech. Three dimensional animated computer characters produce natural speech, a wide variety of facial expressions and emotions, and natural body movements. They can narrate the book or engage the user in conversational interaction. For example, the animated character Ms. Gurney can narrate an entire story while individual words, sentences or paragraphs are optionally highlighted. Or, Ms. Gurney can narrate portions of the text, while individual characters displayed in illustrations “come alive” to speak their parts. In addition to telling stories, Ms. Gurney can pronounce individual words (or syllables in words) that are selected by the student, or provide hints to help the student decode the word. Finally, Ms. Gurney and other animated characters can engage the student in dialogues to train and test comprehension.

In addition to producing accurate visible speech with associated facial expressions and gestures, animated characters can provide visual feedback to students during learning and conversational interaction. For example, the character can nod while the student is speaking or look puzzled if the system does not recognize what the student is saying. The character can also provide visual feedback and reinforcement, in the form of a head nod, smile, “thumbs up” or other gestures when the student provides correct answers. We will argue below that visual components of conversational interaction are critically important to effective communication and learning.

        Speech Recognition. There are many ways in which speech recognition is used in interactive books. Speech recognition enables students to read aloud while receiving real time feedback about their word recognition and pronunciation accuracy. In this application, the recognizer determines if words are pronounced accurately and provides immediate visual feedback about correct and incorrect pronunciations. Speech recognition is also used during spoken dialogue interaction with animated characters. The animated agent may ask the student to provide specific answers to questions (“What did Mary eat for breakfast”), or elicit more complex utterances that require inferences (“Why did Mary get angry?). Or, the animated agent may ask the student to produce a spoken summary of what she just read. In each of these cases—reading aloud, providing specific answers to questions, producing open-ended responses; and summarizing stories—different speech recognition systems are used to process the speech.

        Natural Language Processing. Utterances that are transcribed by the speech recognition system or typed by the student are processed to interpret semantic content. When students converse with animated agents in spoken (or typed) dialogues, robust semantic parsing techniques are used to assign word strings to semantic frames and interpret the users’ intended meaning. When students produce spoken or typed summaries of stories, Latent Semantic Analysis is used to analyze the summaries and provide feedback about their quality and conciseness.

        Computer Vision. Computer vision plays a key role in interactive books, enabling the system to locate and identify the student, track the student’s movements and interpret his or her visual behaviors. An accurate face tracking system, developed by Dr. Javier Movellan and his colleagues at UC San Diego, has been incorporated into interactive books. It is now possible for the animated agent to orient to the student, and move its head or eyes to follow the student’s movements. Once the location of the student’s face is determined, face recognition algorithms can be used to identify individuals, to recognize visible speech, and to interpret facial expressions and gestures. We note that interactive books are an ideal test bed for research on the integration of auditory and visual information to improve human computer interaction. For example, auditory and visual information can be combined to improve speaker identification, recognition and understanding of speech, and to more accurately interpret the cognitive and emotional state of the student.

        Face-to-face conversations. Face-to-face conversational interaction with animated characters occurs through real-time integration of speech recognition, natural language understanding, speech generation, facial animation, computer vision and dialogue modeling technologies. Below we describe how these modules communicate with each other within the Galaxy architecture.

Authoring Tools

Interactive books are accompanied by a set of powerful authoring tools for designing interactive books, or more generally, learning applications. These tools enable authors to create stories that are displayed in electronic form as books with text, illustrations, and animation.

Authors can design animated productions of the text, in which different characters narrate their parts, much like characters in an animated movie. A library of facial expressions, gestures and animation sequences are available to animate characters, giving designers great flexibility and control of how characters behave during interactions with the user. Authors can also construct structured or unconstrained (mixed-initiative) dialogues between animated agents and students to test comprehension or to encourage students to think deeply about the events within a story.

Thus, Interactive Books provide a powerful test bed for research and development of applications that can be used to understand the principles and implementation of good learning tools and communication systems.

Theoretical Foundations

Research has shown that learning is optimized when students are actively engaged in interesting and challenging learning tasks with individual tutors or in small groups of students. In these environments, teachers can design and customize learning tasks to the interests and needs of each student, observe and analyze each student’s behavior, and provide individualized feedback and guidance. Individualized instruction is so effective because the teacher can apply all of her experience and knowledge to the learning process while keeping the student engaged, focused and motivated.

A primary assumption underlying our work is that it is possible to enable such optimized learning experiences by inventing intelligent animated agents that behave like sensitive and effective teachers in specific learning tasks. Inventing such agents requires research to improve auditory and visual recognition and synthesis technologies so the animated agents can recognize and interpret students’ behaviors and respond to these behaviors like master teachers, and research to understand and model the behaviors of effective teachers. The seminal work of Reeves and Nass (1996) suggests that, if such embodied animated agents are designed well through appropriate research, learners will interact with these agents just as they would with real teachers.

Our work on learning and comprehension in interactive books is motivated by the theory of text comprehension developed by Walter Kintsch and his colleagues (Kintsch, 1998; van Dijk & Kintsch, 1983). This cognitive theory is guiding the design of interactive books, the structure and content of the stories we are developing, and the manner in which animated characters interact with students to stimulate both the acquisition of new knowledge and the application of this knowledge to new problems and domains. The idea is that that one learns good comprehension strategies and writing skills by using them multiple times in multiple ways with support, guidance and appropriate feedback.

The relationship between interactive books and the Kintsch’s constructive theory of text comprehension is synergistic: while the theory guides the design, content and use of interactive books, the books provides provide an ideal platform for new research on comprehension training and a vehicle for widespread deployment and evaluation of a comprehension training program—the Colorado Literacy Tutor—being developed at the Institute of Cognitive Science under the direction of Walter Kintsch, supported by grants the Department of Education, NIH and NSF. This project aims to train students to comprehend what they read by analyzing students’ written or spoken summaries of stories and providing feedback through Latent Semantic Analysis (LSA) techniques. In the typical LSA comprehension training paradigm, students read expositions or stories, and then provide typed summaries. The summaries are analyzed using LSA, and students are provided with feedback about their summaries.

Interactive books provide a powerful platform for research on text comprehension. An obvious advantage is the ability to automate the LSA-based comprehension training process—the text can be presented for reading, students can type summaries directly into a window within the book, student summaries can be analyzed in real time, and feedback can be presented to students in near real time. Beyond this obvious benefit, interactive books enable us to conduct research to extend comprehension training from the current paradigm of students reading text, typing a summary, and receiving visual feedback of their summary, to natural dialogue interaction with intelligent animated agents. The conversational agent can ask questions to help students discover and integrate knowledge that was not included in their summaries. Moreover, Interactive Books extend the scope and power of LSA-based comprehension training to students who cannot yet read or type. For these students, animated agents can narrate the stories. The animated teacher or coach can then instruct the student to summarize the story in his or her own words. The student’s speech is then transcribed automatically by the speech recognition system and analyzed using LSA. The animated character can then converse with the student to provide feedback about the summary and guide the student—by asking questions to which good answers require key information that was not included in the summary, by again narrating the text that contains the key information, and so forth.

The development of stories for presenting knowledge in interactive books is guided also by research conducted by Carolyn Sumners and her colleagues at the Houston Museum for Natural Science. The idea behind this work is that students will acquire knowledge from books more effectively if the knowledge is integrated effectively into the plots of interesting stories with interesting characters. If students identify with the characters and become engaged in the stories, and if acquiring new knowledge is integral to story, then students will acquire this knowledge to some extent.

To test this idea, Sumners and her team developed a set of nine stories at each grade level for grades one through six. Each of the illustrated stories, which centered on the adventures of children about the same age as the students, integrated science into the plots. Students in classrooms in the Houston Independent School District (HISD) who read these stories had greater learning gains in science at all grade levels then students in control groups who were taught science by traditional means. These stories are now in widespread use in HISD; students in every second grade class read these stories as part of an integrated science-learning program that includes a visit to the Houston Museum of Natural Science. We are planning to use this approach in our work, both for the Houston Science Stories (which we have received permission to use as interactive books) and in stories we are writing in collaboration with Dan DiStefano, a professional storywriter.

The design of well-organized stories, in which new information is integrated into the plot, combined with good questions presented by animated agents that require students to think about what they have read, provides a powerful foundation for comprehension training.

Research Challenges

Human Communication technologies have matured to the point where it is now possible to conceptualize, develop and investigate computer systems that interact with people much like people interact with each other. We envision a new generation of human communication systems in which intelligent animated agents engage users in natural face-to-face conversational interaction. By integrating computer vision into spoken dialogue systems, the animated agent will interpret the user’s auditory and visual behaviors to more accurately infer their user’s intentions and emotional state. The animated agent will orient to the user, engage in eye contact at appropriate times, and produce accurate visible speech, facial expressions and hand and body gestures. The character will mimic the actions of real persons and behave intelligently, gracefully and appropriately in the context of specific task domains.

Inventing systems that enable face-to-face communication with intelligent animated agents requires a deep understanding of the auditory and visual behaviors that individuals produce and respond to while communicating with each other. Face-to-face conversation is a virtual ballet of auditory and visual behaviors, with the speaker and behavior simultaneously producing and reacting to each other’s sounds and movements. While talking, the speaker produces speech annotated by smiles, head nods and other gestures, while the listener provides simultaneous auditory and visual feedback to the speaker (e.g., “I agree,” “I’m puzzled,” “I want to speak.”). The listener may signal the speaker that she desires to speak; the speaker continues to talk, but acknowledges the nonverbal communication by raising his hand and smiling in a “wait just a moment” gesture. Face-to-face conversation is often characterized by such simultaneous auditory and visual exchanges, in which the sounds of our voices, the visible movements of our articulators, direction of gaze, facial expressions and head and body movements present linguistic information, paralinguistic information, emotions and backchannel cues, all at the same time.

Building systems that engage users in natural face-to-face conversational interaction is a challenging task. The system must simultaneously interpret and produce auditory and visual signals. The system must simultaneously interpret the user’s auditory and visible speech, eye movements, facial expressions and gestures, since these cues combine to signal the speaker’s intent—e.g., a head nod can clarify reference, while a shift of gaze can indicate that a response is expected. In addition, auditory and visual cues provide paralinguistic information that annotates and enriches the linguistic message, indicating new and important information, imparting emotion to the message (e.g., anger, surprise) and indicating whether the speaker is being serious, sarcastic, etc.

In addition to interpreting the auditory and visual cues provided by the student, the animated agent must also produce accurate, natural, and expressive auditory and visible speech with facial expressions and gestures appropriate to the physical nature of language production, the context of the dialogue, and the goals of the task. Most important, the animated interface must combine perception and production to interact conversationally in real time – while the animated agent is speaking, the system must interpret the user’s auditory and visual behaviors to detect agreement, confusion, desire to interrupt, etc., and while the user is speaking, the system must both interpret the user’s speech and simultaneously provide auditory and/or visual feedback via the animated character.

Developing such systems requires advances in speech recognition, natural language generation and synthesis, facial animation, recognition of facial expressions and gestures, dialogue interaction and imparting personalities to computer agents. As well, realizing these scenarios requires a deeper understanding of the nature of human communication and human computer interaction. To address these issues, we must record, analyze and study the behaviors of master teachers and their students during learning sessions, and then interpret and model these same perceptual and production behaviors in interactive books, and evaluate their effectiveness.

Architecture of Interactive Books

Interactive books run on client machines connected to servers. Since all client software is being developed in Java, interactive books will run on PCs or Macintoshes. Except for real time animation rendering engine, which runs on the client, all of the technology modules that enable natural conversational interaction with conversational agents run on the server.

Interactive books are actually advanced dialogue systems—computer systems that enable natural, unconstrained, mixed-initiative dialogues between people and machines in specific task domains. (In interactive books, the task domains are learning to reading and learning to comprehend what is read.) The key feature of advanced dialogue systems is that users can say anything they want to say, any way they want to say it, as long as what they say is relevant to the task at hand. Advanced dialogue systems are called “mixed-initiative” because either the user or system can seize the initiative and attempt to take control of the dialogue at any time.

Advanced dialogue systems combine a number of different technology modules, also known as technology servers shown in the Figure below. These servers communicate with each other using the Galaxy architecture, developed and placed in the public domain by the MIT Spoken Language Systems lab (Seneff et al., 1998). Galaxy has proven to be an excellent platform for researching, developing and evaluating advanced dialogue systems. It is also an excellent architecture for building advanced dialogue systems that incorporate intelligent animated agents.

The Galaxy architecture and the technology modules that underlie interactive books are shown in the figure below. It can be seem that technology modules communicate with each other through the hub, which passes messages between the servers using a simple protocol. All of the technology servers were developed at CSLR, except for the face tracking system developed by Javier Movellan and his colleagues at UCSD. Detailed descriptions of the modules and related publications are available on the CSLR Web site at http://cslr.colorado.edu and the UCSD web site.

Galaxy Architecture, with message-passing Hub and Technology Servers

Technology servers used in interactive books include:

        Audio Server – Receives signals from microphone or telephone and sends them to the recognizer. Also sends synthesized speech to PC speakers or telephone.

        Sonic Speech Recognizer – Takes signals from an audio server and produces a word lattice. Developed by Bryan Pellom at CSLR, it will be extended as part of the proposed work.

        Phoenix Natural Language Parser – Takes word lattice from recognizer and produces the “best” interpretation of the recognized utterance.

        Confidence Server – Takes hypothesis and semantic parse from the speech recognizer and parser as input and annotates the words and concepts with levels of confidence.

        Dialogue Manager – Resolves ambiguities; estimates confidence in the extracted information; clarifies with user if required; integrates with current dialogue context; builds database queries; sends data to NL generation for presentation to user, prompts user for information;

        Database / Backend Server – Receives SQL queries from Dialogue Manager; interfaces to SQL database; retrieves data from the web to enable learning tools to access online information;

        Natural Language (NL) Generator – Constructs strings of words to speak back to the user based on the current dialog action;

        Text-to-Speech (TTS) Synthesizer – Receives word strings from NL generation; synthesizes them to be sent to the audio server;

        CU Animate Character Animation Server – Receives a string of symbols (phonemes, animation control commands) with start and end times from the TTS server, and produces visible speech, facial expressions and other gestures in synchrony with the speech waveform. A complete description of CU Animate is available here.

Tutors

Children need foundational speech, language, and reading skills, to help them achieve academic and social success and to gain self-esteem. Our programs help children accomplish this with programs that integrate powerful, individualized, tutorial activities (Tutors) with engaging Interactive Books. While all children can benefit from our programs, the programs are critically useful for children with special needs, who represent between twenty to twenty five percent of all K-12 students, in the following four populations: 1) students with reading disabilities, 2) Spanish-speaking students with limited English language proficiency, 3) students with autism spectrum disorder, and 4) students with hearing impairments.

Students in these populations must overcome significant barriers to learning, and are at great risk of ever realizing their potential within the public school system. We can help these students overcome their barriers to learning and achievement by providing them and their teachers with computer-based learning tools that enable significant learning gains while reducing teacher load. At all phases of development, teachers and students help design, evaluate, and modify activities. This process is described in the Participatory Design document. Teachers appreciate greatly that the programs can provide powerful individualized instruction for some of their students, freeing the teacher's time to help other children one on one or in small groups.

Integration of Tutors and Books

Our Intelligent Tutors integrate fully with our Interactive Books. Children learn and practice foundational skills in completely individualized ways in Tutors. Book programs evaluate application of these skills, and assign review activities. Both Tutors and Books work with the help of an intelligent, supportive animated coach, who gives focused hints adapted to the child's rate of progress. Both Books and Tutors teach and support vocabulary learning.

This document first describes the scientifically-based reading research which grounds and guides our programs, and then for all Tutors describes a) their foundational skill domains, b) their purposes within domains, and c) how they are individualized, both for students and for teachers.

The Reading Research that Motivates the Tutors

The Impact of Reading Problems

Reading is essential to academic achievement and to producing literate citizens. Poor academic achievement negatively impacts our society in many devastating ways, including the reduced well-being of school-aged children, reduced numbers of qualified personnel for jobs, increased risks to the national economy, and increased burdens to taxpayers for subsidized health care. Great financial costs are associated with this problem; the U.S. spends about $38 billion per year on special education (12% of total education spending), up to 2.28 times the financial resources spent on regular education (Parrish, 2000; Chambers, Parrish, Lieberman, & Wolman, 1998). Even this is insufficient. Currently far too teachers are qualified to cover these needs (Brady & Moats, 1997; Lyon, 1999; National Reading Panel Report, 2000). Even with Reading First Initiatives designed to prepare more qualified teachers, the small-group and individualized instruction favored in reading research will be hard to achieve without strong and flexible learning tools to expand teachers' resources. While the Bush administration is demanding improved performance from schools-even threatening sanctions for schools that do not show improvement-few new tools are made available for helping schools to achieve these goals.

Causes and Remedies for Reading Disabilities

A broad consensus has emerged about causes of reading disabilities and how to help them (see Rayner, Foorman, Perfetti, Pesetsky, & Seidenberg, 2001 for a recent review). Weak phonological processes underlie the problems of most children with reading disabilities (Lyon, 1995). Most seem to have "imprecise" or poorly differentiated phonological representations for words, resulting in subtle difficulties in spoken and heard language, such as in repeating nonsense words (Snowling, 2000) and in judging correct from incorrect pronunciations (Elbro, Borstrum, & Petersen, 1998). Most children with reading disabilities have weaknesses in short-term memory, articulatory awareness (Montgomery, 1984), and in phoneme awareness (analyzing sounds within spoken words) and phonological decoding (sounding out print to speech) (Lyon, 1995; Olson, Wise, Conner, Rack & Fulker, 1989). These last two deficits lead directly to problems in word reading and spelling, and lead to secondary difficulties in reading comprehension (Olson, et al., 1989; Perfetti, 1985). While inherited, brain-based factors relate to these difficulties (Frith, 1997; Olson et al., 1989; Shaywitz, 1996), by far the most encouraging research finding is that these weaknesses can indeed be remedied. Intensive, structured, and sustained instruction in phoneme awareness and phonics, carried into extensive, accurate practice in engaged reading for meaning helps these children immensely (Hatcher, Hulme & Ellis, 1994), and helps them more the younger they are identified (Torgesen, Wagner, & Rashotte, 1997; Wise, Ring, & Olson, 2000).

The above research shows the importance of early intensive phonological work for improvement in reading for children with reading disabilities. The necessary components of good instruction are becoming clear (Phonological Awareness, Phonics, Fluency, Vocabulary, and Comprehension, report of the National Reading Panel, 2000), but still unclear is exactly which methods are best for improving phonological processes. Speech-motor based phonological awareness has proven to be one highly effective method for helping children with reading disabilities both with (Torgesen, 1997; Wise & Olson, 1992; 1995; Wise, Ring, & Olson, 2000) and without computer support (Torgesen, Wagner, & Rashotte, 1997).

Wise and Olson's (Wise & Olson, 1992; 1995; Wise et al, 2000) computer-supported programs linked the best synthetic speech available at the time to book-reading programs and phonological exercises. Children made significant gains in reading-related skills, relative to both trained and untrained controls. Most of the computer programs in these studies merely supplied correct answers when asked; they were not able to encourage active problem-solving nor self-checking by the students. The speech-motor based phonological programs also used an elaborate naming system for sounds, robbing more time than necessary from reading practice. Our programs have far greater capabilities, ensuring that they can only surpass these gains. We incorporate the best ideas from these studies, and we improve on them greatly in terms of higher-quality speech production, new technology, and programs not available in that earlier research. Our capacity for speech recognition, animation, and advanced programs with much built-in linguistic knowledge allow our animated "coaches" to give focused, supportive hints and adaptive feedback, based on the programs' recognition of students' particular responses. This encourages our students to be engaged and active problem-solvers while they learn skills.

The Development of the Tutors

Design of Tutors

Our tutors have been developed as part of a comprehensive computer-based remedial reading program, incorporating the strengths of the computer programs above, and the principles and many of the methods of Linguistic Remedies (LR; Wise, 2002), created by co-PI Barbara Wise (see figure 1). LR is a powerful program for reading remediation that is continually refined to incorporate research findings. It teaches a speech-motor, guided discovery base for phonological awareness and word study. Children learn simple, consistent linguistic terms for a quick but deep understanding of the language base (e.g., "Lip, Tip, and Back Stops" for "twin" stop sounds of /p/-/b/, /t/-/d/ and /k/-/g/; "Lip, Tip, and Back Noses" for nasals /m//n//ng/). All concepts are grounded in the child's imagination, body, speech, and in print, and are applied extensively in reading and writing in meaningful and engaging contexts. Guided by the above research, children speed every skill to automaticity after they attain competence. Fluency and comprehension are taught and practiced within engaging reading for interest and enjoyment. The program encourages transfer of all concepts to new material away from the teaching situation, for long-lasting improvement in independent, eager reading after training is complete. Anecdotal evidence from more than 50 therapists in Colorado and Washington support the success of the methods, and a recent case study reports the exact methods, progress, and maintenance of gains by a typical child in the program (Wise, 2001a, 2001b).

Our computer-based program incorporates the key principles of scientifically based reading research implemented in the LR program in Interactive Books and Tutors (tutorial activities). Perceptive, intelligent animated agents coach the students through reading and comprehension of stories in Interactive Books, and through focused applications that teach and exercise foundational reading skills in Tutors. Tutors and Books are fully integrated, with skills learned in Tutors being applied in Interactive Books, and error patterns in Interactive Books being assigned for teaching or review in Tutors. The Vocabulary Tutor instructs vocabulary for Books and Tutors, and it has also been enhanced to provide more tools for reading and spelling instruction.

Assessment

Computerized assessment and delivery of instruction ensure success with extensive practice at individualized levels. Students are initially assessed using animated instruments we have developed in phoneme awareness, letter-sounds, and word reading based on verified instruments (Olson et al., 1994). Mid- and end-of-year tests allow for growth modeling analyses of progress. All other assessment is ongoing and formative, within the instructional programs, and is described more fully in the Assessments section of this website.

Moving through the Programs

Our Tutors follow a default sequence from phonological awareness and decoding and encoding of simple consonant-vowel-consonant (CVC) words to more complex orthographic patterns into multisyllable words (Wise, 2001c). Tutors choose input words for lessons from this sequence to match students' needs, and authoring tools allow teachers to insert particular words of their own into any Tutor. The programs have default settings of time per domain, balanced between skills work in Tutors and application, fluency, and comprehension in Books. The balance of instruction leans more towards foundational skills work at kindergarten and first grade levels and moves to more and more reading in context as students progress.

A teacher menu allows teachers to accept default settings, to modify those settings to match their instruction that day, or to select different selections of pre-programmed sequences or times of instruction per domain. Teachers and students help in the design and modification of activities. To date, over 60 Interactive Books and 10 Tutors have been developed in English and Spanish. We are scaling up the new foundational skills Tutors through participatory design activities with teachers and reading specialists, as part of the Coleman grant. The 15 educators in our participatory design workshops are looking forward to how the programs will extend their teaching resources, so they can work with more students one-on-one, knowing that others are receiving effective, individualized instruction from the computer programs.

The Balance of Instruction

Our programs balance reading for meaning with instruction of foundation-level skills. The balance of instruction shifts as students progress, from more skills work for early readers to more book-reading as children improve. Beginning readers read Books with decodable and with predictable language, and practice comprehending stories that are read to them by coaches. Based on earlier work by Wise, Ring, & Olson (2000), our working hypothesis is that by a third grade reading level, children spend more time in reading Books than in skills work in Tutors.

Knowledge Domains covered by the Tutors

Tutors cover the following essential knowledge domains which are taught if assessed as needed: The default sequence for Tutors includes the first 8 basic Skill Domains underlying good word reading listed below. Domains 9 through 14 are available as teacher choices. A default set of words...

Basic Domains

1. Phonological Awareness (word, syllable, rhyme, phonemes). With all, practice identifying, matching, blending, segmenting, and manipulating these units of spoken language).
2. Alphabet and Letter-Sound Knowledge
3. Reading of Regular Words, from CVC to complex words.
4. Spelling of Regular Words
5. Reading Sight Words
6. Spelling Sight Words
7. Vocabulary
8. Comprehension strategies, if the children are not successful with the comprehension support and practice within the Books. Word reading, vocabulary, fluency, and comprehension are taught and practiced in Books, which also assess needs and assign Tutors based on those needs.

Optional Expanded Domains

9. Articulatory Awareness (of how speech sounds are produced in the body)
10. Morphology
11. Syntax
12. Fluency, or prosodic expression
13. Visual Persistence
14. Composition

Figure 1: Some Working Tutors

Purposes of Tutors, from Teaching to Practice to Transfer

Within each of the above eight domains, different Tutors accomplish different levels of instruction, so that all concepts are taken from instruction to automaticity and transfer. Following assessment of need, some Tutors motivate and teach or guide discovery of new concepts. Others then engage students in practice of new skills and concepts, until students become competent (a default 80% or set by teacher). All practice tutors begin with one or two previously successful "comfort zone" items, move to seven to ten instructional items, and finish with a comfort zone item, following Vygotsky (1978). After students attain competence with a skill or pattern, other Tutors speed these automaticity, apply them in context of sentences prior to assignment to books, or set "send-home" activities to encourage transfer away from the computer. Here follows a list of the types of purposes of different Tutors.

0. Get a buy-in: what the Tutors expect to accomplish, and why
1. Teach and
2. Practice needed skills to "competence" (default = 80% correct in two consecutive item sets), beginning and ending in a "comfort zone," a level below current knowledge, to ensure success.

After competence, students move to more complex items with the practice Tutors. At the same time, other tutorial activities exercise the competent skills to

3. Speed them until they become effortless (automatic)
4. Apply then in reading or writing in context
5. Encourage their transfer to new words, to a related skill, and away from the computer.

Students advance by performance. Students progress through sequences of words with these levels of activities until they reach a preset goal (e.g., 6 months beyond grade level), or until they or their teacher decide to have them stop. Automaticity programs time children on items they have read correctly in practice programs. Items exit automaticity practice when times level off, within a range of 2 seconds.

Strengths from Cutting Edge Speech Technology

A great strength of our learning tools is the integration of leading-edge language technologies developed at CSLR, including high quality speech production and speech recognition. Speech recognition for children allows our programs to track children's speech and all their responses to items, and to react to these responses. The animated coaches give focused guided suggestions to help children find and correct their errors themselves. Coach's hints adapt to children's progress, with verbose, and supportive help for children making slow progress and less and less scaffolding with steeper progress gradients for students answering quickly and accurately (after Bouwhuis, 2001). Thus our programs provide more finely individualized practice and progress, and require and support more active problem solving, than any other programs currently available that teach and practice foundational skills.

To overcome limitations of text-to-speech synthesis, applications use naturally recorded speech, aligned and synchronized automatically with the visibly accurate production of English or Spanish speech of the animated characters. The characters' heads can be rotated and made semi or fully transparent, so children can watch how sounds are made to improve their own speech clarity and to detect errors. If a child has, for instance, left out the "l" in spelling "sled," the coach can direct him to watch the tongue movement right after the /s/ in "sled" to discover the missing sound. Children can also compare video capture of their own mouths, in speaking a sound or a word, to the articulation of the coach's mouth. This encourages active and clear speech in the exercises, to improve both the clarity of the child's speech and the underlying precision of his phonological representations for words.

Sequences of Instruction

Programs follow a default sequence of spelling-sound patterns (orthography), based on simplicity and consistency of regular words and on frequency of sight words. Programs move children up and back along these sequences, depending on performance. For instance, children practice words with consonant blends (e.g. "spot") only after demonstrating success with simpler words without blends (e.g., "pot"). Item sets in Tutors begin with an "easy" item, proceed to instructional items, and finish with an easier item.

The program also has "default" percentages of time to be spent in skill domains, depending on the student's needs. The program has a default time allotment of 30 minutes, or the teacher or student can select the time they want to spend with the programs, regularly or for a particular day. According to the default settings, students complete a domain's activity, when they have finished 2 item sets or finished the time limit, whichever comes first. Students may choose among activities in a domain, and may choose reward activities, including Books. In general, the time in Interactive Books is the most flexible, taking up any time remaining after Tutors.

Teacher Menu and Choices

The teacher menu allows teachers to select the default sequence, choose among different pre-programmed sequences, or to override a sequence for the teacher's own set of words or choice of orthographic structure to work on that day. Teachers can choose more or less strict per cent correct criteria, and they can vary the number of files at criteria needed to advance. Teachers can also modify percentages or amounts of time for students to spend in skill domains and in Books.

Examples of Tutors, with their Skills Domain and Level and Purpose

4-Square. Domain: Reading of Regular Words

Level and Purpose: Teaching and practice to competence. Teaching occurs with Help?

Speedy Reading. Domain: Reading of Sight Words

Level and Purpose: Speeding to Automaticity

** 4 word-buttons, and Racing graphic. Dark runner is last round's speed; Light is current.

Sound-Ball. Skill Domain: Alphabet and Letter-Sound Knowledge

Level and Purpose: Teaching and Practice to Competence. Teaching occurs with Help?

Assessment

Objectives

The objectives of the Assessment Group are to provide formative and summative evaluation feedback about the interactive books and reading tutors and to develop on-line assessments that can place students into the appropriate instructional levels of tutors and books as well as assess their progress. Formative evaluation includes feedback that will promote, improve and enhance the development and dissemination of the books and tutors. Summative evaluation involves assessment of the educational efficacy of the books and tutors, i.e. whether or not they improve students' reading more than other types of intervention.

Activities

Formative Evaluation

Formative evaluation involves a broad range of activities that provide the project staff with feedback that they can use to enhance and improve the interactive books and reading tutors. Parents and teachers will be surveyed about students' reading and computer habits. This information should help us determine the themes, characters and activities that students enjoy at different age levels. It should also give us a sense of students' access to computers and their computer literacy at different grade levels. This information will help us determine the type and range of software instruction that should preface and accompany students' use of tutors and books.

Parent, teacher and student feedback obtained during the participatory design phase of the project will be analyzed for trends. Feedback will be obtained using an interview rubric or protocol specifically designed for educational technology in order to insure that all participating responders address key aspects of the books and tutors. The trends identified will help us prioritize the design modifications to be made; the most frequently occurring feedback will take the highest priority.

Quantitative and qualitative analyses of actual student use of books and tutors will be conducted. Observations will be guided by an educational technology rubric enhanced by the addition of variables especially relevant to students at risk for failure in reading. Particular attention will be paid to the level student engagement when specific features are present and absent. Those features that increase and sustain student engagement will be retained, while those that have a negligible, minor or even poor effect on student engagement can be discarded.

Once the interactive books and tutors have been fully integrated into targeted schools, feedback from parents, students and teachers will be obtained through surveys and interviews. Their perceptions will be analyzed for trends that indicate the usability and sustainability of the books and tutors.

On-line Assessment Instruments

We have developed computerized assessments, both to place students into the appropriate levels of interactive books and tutors, and later to assess their progress in foundational reading skills. Three instruments using animated agents have been developed for this year: phoneme awareness, letter-sounds, and word reading based on verified instruments (Olson, Forsberg, Wise, & Rack, 1994). These three measures take only five to eight minutes each per student. Items are sequenced by difficulty or in the order that they are commonly taught and learned. Mid- and end-of-year tests allow growth-modeling analyses of progress. All other assessment is ongoing and dynamic, within the instructional programs. In pre-, mid- and post-tests, as well as in dynamic assessment within the books and tutors, items are recorded and analyzed for sound-based or spelling-based confusions, and for particular sound-and-letter patterns that are causing difficulties.

The word recognition measure is a single word reading test. It contains the Olson and Wise (Olson et al., 1994; Wise et al. 2000) measure which contains words ranging from the late second through twelfth grades. It has been extended downward to the kindergarten level, beginning with letter identification for lower and upper case letters to words at the pre-primer, primer, first through second grade levels. This measure will be administered initially to each student to determine the level at which s/he is decoding single words. These data will then help us place students into the appropriate level of books and tutors as well as provide baseline information on decoding skills upon entry into the project. The concurrent and predictive validity of the Olson and Wise measure are well established (Olson et al.; 1994, Wise et al., 2000). In addition, we analyzed its internal consistency and found it to be quite high. The expanded measure will be studied for validity, reliability and scaling. In the on-line version of this test, the student is asked by Ms. Gurney to read each letter or word that appears in the box. After the student has read an item, a star will appear on a chart and Ms. Gurney gives verbal encouragement that does not indicate the correctness of the response, for example "Wow! Here's another one!" When the student reaches a ceiling level on the measure, five consecutive incorrect responses, a large star will fill the chart, masking the earned stars and Ms. Gurney says "You've answered the lucky question, you've finished the test!" This progress rubric is used for all three of the on-line assessments.

The on-line sound-letter correspondence test assesses students' ability to identify the correct letter or letters that represent a sound. In this measure, Ms. Gurney identifies that a key word and the target sound found at its beginning. Then, she asks students to choose the letter(s) that go with the target sound. For example, she might say "The word "baby" begins with /b/. Choose the letter that goes with /b/." The target letter and three foils appear on the screen. The foils chosen represent different types of phonetic or visual feature confusions. All of the test items are ordered developmentally, in the sequence learned by children. This measure will be pilot tested with teachers and students. Changes will be made based on the results of this pilot work. Validity, reliability, scaling and internal consistency studies will be conducted and changes will be initiated on the basis of information provided by item analysis.

Lastly, the phonological awareness measure examines students' ability to segment and delete parts of compound words and phonemes from single words. Its tasks are based on Torgesen and Bryant's (2000) elision tasks, but do not require students to actually produce the new word. Rather, Ms. Gurney asks students to say the word that will undergo a deletion and then choose a photo that represents the word without a particular sound or syllable. For example, she might say "The word 'cart' ends with /t/. Choose the one that goes with 'cart' without the /t/." In this example, photos representing a car, art, a cat and a wheelbarrow appear as choices for the answer. The target answer is the car, and the foils represent phonetic, instructional and semantic confusions. All of the items are ordered developmentally, in the sequence learned by children. This measure will be pilot tested with teachers and students. This measure will be studied for its validity, reliability, scaling and internal consistency and improved based on information provided by item analysis.

Summative evaluation

The summative evaluation of the books and tutors asks the question: Is the use of interactive books and reading tutors more efficacious than the use of an alternative computer-based reading intervention or the traditional intervention?

The question will be answered by comparing the reading performance of students in the ITR Book and Tutor group with students in two matched groups, one receiving the traditional intervention and another receiving traditional instruction. The reading performance of the two groups will be assessed by the Qualitative Reading Inventory mastery reading levels and accuracy and response times from the three online measures pre-, mid-, and post-program testing and one year after completion of the program. In addition, growth curve analyses of student progress in reading and the foundational reading skills will be conducted for the two groups. In addition, the Colorado Student Assessment Program (CSAP) reading proficiency scores for the three groups will be compared at the end of the program.

Lastly, the number of hours of special education services received by students in the ITR Books and Tutors group will be compared with that of students in the other groups. This final analysis will help us address the more practical aspect of the cost-effectiveness of the interactive books and tutors, a perceived value-added aspect of the ITR Books and Tutors program.

References

Bouwhuis, D. (August, 2001). Multi-agent Environments for Literacy Learning. Paper presented at International Conference on Interactive Literacy Education, Nijmegen, Holland.

Brady, S., & Moats, L. (1997). Informed instruction for reading success: Foundations for teacher preparation. Position paper. Baltimore, MD: International Dyslexia Association.

Catts, H. & Fey, M. (1997). The language basis of reading disabilities. Paper presented at the Society for the Scientific Study of Reading, Chicago, IL.

Catts, H. & Kamhi, A. Eds. (1999). Language and Reading Disabilities. Boston, MA: Allyn & Bacon.

Chambers, J. G., Parrish, T. B., Lieberman, J. C., & Wolman, J. M. (1998). What Are We Spending on Special Education in the U.S.? Center for Special Education Finance, Brief No. 8.

Cole, R. (1999). Tools for research and education in speech science. In Proceedings of the International Conference of Phonetic Sciences, San Francisco, CA,.

Cole R., Massaro, D. W., de Villiers, J., Rundle, B., Shobaki, K., Wouters, J., Cohen, M., Beskow, J., Stone, P., Connors, P. Tarachow, A., & Solcher. D. (1999). New tools for interactive speech and language training: Using animated conversational agents in the classrooms of profoundly deaf children. In Proceedings of ESCA/SOCRATES Workshop on Method and Tool Innovations for Speech Science Education, London, UK.

Cole, R., T. Carmell, P. Connors, M. Macon, J. Wouters, J. de Villiers, A. Tarachow, D. Massaro, M. Cohen, J. Beskow, J. Yang, U. Meier, A. Waibel, P. Stone, G. Fortier, A. Davis, and C. Soland. (1998). Intelligent animated agents for interactive language training". In STiLL: ESCA Workshop on Speech Technology in Language Learning, Stockholm, Sweden.

Elbro, C., Borstrom, I., & Petersen, D.K. (1998). Predicting dyslexia from kindergarten: The importance of distinctness of phonological representations of lexical items. Reading Research Quarterly, 33, 36-60.

Frith, U. (1997). Brain, mind, and behaviour in dyslexia. In C. Hulme & M. Snowling (Eds.), Dyslexia: Biology, cognition, & intervention. London: Whurr Publishing.

Hatcher, P., Hulme, C., & Ellis, A. (1994). Ameliorating early reading failure by integrating reading and phonological teaching: The phonological linkage hypothesis. Child Development, 65, 41-57.

Kintsch, W. (1998). Comprehension: A paradigm for cognition. NY: Cambridge University Press.

Lyon, G. R. (1995). Toward a definition of dyslexia. Annals of Dyslexia, 45, 3-30.

Lyon, G. R. (1999). Reading development, reading disorders, and reading instruction: Research-based findings. Language, Learning, and Education, 6, 8-16.

Montgomery, D. (1981). Do dyslexics have difficulty accessing articulatory information? Psychological Research, 43, 235-243. National Center for Health Statistics (1994).

National Health Interview Survey. Series 10, No. 188, Table 1.

National Reading Panel Report (2000) Teaching Children to Read. NIH Publication # 00-4754.

No Child Left Behind. Web site: http://www.ed.gov/offices/OESE/esea/

Olson, R.K, Forsberg, H., Wise, B., & Rack, J. (1994). Measurement of word recognition, orthographic, and phonological skills. In G.R. Lyon (Ed.) Frames of reference for the assessment of learning disabilities: New views on measurement issues. Baltimore: Paul H. Brookes Publishing Co.

Olson, R.K., Wise, B., Conners, F., & Rack, J. (1989). Specific deficits in component reading and language skills: Genetic and environmental influences. Journal of Learning Disabilities, 22, 339-348.

Parrish, T. G. (2000). Disparities in the Identification, Funding, and Provision of Special Education. Paper presented at the Harvard Civil Rights Project for The Conference on Minority Issues in Special Education in Public Schools, Cambridge, MA, November 6, 2000.

Perfetti, C. (1985). Reading ability. Oxford, England: Oxford University Press.

Perfetti, C., Marron, M., & Foltz, P. (1996). Sources of comprehension failure: Theoretical perspectives and case studies. In C. Cornoldi & J. Oakhill (Eds.), Reading comprehension difficulties: Processes and intervention. Mahwah, NJ: Lawrence Erlbaum Associates.

Pellom, B., Ward, W., & Pradhan, S. (2000). The CU Communicator: An Architecture for Dialogue Systems, International Conference on Spoken Language Processing (ICSLP), Beijing China, November.

Pellom, B., Ward, W., Hansen, J., Hacioglu, K., Zhang, J., Yu, X., S. Pradhan. (2001). University of Colorado Dialog Systems for Travel and Navigation, Human Language Technology Conference (HLT-2001) San Diego, March.

Rayner, K., Foorman, B, Perfetti, C., Pesetsky, D., & Seidenberg, M. (2001). How psychological science informs the teaching of reading. Psychological Science in the Public Interest, 2, 31-74.

Reeves, B., & Nass, C. (1996). The Media Equation, Cambridge University Press, 1996.

Shaywitz, S. (1996, November). Dyslexia. Scientific American, 98-104.

Seneff, S., et. al., (1998). Galaxy-II: A Reference architecture for Conversational System Development, Proc. ICSLP-1998, Sydney Australia, December.

Snowling, M. (2000). Dyslexia (2nd ed.). Oxford, UK: Blackwell Publishers. Pps. 8, 100, 198, 215.

Stanovich, K. (1986). Matthew effects in reading: Some consequences of individual differences in acquisition of literacy. Remedial and Special Education, 5, 11-19.

Torgesen, J. (November, 1997). Issues and conclusions about intervention: A research-based perspective. Paper presented at the annual meeting of the Orton Dyslexia Society, Minneapolis.

Torgesen, J. & Bryant (2000). Comprehensive Test of Phonological Processes (CTOPP). Austin, TX: Pro-Ed.

Torgesen, J., Wagner, R., & Rashotte, C. (1997). Prevention and remediation of severe reading disabilities: Keeping the end in mind. Scientific Studies of Reading, 1, 217-234.

Van Dijk, T. & Kintsch, W. (1983). Strategies of Discourse Comprehension. San Diego, CA: Academic Press.

Vygotsky, L.S. (1978). Mind in Society. M. Cole, V. John-Steiner, S. Scribner, & E. Soberman (eds and translators). Cambridge, MA: Harvard University Press.

Wagner, R., Torgesen, J., & Rashotte, C. (1994). Development of reading-related phonological processing abilities: Evidence of bi-directional causality from a latent-variable longitudinal study. Developmental Psychology, 85, 1-20. Wise, B.W. & Olson, R.K. (1992). Spelling exploration with a talking computer improves phonological coding. Reading and Writing, 4, 145-156.

Ward, W., & Pellom, B. (1999a). The CU Communicator System, IEEE Workshop on Automatic Speech Recognition and Understanding, Keystone Colorado, December.

Ward, W., & Pellom, B. (1999b). The CU Communicator, 1999 IEEE Workshop on Automatic Speech Recognition and Understanding, Dec. 12-15, Keystone Colorado. Article available from CSLR Web site: http://communicator.colorado.edu/

Wise, B. (2001a). The indomitable dinosaur builder (and how she overcame her phonological deficit and learned to read). Journal of Special Education, 35, 134-144.

Wise, B. (October 2001b) Return of the Stegosaurus Builder: Lasting success for a child with a remedied deficit in phonological awareness. Paper presented at symposium at the Annual meeting of the International Dyslexia Society, Albuquerque, NM.

Wise, B. (2001c) Advanced Linguistic Remedies: Morphology and phonology in advanced reading, spelling, and writing. Boulder, CO: Remedies for Reading Disabilities, Inc.

Wise, B. (2002) Linguistic Remedies for Reading Disabilities. Boulder, CO: Remedies for Reading Disabilities, Inc.

Wise, B.W., & Olson, R.K. (1995). Computer-based phonological awareness and reading instruction. Annals of Dyslexia, 45, 99-122.

Wise, B., Olson, R., & Ring, J. (1999) Phonological awareness training with and without explicit attention to articulation. Journal of Experimental Child Psychology, 72, 271-304.

Wise, B., Ring, J., & Olson, R.K. (2000). Individual differences in benefits from computer-assisted remedial reading. Journal of Experimental Child Psychology, 77, 197-235.