Kids' Audio Speech Corpus
NSF/ITR Reading Project

Corpus Overview:

The CU Kids' Audio Speech Corpus has been developed to enable research in auditory recognition of children's speech. The goal of this two-year development effort was to collect sufficient audio and video data from preschool through middle school aged children to enable development of auditory and visual recognition systems enabling face-to-face conversational interaction with perceptive animated agents during learning tasks. The current corpus, which includes audio data from both prompted speech and spontaneous speech, was supported by the following grants:

• NSF/IERI: EIA-0121201 - Kintsch, W., Caccamise, D., Cole, R., Olson, R., Snyder, L., "IERI: Scalable and Sustainable Technologies for Reading Instruction and Assessment,"
• NSF/ITR: IIS-0086107 - Cole, R., van Santen, J., Movellan, J., "ITR: Creating the Next Generation of Intelligent Animated Conversational Agents,"

During the 2000-2001 school year, we developed and tested data collection protocols, in collaboration with Dr. John-Paul Hosom at the Center for Spoken Language Understanding at the Oregon Graduate Institute (OGI). This work resulted in the collection of audio and video data from approximately 200 children. During the 2001-2002 school year, we further refined the data collection protocol, and captured audio and video data from an additional 580 students in schools within the Boulder Valley School District in Colorado. The audio speech data were verified and transcribed at CSLR. Dr. Javier Movellan and his associates at University of California, San Diego are currently working on transcribing the video data, which will comprise the visual portion of this corpus. Written permission to distribute audio and visual data in this corpus was obtained from the parent or guardian of each student.

The CU Kids' audio and video speech corpora will continue to be developed over the next five years to incorporate audio and video data from schools throughout the state of Colorado. In addition to the data described here, which includes the initial release of audio data, we will be collecting speech of children reading out loud from books, and speech of children summarizing stories they have listened to or read. Our goal is to collect, analyze, understand and model the speech and visual behaviors of a diverse population of children, including children who speak Spanish and Native American languages at home. In addition, the data collection protocol will evolve over the course of the NSF ITR and IERI projects to enable a range of capabilities, including pronunciation training, recognition and transcription of natural continuous speech, and natural dialogue interaction with intelligent animated agents during reading and comprehension training.

Collection Logistics:

Location

Speech data were collected in 5 public schools within the Boulder Valley School District (BVSD), Boulder County, Colorado.

Speakers

Initial data collection efforts were focused on children between Kindergarten and fifth grade. Two of the schools included in the collection have a high percentage of non-native, predominately Hispanic English speakers.

Auxiliary information about each child is collected for the sole purpose of gathering statistics to be used in research. This information, collected mostly from school records, is summarized in the table below.

Table 1: Auxilary information collected for each child

Schools

The speech data were collected from students in five schools in the Boulder Colorado area (summarized in Table 2). During subsequent years, data collection will move to other cities and towns throughout Colorado. Recording is being done in different locations within schools (instead of a sound attenuated booth) in order to maximize, to the extent possible, the similarity between the recording conditions and "real-life" usage of speech technologies developed with such data. In total, 780 children were recorded. Due to several transcription issues, 660 speakers were retained in the final audio corpus.

Table 2: Number of children by grade level and Boulder Valley School collected

Recording and Collection Specifications

The audio portion of the corpus was recorded in relative quiet schoolroom environments (e.g., utility rooms, libraries) at 16-kHz sampling frequency with 16 bits per sample, using a Sound-Blaster audio card. Recording was done using one of three types of microphones: a commonly available head-mounted noise-canceling microphone LABtec LVA-8450, an array microphone CNnetcom-Voice Array Microphone VA_2000, and a commonly-available desktop farfield microphone. Recording was done in a room within the school instead of a soundproof booth in order to maximize, to the extent possible, the similarity between the corpus and "real-life" conditions envisioned for the reading tutors. Each waveform was automatically checked for clipping and appropriate energy levels after each utterance was produced during the recording session; in addition, each recorded waveform was displayed on the screen immediately after recording to allow manual checking for waveform cutoff. If the automatically determined recording quality was unacceptable, the child was asked to repeat the utterance; only the final repeated utterance was saved. If the recording was otherwise acceptable, the child and experimenter had the opportunity to re-record an utterance to correct pronunciation errors or waveform-cutoff errors. In this case, both the original and new versions of the waveform were saved.

The corpus consists of three types of speech: prompted, read, and spontaneous speech. An animated face with natural speech was used to elicit the prompted speech; for read speech, the text to be read was displayed on the screen in large letters. Prior to data collection of each task, the animated face gave instructions about what would happen and what was expected from the subject.

File Formats

Each utterance was written to an audio file in RIFF format. All filenames have a two-letter code to identify the corpus ("CC" for the comprehensive children's speech corpus), and codes that identify the grade, gender, sub-protocol, child, utterance, attempt number, and microphone type; each code field is separated by a dash. The format is CC-GG-PP-NNNNN-UUUUU-A-C.wav where "GG" indicates the grade, "PP" indicates the sub-protocol, "NNNNN" specifies the unique speaker number, "UUUUU" specifies the utterance number, "A" indicates which attempt has been made at this utterance, and "C" identifies the microphone channel. The ranges of each of these fields are given in Table 2. Each utterance is assigned a unique number; in the case of spontaneous speech, the number indicates the topic instead of the expected word or word sequence.

The auxiliary information was written to a text file in ASCII format. The filenames are of the format CC-GG-PP-NNNNN-info.txt, where "CC" indicates the comprehensive children's speech corpus, "GG" indicates the grade, "PP" indicates the sub-protocol, "NNNNN" provides the speaker identification number, and the keyword "info" indicates that the file contains auxiliary information about the speaker.

Table 3: Explanation of file naming conventions in the CU Kids' corpus

Video Collection

Simultaneously with the audio recording, a video recording was made of the child's face. The video recordings are not included as part of the CU Kids' audio corpus. For collection, Canon digital Elura 2mc video recorder with minimal machinery noise was used. (The noise was, in informal tests, inaudible). The video recorder was placed next to the computer monitor on the top left side of the screen (from the user's perspective) so that the child's mouth was visible at all times during speech. In preliminary recordings, the video recorder's internal microphone was used to record audio data with the video data; in later recordings, the array microphone input to the computer sound card will be split and sent to the video camera as well, to ensure that the video and audio data contain the same signal.

Transcription Guidelines

Transcriptions of the data include word-level text The word-level text transcriptions include mispronunciations, false starts, wrong words, and filled pauses. A summary of transcription markups are provided in Table 7.

Inter-Labeler Reliability

To assess reliability among transcribers, we compared transcriptions produced independently by transcribers for 40 speakers. This amounted to 2744 utterances. The results of the comparisons are displayed below. Labeler disagreement rate is measured using one transcription as a reference standard and the other as a hypothesis (swapping reference and hypothesis does not change the results). The NIST Sctk-1.2 scoring package was used to measure the disagreements (a summazation of the substitution, deletion, and insertion insertion errors) between the two transcriptions.

Table 4: Interlabler reliability for 40 speakers in the CU Kids' Corpus

In general, we see that there is roughly a 20% disagreement between the raw text between two transcribers. However, 2.2% of this disagreement is simply differences in punctuation and an addition 6.7% of the differences are due to disagreements in markups such as background noise () or non-primary speaker (). Disagreements of where to mark a word cutoff account for an additional 1.6% of the error (e.g., "part-" vs. "partia-" for the word "partial").

Data Collection Protocol Overview:

The protocol was recorded in the order of prompted speech, read speech, and spontaneous speech. A summary of the data collected by data type, grade level, and school is shown in Table 5. A summary of the protocol used is provided in Table 6. The protocol sentences and words are provided as an external file (protocol-spec.txt) included on this CDROM.