SPRING/SUMMER 2007
Kids and adults with certain speech-disrupting special needs (e.g., austism) use communication assists to talk. Devices take many forms, from simple picture boards to speaking word processors to complicated PDAs. Users' needs vary, target markets vary, but the devices are prohibitively expensive, limiting the development of and quality of life for many.
Inspired by my involvement in
e-soccer, the
BACC, and the
HOPE Technology School, I wanted to develop an affordable alternative--these devices are quite simple and can be replaced by commodity components and perhaps even surpassed with custom development.
The plan is to produce instructions for building your own communication assist (in a sort of open source way), or we may create a commercial product.
This page chronicles the project and provides a summary of my research and ideas, not always in a user-friendly form.
The Team
Austen McDonald
Diana Matsuoka
Sean Kiluk
The Status: AAC Device
Literature Review
- Existing Commercial Products
- Facilitated Communication
- Previous HCI research into AAC
- Cutting Edge TTS
FundingDesign
- Device Evaluation
- TTS Evaluation
- Keyboard Evaluation
Implementation
- Language and Toolkit Evaluation
Testing
IDEAS & POSSIBLE FEATURES
JUNE 13, 2007
It's important to identify what we want to build, namely what it's capabilities will be. This is an exhaustive list of all our ideas, not a feature list for any existing or to-exist product, although some value judgements are made as to how important certain features are. Pure speculation on my part is marked.
Speech
As per the interview below, the ability to produce quality speech from text (text-to-speech or TTS) is important:
- Users make phone calls.
- Speculation: when kids become teens and adults, their interactions with the outside world will be easier given more life-like speech.
Input
- Keyguards
- specifically mentioned in the interview but that same child can use a laptop unassisted, although teachers report his inability to produce converstaion without assistance.
- shouldn't be that hard to find someone to custom build keypads with keyguards.
- Simplified input schemes like smaller, multi-letter keyboards such as FrogPad or rollable keyboards designed for easy storage.
- advantage is portability, even wearability, and durability for rollable (usually they're spill-proof, drop-proof, etc.)
- FrogPad requires two fingers at times; some users will find this doable, others won't.
- Clearly we want options.
DEVICES
JULY 20, 2007+SEPTEMBER 28, 2007
Just got a
Nokia N800 from Mendel Rosenblum at Stanford, through the Nokia Developers Forum. It's a nice internet tablet and I think it shows promise. It has a hacked version of Debian Linux and an ARM processor, so it will require some cross compiling.
Watched Janice Light's
presentation on her AAC research project at Penn State, et al. Notes:
- Diverse Environments: bringing the device around water, food, etc., especially for young children.
- Attractiveness: bright colors, music, recognizable characters, decorations, allow the chlid to choose.
- Playing Games: like phone games.
- Constant Availability: battery power :(
- She talks about symbols and the choices.
- Visual Scene Layouts: (vis a vis blocks of pictures) hotspot-interactive scenes to describe objects/actions.
- Kids can locate pages better w/ buttons that represent entire menu pages (memory).
IMPLEMENTATION: LANGUAGES AND TOOLKITS
JULY 20, 2007
Nokia n800
- maemo.org is a development environment, integrating GTK+ and cross-compiling.
- Officially supports only C, but unofficially C++ and Python
FUNDING
JULY 10, 2007
We need some funding to purchase equipment and software, mainly to try out different ideas. Possible sources include:
- (see the literature review for autism-specific foundations)
- Stanford CS dept. has a number of Nokia N800s under the direction of Mendel Rosenblum
- Potential funders, gleaned from Foundation Center's Grants for People with Disabilities [giver (recipient)]
- Autism:
- Thomas J. Long Foundation (California Autism Foundation)
- David and Lucile Packard Foundation (Soaring Eagles Center for Autism)
- Chartwell Charitable Foundation (Autism Speaks)
- Lincy Foundation (Autism Research Institute, San Diego)
- AT&T Foundation (KESHET, Northbrook, IL)
- Other, smaller institutions where we have connections...
THE AUTISM COMMUNITY
JULY 10, 2007
Found a number of parents and educators who keep blogs or summaries of autism- and AAC-related material. Overwhelmed by the number of personal stories and chronologies.
- Utter Autism
- Mother of an autistic son
- Experience w/ something from Gus Communications Dynavox V
- Made attempts to improve communication devices, I think by programming her own dynamic displays
- Some comments on evaluation techniques for trying out AAC
- suggest doing it at home instead of at school
- http://autismahead.org/
- Summary of news related to autism, looks like a lot of device news
USER INTERVIEW
JUNE 11, 2007
Brief, facilitated interview with a 14-year old, Down syndromatic, user of assistive technology, namely the lightwriter. The goal was to assess the strengths and weaknesses of current technology as well as to float the idea of using laptops instead of custom-built keyboards.
What do you like about the lightwriter?
I like the ease when you type. It is easy to type with because the buttons are nice. The keyguard makes it easier.
What do you dislike about it?
It's too expensive. I want a printer that is adaptable. I want to be able to send email from it. That's it.
How do you like typing on a computer? Is it better or worse than the lightwriter and why?
It is better than typing on a computer because I can do anything I want because the keyboard is broken. It is broken because it is hard to type with someone else because I can't on my own or with someone else.
Is the speech function important to you? How important is voice quality?
Yes it is important because it is my voice. Voice quality is very important because I want to sound good.
Would you be willing to type extra characters to help the speech be more lifelike?
No.
HCI RESEARCH DISCUSSION
MAY 29, 2007
Talked w/ Scott Klemer on interfaces for autistic kids. Great conversation:
- Dan Gillette (at the Greenleaf Institute) did a CS 377 on interfaces for special needs kids (of some kind) recently
- Gregory Abowd at GT and his student Gillian Hayes have done a bunch of video stuff for kids w/ autism
- HCIC is an elite HCI conference run by Beth Mynatt (at GTs GVU) and there was a summary of Gregory's work on autism in 2006
At
CHI 2007, there was a SIG (Special Interest Group) entitled "
Interactive Technologies for Autism," sponsored by CAN.
Participants: Daniel Gillette of Greenleaf Institute; Gillian R. Hayes of Georgia Institute of Technology's College of Computing; Gregory D. Abowd of Georgia Institute of Technology's College of Computing; Justine Cassell of Northwestern University's Center for Technology and Social Behavior; Rana el Kaliouby of MIT's Media Laboratory; Dorothy Strickland of Do2Learn; and Patrice (Tamar) Weiss of University of Haifa in Israel.
LITERATURE REVIEW
JULY 4, 2007
Foundations that support autism research
Facilitated Communication
- From Wikipedia: "Facilitated communication is most often used with persons with developmental disabilities, most commonly autism and Down syndrome, populations in which some neurologists and psychologists believe there is a high incidence of dyspraxia, or difficulty with planning and/or executing voluntary movement."
- According to W., most studies show that it is not the impaired person producing the communication.
- Used and popularized in the US by Stanford Nobel laureate in the 80's and described here.
- W: not having the person look at the keyboard is contrary to FC training standards.
- History of FC.
- Debate seems rather polarized, with proponents hyping parents' expectations, making them instant believers.
- Annotated Bibliography
- seems to be from a proponent
- Pro:
- negative studies often negatively impacted by sterile laboratory setting
- Con
- even practitioners admit to conscious and unconscious control of the output
- Summary website: All Viewpoints
- well, summary w/ their own opinion:
- Contested Words Contested Science, Douglas Biklen and Donald Cardinal 1997. excerpts
- Seems like some users "graduate" to unassisted typing
- There are also hand-on-shoulder techniques.
http://scaut.ugr.es/
- Spanish PDA-based software w/ TTS, summary
- Looks like dynamic display
CareLog and
Abaris
- Automated capture of and access to recordings of sessions with autistic children, with a sensitivity for privacy.
- Supports so-called "evidence-based care."
- Supports "Discrete Trial Training."
- Abaris seems to aid data collection for this type of trial.
- Some slides from 2007.01.25.
- Gregory Abowd and Gillian Hayes at Georgia Tech.
UNORGANIZED JOT
BEFORE NOW, 2007
features of sw i would build:
- web interface in rev 2
- prediction
- full screen
- simple commands and markup
Advantages:
- cheap
- customizable
- if we had the interest, we could build sw to better facilitate communication (like picture programs, etc.)
- better speech software (ive heard its not very good on the lightwriter [eg "guys" --> "guise", etc.]
- most text-to-speech has multiple voices, which is always fun
- completion (type in "mot" and it suggests "mother" "moth" "motor" "motown")
- this sw is pretty cheap, like under $100 / license.
- portable and useful for other stuff
- allowing kids to use internet or other PC functions
Challenges:
- may need to engineer external keyboards to make it less prone to error
- physically larger than a lightwriter, though arguably more cool to a 14 year old :)
- whatever else I havent thought of in the last 10min...
I dont see why we have to play the "we sell a $100 device for $6000 b/c insurance and medicare will pay for it"-game.
http://www.texttospeechblog.com/
Commercial Solutions:
http://www.gusinc.com/Communicators/?gclid=CNKG-czzoowCFSA-QQod7SUf0Q
- they sell complete laptop/handheld solutions
- these are more advanced than below
- uses neospeech
- still $thousands
nextup
- packaged sw, usable w multiple voices
- downloaded demo version of nextup talker
- received uncaught exceptions (bugs)
- poor user interface---complicated, requires mouse
dynavox
- dynamic display devices (like images & stuff)
- lightwriter
- dynawrite
- traditional laptop style keyboard
- looks very portable
- need to get demo of voice capabilities
- still $thousands
- they have autism focused stuff
Prentke Romich
- seems like theyre focused on dynamic display devices
- they also have sw that emulates their interfaces
Raw Text to speech tools:
- cepstral
- http://www.naturalvoices.att.com/demos/
- festival: open source (?) mit stuff: http://www-2.cs.cmu.edu/~awb/festival_demos/index.html
- supports sable
- lexicons of building words from sounds, emphasis, etc.
- http://www-2.cs.cmu.edu/~awb/festival_demos/index.html
- neospeech
TTS quick starter:
- text analysis + waveform synthesis
- waveform synthesis
- concatenative is modern method (database of stored speech)
- diphone synthesis: using two half-tones (b/c the boundaries between phones is the hardest thing to model)
- prosidy: emotion and intonation detection
Markup:
APIs