The latest Smart Phones feature an assortment of multimedia capabilities (cameras, mp3 playback), support local (Bluetooth, Infrared) and global (GPRS/UMTS) communication facilities and can be programmable more easily than ever before. As it was explained in the MobileEye section, those factors motivated our adaptation of the TRIP technology to camera phones. Now, with the MobileSense system we want to go one step forward. We aim to add to a mobile not only the sight sense but also other capabilities such as hearing or speaking. In fact, MobileSense is the combination of three services: MobileEye, MobileEar and MobileVoice.

MobileSense enables a mobile to see TRIP ringcodes, read text through OCR, distinguish colours, capture the user's voice or synthesize voice from text. Rather than undertaking those tasks on the mobile itself, MobileSense benefits from the mobile´s Bluetooth capabilities to delegate the processing of the captured multimedia content to a nearby Bluetooth server. The picture below illustrates the architecture of the MobileSense system. The open source tools used to process the multimedia content are:

The MobileSense system transforms a mobile phone into a device that can help blind or deaf people to augment their sensorial capabilities. By means of MobileSense, a blind person may, for example,  be able of determining what is written on a plate outside an office. MobileSense will apply OCR to the captured image through GOCR and latter synthesize a WAV file (through FreeTTS) that will be reproduced back in the mobile dictating what it was read.

In order to add accessibility features to the maximum spectrum of mobile phones, we have only assumed that the mobiles utilised are MIDP 2.0 compliant and support the MMAPI and Bluetooth optional packages. Likewise, for the first release of the system we have assumed the user is located in a controlled environment (university, hospital) where there are a group of reachable Bluetooth servers capable of undertaking the multimedia content processing tasks. We will soon extend the system so that the MobileSense processing capabilities are available globally through a GPRS reachable web server. The following links give you access to both the server and client-side of the MobileSense system. A README file is also provided which offers detailed explanations on how to install MobileSense.

README File with Installation and Execution Instructions

Read carefully this file to learn how to compile and execute the MobileSense system.

Download Source and Binary Code for MobileSense MIDlet

Java application for a MIDP 2.0 compliant mobile phone equipped with the MMAPI and Bluetooth optional packages.

Download Source code of MobileSense Server

Java implemented Bluetooth server that processes multimedia content by means of OCR, TRIP, Colour Detection and Speech Synthesis. Apart from the MobileSense server's source code, you will find utilities to generate TRIP tags, binary versions of the open source tools used and a database containing mappings from IDs to the objects represented by them.

Send mail to d i p i n a [at] e s i d e [dot] d e u s t o [dot] e s with questions or comments about this web site.
Last modified: 20-12-04