Ubiquitous Compu-ng and Urban Organisms
Nicola Bicocchi – nicola.bicocchi@unimore.it
“Any technology sufficiently advanced is indistinguishable from magic”
Arthur C. Clarke
“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they
are indistinguishable from it”
Mark Weiser
Introduc-on
Mainframe (Past) 1:N one computer shared by many people
â
Personal Computer (Present) 1:1 one computer, one person
å æ
N:1 *Internet - Widespread
Distributed Computing* Ubiquitous Nk:1
Computing
Trends
Computers were a scarce resource run by experts behind closed doors.
Phase I -‐ Mainframes
• In 1984 the number of people using PCs surpassed that of people using mainframe computers.
• PC Era: You have your computer, it contains your stuff, and you interact directly and
deeply with it.
• The PC is most analogous to the automobile.
Phase II -‐ PC
The Internet brings together elements of the mainframe
era and the PC era.
⇓
Client = PC
Server = Mainframe
Transi-on – The Internet
• The UC era will have lots of computers shared by each one of us.
• UC is fundamentally characterized by the connection of things in the world with
computation.
• Frequently used related terms:
Pervasive computing, Wearable computers, Intelligent environment, Things That Think (T³), Wearware, Personal Area Networking (PAN).[3]
Phase III – Ubiquitous Compu-ng
UC -‐ Defini-on
• Ubiquitous compu-ng [4]:
– Ubiquity/Pervasiveness – lots of devices
– Connectedness – the devices are networked
– Context-‐awareness – the system is aware of the context of users
– Invisibility – device effec-vely becomes invisible (implicit interac-ons)
• UC goal: enhancing computer use by making many computers available throughout the
physical environment, but making them effectively invisible to the user.
• Ubiquity
– Everywhere
– Adaptation to environment
– Intuitive, transparent, natural interfaces
UC -‐ Goal
From UC to Urban Organisms
• Business: estimated 10 T$ worldwide (next 15y) – Motley Fool
• Personal: increased productivity
• Health: physical & psychological influence, decreased alienation
• Privacy and Security: ?!?!?!?
Social Influence
A vision-‐based sensor
Mo-va-ons
• Pervasive services oSen rely on mul--‐modal
classifica-on to implement situa-on-‐recogni-on capabili-es (e.g., loca-on, physical ac-vi-es,
health parameters, daily rou-nes, etc.)
• Vision is an informa-ve source of informa-on.
Few aWempts have been made to integrate it within pervasive scenarios
• Is it possible of make use of visual data within mul5-‐modal situa5on recogni5on schemes?
• Images are extremely informa-ve (vast areas of the human brain are devoted to vision)
• The demand of life-‐logging devices is increasing
• Decent-‐quality, miniaturized cameras are already affordable
• It is s-ll troublesome to detect and classify a wide range of objects, people, and scenes
• Many research works focus on recognizing specific classes of en--es with complex parametric representa-ons
• UnaWrac-ve for constrained devices dealing with
unpredictable, open-‐ended environments (a common pervasive scenario)
• Wearable cameras frequently collect meaningless shots
The good and the bad
Goal
SCENE = Laboratory
Situation Recognition
Workflow
1. Image acquisi1on 2. Scene segmenta1on
3. Sub-‐scenes classifica1on 4. Scene reconstruc1on
person
robot robot person
SCENE = Laboratory
Scene Segmenta-on
• Scene classifica-on stage works with 32x32 pixels images
• To avoid drama-c informa-on losses, segmenta-on proved to be useful
Scene Segmenta-on
• GrabCut has been implemented to segment images into a set of sub-‐images
containing foreground areas. To make it completely unsupervised, border pixels have been used as the background model
C. Rother, V. Kolmogorov, A. Blake. GrabCut: Interac-ve Foreground Extrac-on using Iterated Graph Cuts. ACM Transac5ons on Graphics (SIGGRAPH'04), 2004
Neighbors Search
• Data-‐driven
• Neighbors are searched by execu-ng an exhaus-ve search on the largest dataset publicly available
• (images = 80M, size = 0.3TB, search -me = 1h)
• Hashing techniques can be used to substan5ally speed up the process
• Images are represented as ordered vectors of 3072
(32x32x3) elements normalized to have zero mean and unit norm
• The sum of squared differences is computed
• Each query returns n nearest neighbors, each one associated with a WordNet label
Neighbors Search
80 Million Tiny Images Dataset (MIT), publicly available.
Neighbors vo-ng
ConceptNet
• The ideal knowledge base should exhibit two main features:
– it should include a vocabulary covering a wide scope of topics – it should incorporate seman-c rela-ons between concepts
• ConceptNet is organized as a massive directed and labelled graph about real world facts and activities
– 300K nodes – 1.7M edges
Scene Reconstruc-on
• It is difficult to recognize a scene from low-‐quality images. However:
– mul-ple images can be collected
– scenes can be treated as sets of keywords – similar scenes share a number of keywords
• Each class, interes-ng for classifica-on, is marked on ConceptNet as a class node C
• Each label is marked as a label node l
• For each class node C, the average shortest path AP to all the label nodes is computed
• Scene reconstruc-on is performed by assigning to the -me window taken into account the class C with the shortest average path
Kitchen (C)
Park (C)
Scene Reconstruc-on
ARMCHAIR
OVEN KNIFE
COUCH
STOVE
TELEVISION
LIVING-ROOM KITCHEN
CURTAIN
GUITAR PILLOW
ROOM SINK
Scene Reconstruc-on
ARMCHAIR
OVEN KNIFE
COUCH
STOVE
TELEVISION
LIVING-ROOM KITCHEN
CURTAIN
GUITAR PILLOW
ROOM SINK
Experimental Methodology
• GoPro HD camera
• 1920 images dataset
• 16h sampled every 30s during a realis5c ordinary day
• Automa-cally -me stamped, manually annotated
• 64 neighbors retrieved for each image query
• 300s -me window
• The proposed approach (segmenta-on, classifica-on, reconstruc-on) has been compared with a the original WordNet vo-ng scheme along the same -me window
Experimental Results
0 0.2 0.4 0.6 0.8 1
0 0.2 0.4 0.6 0.8 1
Precision
Recall
WordNet w/o Grabcut ConceptNet w/ Grabcut
Domes1c Environments
Conclusions
• Vision is a rich source of informa-on for pervasive system
• Vision can be used for situa-on recogni-on in open-‐ended, unpredictable environments by combining:
• Unsupervised image segmenta-on techniques
• Data-‐driven, non-‐parametric classifica-on models
• Commonsense knowledge
• Commonsense knowledge allows vision-‐based sensors to be integrated with other informa-on sources for both:
• Improving classifica-on accuracy
• Dealing with missing labels
A Self-‐Aware, Reconfigurable Architecture
for Context Awareness
Mo-va-ons
• Autonomous machines need sensors to collect knowledge, and mechanisms to adapt their behavior to the
environment
• Collec-ng knowledge basically consists in finding a paWern recogni-on algorithm delivering certain performance on a given dataset.
• Researchers are oSen re-‐inven-ng the wheel on each specific problem:
– Kinect smart cameras
– Amazon recommenda-on systems – Shazam
– An- Spam filters – Network profilers
30
Requirements
• PaWern Recogni-on modules requirements:
– Self-‐Aware
– Environment-‐aware
– Able to integrate a-‐priori specifica-ons (Top-‐Down Design)
• Architecture requirements:
– Improve classifica-on accuracy
– Eventually reduce energy consump-on on constrained devices
– Simplify training setup
– Improve soSware engineering
31
Goal
• We developed a general architecture enabling the development of paWern recogni-on soSware modules.
• We made use of reconfigurable components (OSGi containers) to enable self-‐ and
environmental-‐based adapta-on
• We embedded well-‐known paWern recogni-on libraries (Weka, Jmir, OpenCV) within OSGi
containers
• We defined an automata-‐based meta language to program internal reconfigura-ons
32
Internal Architecture
• OSGi, is a Java framework providing the features of a Service oriented Component architecture.
Components can be installed, started, stopped, updated, and uninstalled without requiring a reboot.
• Apache Felix iPOJO is a container-‐based framework suppor-ng some management facili-es like dynamic dependency handling, component reconfigura-on, component factory, and
introspec-on.
• Apache Camel enables components to process data streams in an asynchronous way
33
Internal Architecture
34
• Simplify training setup
• Improve soOware engineering
Code Perspec-ve
35
Smart Camera Applica-on
36
Life-‐Log Applica-on
37
indoor
A Start
B C
outdoor D
fast
slow
other outdoor
car OR train OR bus indoor
indoor
Speed (fast, slow) Location_1 (indoor, outdoor) Activity_1 (walk, run, stand, stair)
Vehicle (car, train, bus, bike, other) Talking(no voice, phone, dialogue, group) Location_1 (indoor, outdoor)
F run
run stand, walk, stair
Activity_1 (walk, run, stand, stair) Location_3 (park, street) Location_1 (indoor, outdoor)
Location_2 (home, office, other)
E
Location_4 (pub, restaurant, concert, gym, stadium, other)
other other
Life-‐Log Applica-on
– Dataset collected by 2 users for 2 days
– Improve
classifica1on
accuracy (+10%) – Eventually reduce
energy
consump1on on constrained
devices (-‐50%)
38
Conclusion
• We developed a general architecture capable of simplifying the development of paWern
recogni-on modules able to:
– Improve classifica1on accuracy
– Reduce energy consump1on on constrained devices – Simplify training setup
– Improve soOware engineering
• Available on BitBucket:
hWps://bitbucket.org/damiano_fontana/
awareness
39 Nicola Bicocchi (UNIMORE)
Thank you for listening!
• [1] Bruce Sterling speech at the "CRA Conference on Grand Research Challenges in Computer Science and Engineering“, Airlie House,
Warrenton, Virginia June 23, 2002.
• [2] Jim Morris of Carnegie-Mellon University
• [3] MIT: http://www.mit.edu, http://oxygen.lcs.mit.edu/
• [4] Mark Burnett, Chris P. Rainsford, Department of Defense,
Australia, “A Hybrid Evaluation Approach for Ubiquitous Computing Environments”
• [5] Mark Weiser, “Some Computer Science Issues in Ubiquitous Computing”, July 1993.
• [6] Mark Weiser, “Ubiquitous Computing” IEEE Computer "Hot Topics", October 1993.
• [7] Mark Weiser and John Seely Brown, Xerox PARC
“The Coming Age of Calm Technology ”, October 1996.
• [8] Xerox PARC: http://www.xerox.com
• Ubicomp: http://www.ubiq.com
• [9] www.cs.hut.fi/Opinnot/Tik-86.161/2001_files/Merviranta.pdf
• [10] Computer History:
http://www.computersciencelab.com/History.htm
