Introducing the NicIcon database of handwritten icons

Ralph Niels, Don Willems and Louis Vuurpijl
Introducing
of handwritten icons
Introducing the NicIcon database
of handwritten icons
NicI
Icon
Ralph Niels
Don Willems
Louis Vuurpijl
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
Overview
• Introduction
• Domain: crisis management
• Data collection
• Icon design
• Method
• Data
• Classification experiment
• Method
• Results
• Conclusion and URLNEW!
=
Introducing the NicIcon database
of handwritten icons
Crisis management (CM)
• Scenario: tunnel disaster
• Distributed computer systems to support CM
• Multiple modalities: speech, gestures, and pen
The car
is here
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
Iconic pen gestures
• Faster than handwriting
• Easy to learn & remember
• Visual meaningful shape
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
Ralph Niels
Don Willems
Louis Vuurpijl
Database
• Public databases available for several handwriting
applications
• Not for iconic pen gestures
• Based on symbology reference by USA Homeland Security
Workgroup
– Used in e.g., USA, Australia, New-Zealand
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
Icons
Flood
Electricity
Accident
Car
Casualty
Injury
Fire
Gas
Bomb
Roadblock
Fire brigade
Paramedics
Police
Person
Introducing the NicIcon database
of handwritten icons
Data collection
•
•
•
•
35 participants
Online and offline
Variation in size
Per person:
– 22 pages
– 55 instances / icon
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
Database content
• Online: 26,163 iconic gestures
(24,144 reported in paper)
Tablet: Wacom Intuos2 A4 oversize
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
Database content
• Offline: 770 scanned pages
Scanner: HP Scanjet 7400C flatbed
(@ 300dpi, 24 bit colour)
Ralph Niels
Don Willems
Louis Vuurpijl
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
Classification experiment
Data sub sets
• Stratified
Train set
(36%)
Complete
Testdata
set set Evaluation set
(26,163
(24%)
icons)
(40%)
• Writer dependent (WD), writer independent (WI)
WD
Train set
(36%)
WD
Test set
(24%)
WD
Evaluation set
(40%)
WI
Train set
(36%)
WI
Test set
(24%)
WI
Evaluation set
(40%)
Introducing the NicIcon database
of handwritten icons
Classification experiment
on online data
Multi classifier system
• Feature sets
• 28 geometric features (Willems & Vuurpijl)
• 30/60 coordinates running features (Schomaker & Vuurpijl)
• 1185 features (Willems, Niels, Van Gerven, Vuurpijl)
NEW!
• Feature classifiers
• Support Vector Machine
• Multi-Layered Perceptron
• Template matching
• Dynamic Time Warping (Niels & Vuurpijl)
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
New feature set (‘m-fs’)
• Features from literature
– Geometrical, temporal, pressure
D. Rubine, Specifying gestures by example, Computer Graphics.
25 (4) (1991) 329–337.
J. LaViola Jr. & R. Zeleznik, A practical approach for writerdependent symbol recognition using a writer-independent symbol
recognizer, IEEE Transactions on pattern analysis and machine
intelligence. 29 (11) (2007) 1917–1926.
L. Zhang & Z. Sun, An experimental comparison of machine
learning for adaptive sketch recognition, Applied Mathematics and
Computation. 185 (2) (2007) 1138–1148.
and many others…
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
New feature set (‘m-fs’)
• Over complete icon, but also…
– Mean over strokes
– Std. dev. over strokes
Feature definitions in technical report at website
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
New feature set (‘m-fs’)
• Feature selection
– 1185 features, which are the best?
– Sort them on best individual performance
– Add 1 by 1 to classifier
• Performance maximizes at:
– 545 features for WI
– 660 features for WD
• Selected features:
– ± 1/3 full icon
– ± 1/3 mean
– ± 1/3 standard deviation
The best features
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
Ralph Niels
Don Willems
Louis Vuurpijl
Breaking news: the best features
Area
Sine first /
last sample
Length of
diagonal
Vertical
offsets
Average
centroidal
radius
Introducing the NicIcon database
of handwritten icons
Ralph Niels
Don Willems
Louis Vuurpijl
Classification results
Classifier
Features
WD
WI
SVM
g-28
84.8
73.0
m-fs
99.3
96.4
af-30
98.1
90.3
af-60
98.0
98.6
g-28
84.0
78.5
m-fs
98.7
96.3
af-30
94.5
85.2
af-60
94.4
85.1
DTW
-
98.4
93.6
MCS
-
99.51
97.83
MLP
g-28: 28 geometric features
(Willems & Vuurpijl)
m-fs: 1185 features
NEW!
(Willems, Niels, Van Gerven, Vuurpijl)
af-30/af-60: 30/60 running features
(Schomaker & Vuurpijl)
DTW: Dynamic Time Warping
(Niels & Vuurpijl)
MCS: Multiple classifier system
(majority voting)
Introducing the NicIcon database
of handwritten icons
Misclassifications
• Wrong box
• Sloppy drawing
• Retracing
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
Discussion
• Results already quite good (WD: 99.5%, WI: 97.8%),
but gain is still possible
• Which features are important?
– For different domains
• Performance in interactive experiment
• Offline data still open
• Mapping online -> offline
Ralph Niels
Don Willems
Louis Vuurpijl
Introducing the NicIcon database
of handwritten icons
http://unipen.nici.ru.nl/NicIcon
Freely available:
• Online (Unipen)
• Offline (PNG)
• Technical report about
new feature set
Ralph Niels
Don Willems
Louis Vuurpijl