Presentation slides

http://www.imagelab.ing.unimore.it
Post-CVPR AC Meeting
Workshop on Recent trends in computer vision
University of Maryland , Feb 2014
The challenge of tracking
Social Groups in Crowd
Rita Cucchiara, Simone Calderara, Francesco Solera
Imagelab
DIPARTIMENTO DI INGEGNERIA «Enzo Ferrari»
Università di Modena e Reggio Emilia, Italia
UNIMORE
University of Modena and Reggio Emilia
ImageLab: current projects..
Computer Vision
Pattern Recognition
and machine learning
Multimedia
(NATO, EU projects, collaboration with
companies, SMART TOURISM project
Emilia R. Region with S.Calderara)
Sensing floors (collaboration
with FLORIM spa with R.Vezzani)
Document analysis
(collaboration with Treccani Italian
Enciclopedy and Miniature Libs with
C.Grana)
Ego-Vision
Web image retrieval for
cultural heritage (ITALIAN PON
(collaboration with ETHZ
with G.Serra)
Animal behavior (collaboration
with Italian Health Ministry with
S.Calderara)
Project EU-FESR DICET)
Medical imaging
(EU projects in dermatology , C.Grana)
Natural interaction for
children
( Cluster Project smart city CITYEDU)
University of Modena and Reggio Emilia
Surveillance
UNIMORE
Current PROJECTS
In Smart City Projects..
From surveillance to human behavior analysis….
Goal
IN GROUPS
IN THE CROWD
TOURISTIC TOURS – CULTURE ENTERTAINMENT – CHILDREN IN SCHOOLS
University of Modena and Reggio Emilia
ALONE
UNIMORE
Understand what the people want/their intentions in the city while they are:
Reasoning about CROWDS
What is a CROWD?
We are working on crowds
where single person and
groups can be recognized.
University of Modena and Reggio Emilia
What does LARGE mean?
UNIMORE
a large number of persons gathered closely together
Before understanding groups..
ENVI-VISION
EGO-VISION
Many challenges:
University of Modena and Reggio Emilia
UNIMORE
What we are doing at Imagelab:
• Detecting single people
• Tracking single people
• Tracking multiple people
• Working on trajectories (or tracklets)
• Recognizing (socially consistent) groups in crowd
• By shape classification
• By trajectory analysis
Detecting people ..
Pedestrian detectors a long story…
Improving speed and accuracy
“Multi-Stage Particle Windows for
Fast and Accurate Object Detection”
[Gualdi, Prati, Cucchiara TPAMI12]
form sliding windows to particle windows
search for people (and other targets)
University of Modena and Reggio Emilia
Detectors: Dalal, Triggs CVPR05, Felzenszwalb, CVPR08, Gavrila et al PAMI09……….
Benchmarks: Dollar et al CVPR09
Search modes : Lampert et al CVPR08
Detection in crowd: Ge Collins PETS09, Li et al. CVPR13
Detection and tracking in crowd: Rodriguez et al. ICCV11
Survey: Dollar et al TPAMI11…
UNIMORE
•
•
•
•
•
•
..and tracking single (people) target
Is tracking a solved problem?
Another long story from L.Davis W4 CVPR98 ICIAP99…..
- a large set of performance measures
- a large experimentation
(with code available over 3 clusters in 3 labs)
MOTA; OTA; Deviaton….
F-Measure
SURVIVAL CURVES..
19 trackers
BASELINES
STATE OF THE ART
* D.Chu, A.Smeulders, S.Calderara, R.Cucchiara, A. Dehghan, M.Shah Visual Tracking: an Experimental Survey
[TPAMI 2013]
University of Modena and Reggio Emilia
- a very large dataset
of 14 categories of challenges
UNIMORE
We tried to answer this questions in an “experimental evaluation”
Even in case of single target tracking*
19 Trackers
A.
Tracking by Matching
•
•
•
[FRT] Fragments-based Robust Tracking
A. Adam, E. Rivlin, and I. Shimshoni, CVPR2006
[KLT] Lucas-Kanade Tracker
[MST] Mean Shift Tracking
S. Baker and I. Matthews, IJCV2004
D. Comaniciu, V. Ramesh, and P. Meer, CVPR2000
[KAT] Kalman Appearance Tracker
•
[LOT] Locally Orderless Tracking
H. Nguyen and A. Smeulders, TPAMI 2004
B.
S. Oron, A. Bar-Hillel, D. Levi, S. Avidan, CVPR2012
Tracking by Matching with extended model (ST memory)
•
[IVT] Incremental Visual Tracking
D. Ross, J. Lim, and R.S.Lin, IJCV2008
•
•
[TST] Tracking by Sampling Trackers
J. Kwon, K.M. Lee, ICCV 2011
[TAG] Tracking on the Affine Group
J. Kwon and F.C. Park, CVPR2009
C.
Tracking by Matching with constraints
•
[TMC] Tracking by Monte Carlo sampling
J. Kwon, K.M. Lee,CVPR 2009
•
[ACT] Adaptive Coupled-layer Tracking
•
[L1T] L1-minimization Tracker
X. Mei and H. Ling, ICCV2009
•
[L1O] L1 Tracker with Occlusion detection
X. Mei, H. Ling, Y. Wu, E. Blasch, L. Bai, CVPR2011
L. Cehovin, M. Kristan, A. Leonardis, ICCV2011
D.
Tracking by Discriminant Classification
•
[MIT] Multiple Instance learning Tracking
B. Babenko, M.H. Yang, and S. Belongie, CVPR2009
•
•
[FBT] Foreground-Background Tracker
H. Nguyen and A. Smeulders, 2006, IJCV2010
•
[HBT] Hough-Based Tracking
[TLD] Tracking, Learning and Detection
M. Godec, P.M. Roth, H.Bischof, ICCV2011
[SPT] Super Pixel tracking
Z. Kalal, J. Matas, and K. Mikolajczyk, CVPR2010
E.
Tracking by discriminant Classification
with constraints
S. Wang, H. Lu, F. Yang, M.H. Yang, ICCV2011
•
[STR] STRuck
S. Hare, A. Saffari, P. Torr, ICCV2011
University of Modena and Reggio Emilia
K. Briechle and U. Hanebeck, SPIE 2001
UNIMORE
•
[NCC] Normalized Cross-Correlation
14 tracking challenges in 313 videos
01-LIGHT
02-SURFACECOVER
03-SPECULARITY
06-MOTIONSMOOTHNESS
07-MOTIONCOHERENCE
08-CLUTTER
09-CONFUSION
10-LOWCONTRAST
11-OCCLUSION
12-MOVINGCAMERA
13-ZOOMINGCAMERA
14-LONGDURATION
University of Modena and Reggio Emilia
05-SHAPE
UNIMORE
04-TRANSPARENCY
The dataset: an example
http://www.alov300.org or http://imagelab.ing.unimo.it/dsm
University of Modena and Reggio Emilia
UNIMORE
email to simone.calderara@unimore.it
A comprehensive view Survival curve
The upper bound, taking the best
of all trackers at each frame 10%
About the 30%, correctly tracked only
[TST]
A
[L1O]
B
[NCC]
[TLD]
C
D
E
The lower bound, what all
trackers can do 7%
University of Modena and Reggio Emilia
[STR]
UNIMORE
[FBT]
Confusion challenge: trackers comparison
CONFUSION.. CROWD short term tracking
University of Modena and Reggio Emilia
UNIMORE
[FBT][NCC][STR] [TLD][TST]
[L1O]
Long term challenge: trackers comparison
University of Modena and Reggio Emilia
UNIMORE
[FBT][NCC][STR] [TLD][TST]
[L1O]
We need more effort
Welcome to
“Long term tracking workshop”
at CVPR2014
What we learned?
• State of the art papers
•
•
•
•
Discrete –continue optimization Andriyenko et al CVPR2012
Continue energy minimization Milan and Roth PAMI2014
Generalized minim clique Zamire et al ECCV2012
K-shortest path optimization Berclaz et al PAMI 2011
What do they all have in common?
They are data association techniques that work on already detected pedestrians
University of Modena and Reggio Emilia
• Moving from single target to multiple targets in long term cannot be done
with multiple instances of a good single-target tracker
UNIMORE
Many observations…
• In cluttered and confusion scenes, Tracking-by-detection methods that
use data association , based on discriminative classifiers seem to be
promising…..
Work in progress…
•
•
http://imagelab.ing.unimore.it/files2/RitaWashington/video/influence
zones_tracking.avi
We use distance only when is possible
Motion prediction and appearance is a plus when useful
Thus?
1. Split the crowd in influence zones
(latent knowledge)
2. Decide whether those zones are
ambiguous (also latent)
3. Solve unambiguous associations with
distance only
4. Employ different level features in
ambiguous cases ( ask for shapes,
color.. edges.. motion)
University of Modena and Reggio Emilia
(Kahnemann, Treisman, Gibbs 1995)
UNIMORE
Cognitive Visual Tracking with latent structural svms
• From neuroscience : two (connected but different) areas for detection ( people,
faces..) and spatio temporal localization (independently by their shapes)
• From perceptual psychology : the “object file” theory
Detection and tracklets
University of Modena and Reggio Emilia
UNIMORE
Survival curve
With a perfect detector
University of Modena and Reggio Emilia
UNIMORE
With a detector with errors
[KSP] Multiple Object Tracking using K-Shortest Paths Optimization
J. Berclaz, F. Fleuret, E. Türetken and P. Fua, PAMI 2011
Groups of People
If tracking was solved…
University of Modena and Reggio Emilia
UNIMORE
If we were given the trajectories of every pedestrian in the scene (more or
less).
would we be able to discern the presence of groups?
Detecting social groups in crowds
Group detection: learn to partition into groups the pedestrians being part of
a crowd observing pairwise relations and transitivities.*
•
Hall’s proxemics theory 1 defines reaction
bubbles around every individual and
•
the interaction between pairs of
individuals can be classified according to a
quantization of their mutual distance
2. GRANGER CAUSALITY
•
Intuition: two pedestrian belonging to the
same group will probably influence each
other position and direction!2
•
The Granger causality test is a statistical
hypothesis test for determining whether one
time series is useful in forecasting another
* Structured learning for detection of social groups in crowd Solera, Calderara, Cucchiara, AVSS 2013
University of Modena and Reggio Emilia
1. HALL’S PROXEMICS
UNIMORE
Integrating two cues:
Results
Features: Proxemics and Granger causality
Structure function: pair-wise correlation clustering
Group detection: Structured SVM [groups]
University of Modena and Reggio Emilia
UNIMORE
Conclusions and Open Problems
• Detection Social groups and interactions
• interesting and growing topic
• Many many many applications
• Social hypotheses Must be considered
.
Detection
tracking
People/ group
Detection
People/ group
tracking
People
Detection
Social group
Detection
People/ group
Tracking
University of Modena and Reggio Emilia
• Multiple target tracking
• more and more challenging ( more if real-time is required)
• tracking-by-detection
People/ group
People/ group
• Cognitive assumptions are useful
UNIMORE
• Single target Detection & Tracking
• tracking is (still) an open problem
• computer visionaries are working a lot.. ( also in the weekend )
Thanks
Giuseppe Serra
Marco Manfredi
Costantino Grana
Paolo Santinelli
Francesco Solera
Roberto Vezzani
Martino Lombardi
Simone Pistocchi
Simone Calderara
Michele Fornaciari
Fabio Battilani
Augusto Pieracci
Dalia Coppi
Patrizia Varini
University of Modena and Reggio Emilia
THANKS!
Rita Cucchiara
UNIMORE
PEOPLE @ http://imagelab.ing.unimore.it
University of Modena and Reggio Emilia
UNIMORE