EECS 442 – Computer vision Shape from reflections

EECS 442 – Computer vision
Shape from reflections
Recovering the shape of an object
A taxonomy
Visual cues: texture  shading  contours  shadows
Recovering the shape of an object
A taxonomy
Visual cues: texture  shading  contours  shadows
Recovering the shape of an object
A taxonomy
Visual cues: texture  shading  contours  shadows
Recovering the shape of an object
A taxonomy
Visual cues: texture  shading  contours  shadows
Recovering the shape of an object
Visual cues: texture  shading  contours  shadows
A taxonomy
Number of observers: monocular  multiple views
camera
Recovering the shape of an object
Visual cues: texture  shading  contours  shadows
A taxonomy
Number of observers: monocular  multiple views
camera 1
camera 2
camera N
Recovering the shape of an object
Visual cues: texture  shading  contours  shadows
Number of observers: monocular  multiple views
Active lighting: laser stripes  structured lighting patterns
A taxonomy
 shadows
camera
Laser/projector/light
Recovering the shape of an object
Visual cues: texture  shading  contours
Number of observers: monocular  multiple views
Active lighting: laser stripes  structured
A taxonomy
lighting patterns  shadows
Limitation: Assumptions on surface reflectance function
Specular
Transparent
Lambertian surfaces
Non Lambertian surfaces
Recovering the shape of an object
Visual cues: texture  shading  contours
Number of observers: monocular  multiple views
Active lighting: laser stripes  structured
A taxonomy
lighting patterns  shadows
Limitation: Assumptions on surface reflectance function
Specular
Transparent
Recovering the shape of an object
Visual cues: texture  shading  contours
Number of observers: monocular  multiple views
Active lighting: laser stripes  structured
Limitations
lighting patterns  shadows
• No intrinsic surface features:
textures – shading…
• No clear contours
Recovering the shape of an object
Visual cues: texture  shading  contours
Number of observers: monocular  multiple views
Active lighting: laser stripes  structured
lighting patterns  shadows
Limitations
• Epipolar geometry doesn’t work
Stereo matching
algorithms fail!
Recovering the shape of an object
Visual cues: texture  shading  contours
Number of observers: monocular  multiple views
Active lighting: laser stripes  structured
Limitations
lighting patterns  shadows
A new perspective
Johann Erdmann Hummel -- The Granite Bowl in the Berlin Lustgarten [1831]
Use specular reflections as additional visual cue
to estimate the surface’s shape
Savarese, et al. CVPR 01, ECCV 02, ECCV 04, IJCV 07
Digital archival
Applications
E-commerce
Digital models of
anatomical parts
Diagnostic of
space structures
Industrial
metrology
Remote sensing
of liquid surfaces
M.C. Escher - Still Life with Spherical Mirror (1934)
Recover local shape of mirror surface by
measuring the deformation of a reflected scene
Mirror surface
Image
Scene
Mirror surface
1. Smooth surface
2. Calibrated scene
Image
C
Scene
Explicit mapping: local scene patch  reflected image patch
From the mapping: reflected image patch  local shape
For now let’s study this mapping
Mirror surface
f (t)
fo
Image
fi (t)
Po
fio
C
P(t) = Po + t P
f : t    f(t)  3
How to find f(t) ?
f(t)  fi (t)
 2
understood
A background note….
fO
PO
How to find fO ?
For spherical mirror:
Alhazen’s problem
Recently
Peter M. Neumann proved
Abu-'Ali Al-Haytham
that the problem is insoluble using a
“compass-andruler” construction
Also named Alhazen
because
the
solution
requires
(ca. 965-1039) - Middle
Ages
extraction of a cube root.
O
Alhazen was an Iranian Philosopher
“Reflections
on reflection
in aextensive
spherical
and Physicist.
Through his
mirror”,
Amer.
1998
researches
on Math.
optics,Monthly,
he has been
considered as the father of modern
Optics.
Mirror surface
f (t)
fo
Image
fi (t)
C
P(t) = Po + t P
- Assume
Why?
fo
is known - how about f( t ), f( t ), at to ?
1 

f ( t )  fo  f ( t o )( t  t o )  f ( t o )( t  t o )2  ...
2
Mirror surface
f (t)
fo
Image
fi (t)
C
P(t) = Po + t P
TOOLS:
GOAL:
f( to ), f( to ), 
• Lagrange Multiplier theorem
• Implicit Function Theorem
• Chen and Arvo, 2000
Summary
• Mapping
v
f (t)
fo
f : t    fo  3
P-O-fo plane
• GOAL :
u
fi (t)
w=N
fio
Po
P(t) = Po + t P
O
A new reference system [u v w]
w = Normal N
v  P-O-fo plane
u=vxw
f( to ), f( to ), 
Summary
• Mapping
v
f (t)
f : t    fo  3
fo
• GOAL :
u
fi (t)
N
w = Normal N
v  P-O-fo plane
u=vxw
fio
Po
P(t) = Po + t P
f( to ), f( to ), 
O
Surface expanded in Monge form:
Summary
• Mapping
f : t    fo  3
w
S(u,v,w) = 0
• GOAL :
f( to ), f( to ), 
w = Normal N
v  P-O-fo plane
u=vxw
v
u
Surface expanded in Monge form:
Summary
• Mapping
v
f (t)
f : t    fo  3
fo
f( to ), f( to ), 
• GOAL :
u
fi (t)
N
w = Normal N
v  P-O-fo plane
u=vxw
Monge form
Qo
Po
P(t) = Po + t P
fi ( to )  A-1 
Similarly, for :
fo , O, Po , ΔP
Observation
O
fo , O, a, b, c
f(t )  B1 
o
Local surface,
scene geometry
Mirror surface
fo
Image
fi (u,v)
Po
v̂
P = [u,v]
û
fio
C
f : u,v    fi (u,v)  3
1  2 1  2 


fi ( u , v )  fio  fu u  fv v  fuu u  fvv v  fuv u v  ...
2
2
Local surface, scene geometry
Mirror surface
fo
Image
fi (u,v)
Po
v̂
P = [u,v]
fio
û
f : u,v    fi (u,v)  3
Conclusion:
image patch = F(scene patch, local shape)
… up to second order
C
Singular specular surfaces
fi ( to )  A-1 
fo , O, Po , ΔP
Observation
fo , O, a, b, c
Theorem: Convex paraboloids are always non-singular
f : t    f(t)  3
not unique
Anish Kapoor
Singular
configurations
f (t)  A 1 (t)
i
inverse problem
Mirror surface
Image
P
Pi
Scene
Inverse problem:
From the mapping: reflected image patch  local shape
Measurements
Unknown
C
Setup
Q
P
= grid of lines
Known quantities:
• Scene
• Camera
Setup
Q
P
= grid of lines
Measured quantities:
• Point position Q
• Orientation of curves at Q
fi (t)
Setup
Q
P
= lines
intersecting
through points
Measured quantities:
Equivalent to
• Point position Q
• Orientation of curves at Q
fi ( t ), fi ( t ), 
• Scale – distribution of points in a neighborhood of Q
at Q
M
Setup
Q
Local geometry we wish to recover:
• Position of the reflection point M
• Normal of the surface at M
• Curvature of the surface at M
• Third order surface parameters around M
Qo
Image plane
Image plane
Surface
geometry
Measurements
• 1 point
Point
position
up to 1
unknown
Tangent
plane
Surface
curvature
Third order
surface
parameters
up to 1
unknown
• 1 point Qo
• Orientation
along 3 lines at Qo


• 3 points



• 5 points



up to 1 unknown

Qo
Image plane
Image plane
Surface
geometry
Measurements
• 1 point
Point
position
up to 1
unknown
Tangent
plane
Surface
curvature
Third order
surface
parameters
up to 1
unknown
• 1 point Qo
• Orientation
along 3 lines at Qo


• 3 points



• 5 points



up to 1 unknown

Summary
Image
plane
Oc
Qo
Q̂o
unknown
surface
s  Oc  Mo
Mo
N
Po
We know:
Po (scene point)
Oc (center of the camera)
Qo
(measured reflect. point)
We don’t know:
• Mo (reflect. point)
?
But:
Mo  sQ̂o
line of sight constraint
Summary
unknown
surface
Oc
Qo
• line of sight constraint:
Q̂o
Mo
N
How about N?
Reflection constraints
•
N  Oc-Mo-Po plane
• distance: s  Oc  Mo
Po
Mo  sQ̂o
Summary
Oc
• distance: s  Oc  Mo
Oc-Mo-Po plane
Qo
N̂ p
Q̂o
r
N
How about N?
• line of sight constraint:
Mo
Mo  sQ̂o
• The principal plane
defined by Po, Oc, Qo
i
Po
Reflection constraints
•
N  Oc-Mo-Po plane
•
i  r
N  F( s )
Summary
Oc
• distance: s  Oc  Mo
principal
plane
Qo
N̂ p
Q̂o
• line of sight constraint:
Mo  sQ̂o
Mo
• The principal plane
defined by Po, Oc, Qo
N
•
N  F( s )
Po
CONCLUSION:
Surface position (Mo) and orientation (N) up to 1 unknown
Qo
Image plane
Image plane
Surface
geometry
Measurements
• 1 point
Point
position
up to 1
unknown
Tangent
plane
Surface
curvature
Third order
surface
parameters
up to 1
unknown
• 1 point Qo
• Orientation
along 3 lines at Qo


• 3 points



• 5 points



up to 1 unknown

Summary
Oc
Q̂o
Qo
̂
• s  Oc  Mo
Mo
• Mo  sQ̂o
fi ( t )
Mo
P(t) = Po + t P
• N  F( s )
•
f(t )
Po
We need a further constraint!
fi ( to )  A-1 
Observation
̂ , fi
s (unknown)
Mo , Oc, Po , ΔP
Mo , Oc, a, b, c
a, b, c (unknown)
known
4
unknowns!
Summary
Oc
• s  Oc  Mo
Qo
Mo
Q̂o
• Mo  sQ̂o
Mo
• N  F( s )
•
N
• F ( ˆ , a, b, c , s )
P1(t) = Po + t P1
P2(t) = Po + t P2
ˆ1 , fi 1
Po
PN(t) = Po + t PN



 f

fi 1  A-1 β1
fi 2  A-1 β2
i
N
 A-1 β N
H
Nx3
ˆ 1,ˆ 2 ,...ˆ N , s
g0
a, b, c , s


det H T H 
 (s)  0
Summary
Oc
• s  Oc  Mo
Qo
Mo
Q̂o
• Mo  sQ̂o
Mo
•
N
• F ( ˆ , a, b, c , s )
•
P1(t) = Po + t P1
P2(t) = Po + t P2
H
Nx3
Po
PN(t) = Po + t PN
Rank THEOREM:
for s=s*, rank(H)=2
s* = actual value of s
• N  F( s )

det HT H

s  s*
0
g 0
s

det HT H

s  s*
0

s*
Summary
Oc
• s  Oc  Mo
Qo
Mo
Q̂o
• Mo  sQ̂o
Mo
N
PN(t) = Po + t PN
•
• F ( ˆ , a, b, c , s )
•
P1(t) = Po + t P1
P2(t) = Po + t P2
• N  F( s )
Po
H
g 0
Nx3

• det HT H

s  s*
 0  s*
CONCLUSION:
Surface position (Mo) and orientation (N) can be computed
using at least 3 orientations
Summary
Oc
• Mo  sQ̂o
Qo
Mo
Q̂o
• N  F( s )
Mo
Hg  0
N
P1(t) = Po + t P1
P2(t) = Po + t P2
PN(t) = Po + t PN
• From the rank theorem
rank(H)=2
H• 3x 3 g  0
ˆ1 ,ˆ2 ,...ˆN ,s*
a, b,c,s*
Po
Rank THEOREM:
for s=s*, rank(H)=2
If s  s*, in general rank(H)=3

det HT H








s  s*
0
a  ao  r a1
b  bo  r b1
c  co  r c1
s*
Summary
Oc
• Mo  sQ̂o
Qo
Mo
Q̂o
• N  F( s )
Mo
Hg  0
N
P1(t) = Po + t P1
P2(t) = Po + t P2
Po
PN(t) = Po + t PN
Theorem: there exists a family of
2nd order surface parameters for
which the observed tangents are
invariant w.r.t. surface curvature
Rank THEOREM:
for s=s*, rank(H)=2
If s  s*, in general rank(H)=3

det HT H








s  s*
0
a  ao  r a1
b  bo  r b1
c  co  r c1
s*
Second order ambiguity







a  ao  r a1
b  bo  r b1
c  co  r c1
Qo
Image plane
Image plane
Surface
geometry
Measurements
• 1 point
Point
position
up to 1
unknown
Tangent
plane
Surface
curvature
Third order
surface
parameters
up to 1
unknown
• 1 point Qo
• Orientation
along 3 lines at Qo


• 3 points



• 5 points



up to 1 unknown

Some experimental results
(3 orientations case)
Experimental results
(2 orientations + scale case)
Conclusions
• Explicit local relationship between scene,
shape and reflections
• Local surface shape can be recovered
up to third order…
… if and only if
- the scene is calibrated,
- at least three (five) points are available in a
neighborhood
Other path minimization
approaches
Dense reconstruction of mirror surfaces
S. Rozenfeld et al. Dense mirroring surface recovery from 1D homographies and sparse
Correspondences, CVPR 07
Such pairs are used to estimate a 1D homography
Dense reconstruction of mirror surfaces
Recover geometry of mixed speculardiffuse objects from two frames.
S. Roth and M.. Black. Specular Flow and the Recovery of Surface Structure. CVPR 06.
Specular flow
Recover radius and location of the sphere
Further reading:
Volumetric approach
•Voxel Carving for Specular Surfaces - T. Bonfort and P. Sturm
•Shape from Distortion: 3D Range Scanning of Mirroring Objects - M. Tarini
, H.Lensch, M.Gösle, HP.Seidel
•Reconstructing Curved Surfaces From Specular Reflection Patterns - M.
Halstead, B. Barsky, S. Klein R. Mandell
•PDE Based Shape from Specularities - J. Solem, H. Aanæs , A. Heyden
•Acquiring a Complete 3D Model from Specular Motion - J.Y. Zheng and A.
Murata
•Structured Highlight Inspection of Specular Surfaces - A.S. Sanderson,
L.E. Weiss, S.K. Nayar
•On Refractive Optical Flow - S. Agarwal, S. Mallick, D. Kriegman, and S.
Belongie
EECS 442 – Computer vision
Next lecture: affine structure from motion