Reconstruction and patte...tti-Petitot-Sarti model

RECONSTRUCTION AND PATTERN
RECOGNITION VIA THE CITTIPETITOT-SARTI MODEL
DARIO PRANDI, J.P. GAUTHIER, U. BOSCAIN LSIS, UNIVERSITÉ DE TOULON & ÉCOLE POLYTECHNIQUE,
PARIS
SÉMINAIRE “STATISTIQUE ET IMAGERIE” 19 JANUARY 2014
0
OUTLINE OF THE TALK:
1. The Citti-Petitot-Sarti model of the primary
visual cortex
2. Image Inpainting
3. Image recognition
THE CITTI-PETITOT-SARTI MODEL OF THE PRIMARY VISUAL CORTEX THE MATHEMATICAL MODEL
ORIGIN OF THE MODEL:
Hoffman (1989): structure of a contact manifold
Petitot (1999): structure of a sub-Riemannian manifold (Heisenberg
group)
THEN REFINED BY:
Citti, Sarti (2006): structure of the rototranslations of the plane
Agrachev, Boscain, Charlot, Gauthier, Rossi, D.P. (2010—)
ALSO STUDIED BY:
Sachkov (2010-)
Duits (2009–)
NEUROPHYSIOLOGICAL FACT
STRUCTURE OF THE PRIMARY VISUAL CORTEX
Hubel and Wiesel (Nobel prize 1981) observed that in the primary visual
cortex V1, groups of neurons are sensitive to both positions and
directions.
Patterns of orientation columns and long-range horizontal connections in V1 of a tree shrew. Taken from
Kaschube et al. (2008).
CITTI-PETITOT-SARTI MODEL
V1 is modeled as the projective tangent bundle:
We need to discuss
RECEPTIVE FIELDS:
How a greyscale visual stimulus
.
is lifted to an state on
SPONTANEOUS EVOLUTION:
How a state on
evolves via neuronal connections.
RECEPTIVE FIELDS
Receptive fields are aimed to describe a family of neurons
with a family of functions
on the image plane.
It is widely accepted that a greyscale visual stimulus
feeds a V1 neuron with an extracellular voltage
given by
A good fit for
is the Gabor filter (sinusoidal wave multiplied by
Gaussian function) centered at
of orientation
in
GROUP THEORETICAL APPROACH
Consider
where
with group operation
is the rotation of .
is the double covering of
, so with appropriate care we can
work here
The left regular (unitary) representation of
acting on
is
The quasi-regular (unitary) representation
is
of
acting on
Definition: A lift operator
is left-invariant if
Theorem: (J.-P. Gauthier, D.P.)
Let be a left-invariant lift such that
is linear,
is densely defined and bounded.
Then, there exists
such that
REMARKS
V1 receptive fields through Gabor filters define a left-invariant lift.
The function
is known as the wavelet
transform of
w.r.t.
.
SPONTANEOUS EVOLUTION IN V1
Two types of connections between neurons:
Lateral connections: Between iso-oriented neurons, in the direction of
their orientation. Represented by the integral lines of the vector field
Local connections: Between neurons in the same hypercolumn.
Represented by the integral lines of the vector field
Connections made by V1 cells of a tree shew. Picture from Bosking et al. (1997)
EVOLUTION IN THE CITTI-PETITOT-SARTI MODEL
Given two independent Wiener process
excitation evolve according to the SDE
Its generator
equation for the evolution of a stimulus
Highly anisotropic evolution
Natural sub-Riemannian interpretation
Invariant under the action of
and
on
, neuron
yields to the hypoelliptic
:
SEMI-DISCRETE MODEL
Conjecture: The visual cortex can detect a finite (small) number of
directions only ( 30).
This suggests to replace
with
.
with
and
LOCAL CONNECTIONS
We replace
follows.
by the jump Markov process
on
defined as
The time of the first jump is exponentially distributed, with probability
on either side.
We obtain a Poisson process with
The infinitesimal generator is
THE EVOLUTION OPERATOR
The semi-discrete evolution operator, acting on
is then
where, for
,
ADVANTAGES
Already naturally discretized the variable.
Invariant w.r.t. semi-discretized rototranslations
.
IMAGE INPAINTING ILLUSORY CONTOURS
Two illusory contours
Neurophysiological assumption: We reconstruct images through the
natural diffusion in V1, which minimizes the energy required to activate
unexcitated neurons.
ANTHROPOMORPHIC IMAGE
RECONSTRUCTION
Let
0.
1.
2.
3.
with
on
be an image corrupted on
Smooth by a Gaussian filter to get generically a Morse function
Lift to
on
Evolve
through the CPS semi-discrete evolution
Project the evolved
back to
REMARKS
Reasonable results for small corruptions
Adding some heuristic procedures yield very good reconstructions
.
SMOOTHING
Even if images are not described by Morse functions, the retina smooths
images through a Gaussian filter (Peichl & Wässle (1979), Marr & Hildreth
(1980))
Theorem (Boscain, Duplex, Gauthier, Rossi 2012)
The convolution of
with a two dimensional Gaussian
centered at the origin is generically a Morse function.
Level lines of Morse functions
SIMPLE LIFT
For simplicity, instead than the lift through Gabor filters we chose to lift
to
EXAMPLE: LIFTING A CURVE
In the continuous model a curve
curve in
The lift of
in
, where
is lifted to
A REMARKABLE FEATURE
When is a Morse function, the lift Lf is supported on a 2D manifold. (This
is false if the angles are not projectivized!)
SEMI-DISCRETE EVOLUTION
The hypoelliptic heat kernel of the semi-discretized operator
has been
explicitly computed, but is impractical from the numerical point of view.
We use the following scheme:
1. For any
compute
2. For any
decoupled ODE on
3. For any
of
the solution
, the Fourier transform of
we let
be the solution of the
.
is the inverse Fourier transform
This scheme can be implemented by discretizing
.
A VIEW THROUGH ALMOST-PERIODIC
FUNCTIONS
To avoid discretizing
we can proceed as follows: Let
be the
grid of pixels of the image and
be the pixel values.
1. Compute
2. Represent
3. Evolve
, the discrete Fourier transform (FFT) of
as the almost-periodic function
splitting the evolution in the ODE’s, for each
The evolution of
is exact!
on
,
PROJECTION
Given the result of the evolution
image by
, we define the reconstructed
RESULTS
Inpainting results from Boscain et al. (2014)
MUMFORD ELASTICA MODEL
With the same techniques it is possible to treat the evolution equation
underlying the Mumford Elastica model, which is associated with the
operator
Comparison of a reconstruction with the Petitot diffusion and the Mumfor diffusion from Boscain, Gauthier, P.,
Remizov (2014)
HIGHLY CORRUPTED IMAGES
RECONSTRUCTION
Based upon this diffusion and certain heuristic complements, we get nice
results on images with more than 85% of pixels missing.
Image from Boscain, Gauthier, P., Remizov (preprint)
IMAGE RECOGNITION
SPECTRAL INVARIANTS
Given a group and a representation of
, a complete set of invariants is a map
functional space such that
on some topological space
, from to some
If the above holds only for and in some residual subset of
are said to be a weakly complete set of invariants.
We speak of spectral invariants whenever the
transform of .
, the
’s
’s depend on the Fourier
IMPORTANT CASES
1. Since any group acts on itself, we look for invariants for the action of the
left regular representation of on
2. Given a semidirect product
, we look for invariants for the
quasi-regular representation of on
.
PLAN
1. Case of an abelian group (e.g. invariants on
w.r.t. translations)
Fourier transform and Pontryagin duality
Bispectral invariants
2. Case of
(non-compact, non-abelian semi-direct product)
Generalized Fourier transform and Chu duality
Bispectral invariants for
Invariants for lifts of functions in
w.r.t. semidiscretized rototranslations.
FOURIER TRANSFORM ON ABELIAN
GROUPS
Let
be a locally compact abelian group with Haar measure
.
The dual
of is the set of of characters of
homomorphism
,
.
, i.e., of continuous group
The Fourier transform of
by
is the map on
Since
defined
is abelian and locally compact, the Fourier transform extends to
an isometry
Fundamental property: For any
w.r.t. the Haar measure on
,
.
EXAMPLE: It holds
, where for
the corresponding element of
is
. The above defined Fourier transform then reduces to
Then,
REMARK
In this case, it is clear that
is indeed a group and that
. This is
an instance of Pontryagin duality, which works on all abelian groups:
Theorem: (Pontryagin duality) The dual of
to . More precisely,
group isomorphism.
is canonically isomorphic
defined as
is a
INVARIANTS FOR ABELIAN GROUPS
Let
be a locally compact abelian group and consider the action on
of the left-regular representation (e.g. for
this corresponds
to translations).
The power spectrum invariants for
left regular representation are the functions
w.r.t. the action of the
These are widely used (e.g. in astronomy), but are not complete:
Fix any
Let
However,
s.t.
, so that
for any
which is not a character of
,
Indeed, if it was the case
equivalent to
,
being a character of
, since
which, by Pontryagin duality, is
.
What is missing in the power spectrum invariants is the phase
information.
The bispectral invariants for
regular representation are the functions
w.r.t. the action of the left
Theorem: The bispectral invariants are weakly complete on
, where
is compact. In particular, they
discriminate on the residual set
of those square-integrable
’s such that
on an open-dense subset of
.
REMARKS
Note that
allows to recover the power spectral invariants
The bispectral invariants are used in several areas of signal processing
(e.g. to identify music timbre and texture, Dubnov et al. (1997))
PROOF OF WEAK COMPLETENESS
Let
be compactly supported and such that
.
Define
, which is a continuous function on an open
and dense set of satisfying
.
Since the bispectral invariants coincide it holds
Since
are compactly supported,
and
are continuous and hence
can be extended to a measurable function on , still satisfying the
above.
Since every measurable character is continuous, this shows that
hence
. That is, there exists
such that
, and
GENERALIZED FOURIER TRANSFORM
Let is a locally compact unimodular group with Haar measure
necessarily abelian.
not
The dual
of is the set of equivalence classes of unitary irreducible
representations of .
The (generalized) Fourier transform of
that to
acting on the Hilbert space
Schmidt operator on
defined by
There exists a measure
is the map
associates the Hilbert-
(the Plancherel measure) on
Fourier transform can be extended to an isometry
Fundamental property: For any
,
w.r.t. the
.
CHU DUALITY
Chu duality is an extension of the dualities of Pontryagin (for abelian
groups) and Tannaka (for compact groups) to certain (non-compact) MAP
groups. Here the difficulty is to find a suitable notion of bidual, carrying a
group structure. See Heyer (1973).
Let
be a topological group
is the set of all -dimensional continuous unitary
representations of in
. It is endowed with the compact-open
topology.
The Chu dual of is the topological sum
is second countable if
is so.
QUASI-REPRESENTATIONS
A quasi-representation of is a continuous map from
such that for any
and
to
1.
2.
3.
4.
The Chu quasi-dual of is the union
of all quasirepresentations of endowed with the compact-open topology.
Setting
and
,
is a
Hausdorff topological group with identity .
The mapping
defined by
is a
continuous homomorphis, injective if the group is MAP.
Definition: The group has the Chu duality property if
topological isomorphism.
is a
CHU DUAL
A Moore group is a group whose irreducible representations are all finitedimensional.
Theorem (Chu)
The following inclusions hold
The group
is Moore (i.e. all its irreducible representations are
finite-dimensional) and then it has Chu duality.
The group
is not MAP (i.e. almost-periodic functions do not form
a dense subspace of continuous functions) and hence it does not have
Cuu duality.
This is why it is more convenient to work in the semi-discretized model
THE CASE OF Let us consider the (non-compact, non-abelian) Moore group
.
The unitary irreducible representations fall into two classes
Characters: Any
.
induces the one-dimensional representation
-dimensional representations: For any
representation that acts on
as
we have the
where
is the shift operator
.
Since the Plancherel measure is supported on the -dimensional
representations, bispectral invariants are generalized to
as the following functions of
:
THE LEFT REGULAR REPRESENTATION
Theorem: The bispectral invariants are weakly complete w.r.t. the
action of on
, where
is
compact. In particular, they discriminate on the residual set
of those square-integrable ’s such that
is invertible for in an
open-dense subset of
.
The proof is similar to the abelian one: Given
Let
Prove that
Since
with
:
for
s.t.
is invertible.
can be extended to a quasi-representation
is Moore, it has Chu duality and hence there exists
s.t.
for any unitary representation
Finally, this implies
THE QUASI-REGULAR
REPRESENTATION
Consider now the quasi-regular representation
corresponds to rotation and translations.
Fixed a lift
invariants
as
acting on
, which
we define the bispectral
.
Corollary: Let
be an injective leftinvariant lift. Then, for any compact
the bispectral invariants
are complete for the action of on the subset
of
such
that
is invertible for
in an open-dense subset of
Proof:
Let
be such that
. Then
.
Unfortunately, the set
is empty for regular left-invariant lifts:
Let
Since
be the vector
Thus
has at most rank
.
we have
and hence
.
Conjecture: The bispectral invariants are weakly complete w.r.t. the
action of on compactly supported functions of
.
REMARK
There exists non left-invariant lifts for which
lift
is residual, as the cyclic
The price to pay is that we have to quotient away the translations before
the lift. This suggest the following.
ROTATIONAL BISPECTRAL INVARIANTS
The rotational bispectral invariants for
and any
the quantities
Observe that
A function
are, for any
is invariant under rotations but not under translations.
is weakly cyclic if
is a basis of
for a.e. .
Modifying the arguments used in the case of the left-regular
representation, we can then prove the following.
Theorem: Consider a regular left-invariant lift with weakly cyclic and
such that
a.e.. Then, the rotational bispectral invariants are
weakly complete w.r.t. the action of rotations on
for any
. More precisely, they discriminate on weakly cyclic functions.
EXPERIMENTAL RESULTS
In Smach et al. (2008), although the theory was not complete, some tests
on standard academic databases have been carried out. They yielded
results superior to standard strategies.
ZM denotes the standard Zernike moments
MD are the (non-complete) power spectrum invariants
MD are the bispectral invariants
Sample objects from and results obtained on the COIL-100 noisy database (from Smach et al.).
OTHER EXPERIMENTAL RESULTS
Sample objects from and results obtained on the faces ORL database (from Smach et al.).
TEXTURE RECOGNITION
Let
be countable and invariant under the action of
.
A natural model for texture discrimination are almost periodic functions
on
in the
Besicovitch class, i.e.,
functions are the pull-back of
functions on the Bohr
compactification.
The theory above can be adapted to these spaces of functions, and an
analog of the weakly completeness of rotational bispectral invariants
holds.
In this space the bispectral invariants are not complete, and thus the
(analog of) the above conjecture is false.
As already mentioned, when is finite this space can be used to exacty
solve the hypoelliptic diffusion.
REMARK: SETTING
The considerations of this part of the talk work in the general context of a
semi-direct product
where
is an abelian locally compact group
is a finite group
the Haar measure on is invariant under the action
the natural action of on the dual
has non-trivial stabilizer only
w.r.t. the identity
.
The group law on is non-commutative:
In our case:
,
and
is the rotation of
.
,
THANK YOU FOR YOUR ATTENTION