Hopfield Model – Continuous Case

Hopfield Model – Continuous Case

The Hopfield model can be generalized using continuous activation functions.

More plausible model.

In this case:

where is a continuous, increasing, non linear function.

Examples

( ) 



 



 +

=

= ∑

j ij j i

i

i

g u g W V I

V

gβ

( ) ]

[

e e

e u e

tanh _{u u}

u

u ∈ −

+

= _β^β − ⁻₋^β_β β

( ) ] [ ^{0 1}

1

1

2

,

u e

g

∈

= +

β

Funzione di attivazione

ß > 1 ß = 1 ß < 1

-1 +1

( )

( )

f = β

Updating Rules

Several possible choices for updating the units :

Asynchronous updating: one unit at a time is selected to have its output set

Synchronous updating: at each time step all units have their output set

Continuous updating: all units continuously and simultaneously change their outputs

Continuous Hopfield Models

Using the continuous updating rule, the network evolves according to the following set of (coupled) differential equations:

where are suitable time constants ( > 0).

Note When the system reaches a fixed point ( / = 0 ) we get

Indeed, we study a very similar dynamics

( ) 



 



 +

+

−

= +

−

= ∑

j ij j i

i i

i i

i

V g u V g w V I

dt dV

β

τ

τ

τ

dV

dt ∀ i

( )

i g u

V = _β

( )

j ij

i i

i

u w g u I

dt

du = − + ∑

+

τ

Modello di Hopfield continuo (energia)

Perché è monotona crescente e .

N.B.

cioè è un punto di equilibrio

( )

( )

2 1 2

1

2

1

 ≤



 



′ 

−

=

−

=



 



 − +

−

=

− +

−

−

=

∑

∑

∑

∑

∑

∑

∑

∑

dt u du

g

dt du dt dV

I u V

dt T dV

dt I dV dt

V dV dt g

V dV T dt V

T dV dt

dE

i i

i i

i i i

i

i i

j j

ij i

i

i

i i

i i

i j i

ij ij

j i

ij ij

β

β

τ τ

gβ

τ

> 0

0

0 ⇔ =

= dt

du dt

dE _i

u

The Energy Function

As the discrete model, the continuous Hopfield network has an “energy” function, provided that W = W^T :

Easy to prove that

with equality iff the net reaches a fixed point.

( ) ∑

∑ ∑ ⁺ ∑ ∫ ⁻

−

=

i i i

i j i

V j

i

ij

V V g V dV I V

w

E

2

1

≤ 0

dt

dE

Modello di Hopfield continuo (relazione con il modello discreto)

Esiste una relazione stretta tra il modello continuo e quello discreto.

Si noti che :

quindi :

Il 2^o termine in E diventa :

L’integrale è positivo (0 se V_i=0).

Per il termine diventa trascurabile, quindi la funzione E del modello continuo diventa identica a quello del modello discreto

( )

( ) ( )

i

g u g u g u

V =

=

β ≡ β

( )

i

g V

u = 1

β

( ) ^{V dV}

g

i V

i

∑ ∫

0

1

β

∞

→ β

Optimization Using Hopfield Network

§ Energy function of Hopfield network

§ The network will evolve into a (locally / globally) minimum energy state

§ Any quadratic cost function can be rewritten as the Hopfield network Energy function. Therefore, it can be minimized using Hopfield network.

§ Classical Traveling Salesperson Problem (TSP)

§ Many other applications

• 2-D, 3-D object recognition

• Image restoration

• Stereo matching

• Computing optical flow

i i

i j

i

i j

ij V V I V

w

E ^{= −}

∑ ∑

∑

2 1