BAYESIAN ANALYSIS OF CHANGE POINT PROBLEM IN AUTOREGRESSIVE MODEL: A MIXTURE MODEL APPROACH

BAYESIAN ANALYSIS OF CHANGE POINT PROBLEM IN

AUTOREGRESSIVE MODEL: A MIXTURE MODEL APPROACH

P. Arumugam, M. Vijayakumar

Annamalai University, Annamalai Nagar, India

kp_aru24@rediffmail.com, Vijayakumarm1975@yahoo.com

Dhanagopalan Venkatesan

Annamalai University, Tamil Nadu, India

sit_san@hotmail.com

Abstract

In this paper, we discuss the problem of gradual changes in the parameters of an autoregressive

(AR) time series model of p

th

order, through Bayesian mixture approach. This model incorporates

the beginning and end points of the interval of switch. Further, the marginal posterior densities of

the parameters are obtained by employing the ordinary numerical integration technique.

Keywords:

Autoregressive model; Bayesian estimation; Structural change; Mixture model,

Numerical integration.

1. INTRODUCTION

In recent years there has been many evidence for the parameter of economic models undergone the structural changes. When a parametric model changes parameter value it is important to know the time when the change occurred and pre and post change value of the parameters.

The literature on structural change problems is by now enormous. Here we consider some of the work related with linear and time series models. Bacon and watts (1971), Ferreira (1975), Holbert and Broemeling (1977), Chin Choy and Broemeling (1980), Smith and Cook (1990) and Moen et al (1985) look at change points in linear models. West and Harrison (1986), Salazar (1982), Broemeling (1985) and Venkatesan and Arumugam (2007) have studied the structural change problems in time series model through the parameter change, while Baufays and Rasson (1985) have studied a variance change in autoregressive model. Most of the work in the literature is based on the parameter change in the time series model. In this paper, a Bayesian analysis of structural changes in autoregressive model of higher order is studied through the mixture model approach by introducing the distribution function of the beta random variable to model the nature of change in a finite interval of time.

In this paper the structural changes are incorporated in a model by the mixture approach. Consider, for example the case of permanent change in a firite internal (t1, t2). It is now assumed that one model operates

before time t1, another model operates after time t2 and in the interval the second model gradually replaces

the first model. That is, at time t (t1<t<t2) the first model operates with probability (1-Pt) and the second

model operates with probability Pt and Pt goes from zero to one as t goes from t1 to t2. Thus, in this

formulation, the likelihood function of the data will be based upon mixture distributions. One special advantage of this approach in the construction of switching models is that the number of parameter describing the nature of switch will always be fixed.

An out line of this paper is as follows. The pth _{order autoregressive model and likelihood function are}

described in Section 2. Section 3 describes the posterior analysis of the model under the mixture model approach.

2. THE MODEL AND LIKELIHOOD FUNCTION

The autoregressive model of order p (AR(p)) time series model {Xt} is defined by

Xt = α1Xt-1 + α2 Xt-2 + …+ αp Xt-p + et … (1)

and suppose that there is a shift in (α1, α2, …, αp) starts at some time point t1 and ends at some time point t2,

the model is given by

MTISD 2008 – Methods, Models and Information Technologies for Decision Support Systems 

Università del Salento, Lecce, 18­20 September 2008   

   

2

Xt = (1 – Pt)

∑

∑

= − = −

+

β

+

α

p 1 i i t i t p 1 i i t i t

e

X

P

X

… (2)

where (α1, α2,…, αp) and (β1, β2,…, βp) are real unknown autoregressive parameters of before and after change

respectively, et’s are iid normal with zero mean and common variance σ2 and denote θ′ = (α1, α2, …, αp, ; β1, β2,

…, βp) (say)

Let X1, X2………Xn be a sequence of n observations. Then the conditional density of Xt / Xt-1 has the

following probability density function. f(Xt / Xt-1) =

⎪

⎩

⎪

⎨

⎧

≥

<

<

+

−

≤

2 t 2 2 1 t 2 t t 1 t 1 t 1

t

t

:

f

t

t

t

:

f

P

f

)

P

1

(

t

t

:

f

… (3)

where f1t and f2t are the probability density function of a Normal random variable with means n 1

Σ

αiXt-i and n

1

Σ

βiXt-i respectively and common variance

σ

2.

Pt =

⎪

⎩

⎪

⎨

⎧

≥

<

<

′

≤

2 2 1 1

t

t

:

1

t

t

t

:

)

t

(

F

t

t

:

0

F(t′) =

∫

′ − β − α

₋

β

α

t 0 1 1

₍

₁

_u

₎

_du

u

)

,

(

B

1

t′ = 2 1 1

t

t

t

t

−

−

; 1 < t1 < t2 < n

B (

α

,

β

) denotes the complete beta function with arguments

α

and

,

β

; t1 and t2 are positive integers with

1<t1<t2<n then,

α

and

β

determine the nature of increase Pt from 0 to 1 as t goes from t1 to t2. Here, (t1,t2)

in the switching interval the problem is to estimate U = (t1,t2,

α

,

β

,

θ

,'

σ

2 ) but attention is mainly focused

on the estimation of t1,t2,

α

and

β

using the observation X=(X1,X2………Xn).

The prior distributions for the parameters are assigned as follows i.

σ

2 is non-informative

ii. Given

σ

2,

θ

' follows the multivariate normal distribution with mean zero and precision τi/σ2; i = 1, 2.

iii.

α

and

β

follows the exponential distribution with parameters ‘a’ and ‘b’ respectively iv. (t1, t2) is uniformly distributed over all possible values.

v. The parameters (

θ

',

σ

2_),

α

_,

β

_{and (t}

1,t2) are apriori independent.

Therefore the joint prior distribution is P(U) ∝

ab

_e

−(αa+βb)

σ

; σ, a, b, α, β > 0 ... (4) Let X1, X2, ..., Xn are observations and X0, X-1, …, X1-p are assumed to be known values.

The likelihood function of the observations X = (X1, X2, ..., Xn) given

the parameter U = (t1, t2, θ′, α, β, σ2) is given by

3

•

exp

_⎢

⎣

⎡

⎩

⎨

⎧

−

+

−

σ

−

∑

∑

∈ * r 1 C t 2 1 t t 1 2 1 t 2

(

X

B

)

(

X

B

)

2

1

+

_⎥

⎦

⎤

⎭

⎬

⎫

−

+

−

∑

∑

+ ∈ n 1 t 2 2 t C t 2 2 t 2 r

)

B

X

(

)

B

X

(

…(5)

which, on simplification, becomes P(X/U) ∝

∑

= −

⎟

⎠

⎞

⎜

⎝

⎛

σ

−

σ

P 0 r 2 n r

2

Q

exp

A

... (6) Where Ar = n C t t C t t r r r

P

P

− ∈ ∈

⎟

⎟

⎠

⎞

⎜

⎜

⎝

⎛

⎟

⎟

⎠

⎞

⎜

⎜

⎝

⎛

−

∑

∏

∏

σ

*

)

1

(

Q = C(X) +

[

θ′

₁

A

₁

(

P

,

X

)

θ

₁

−

2

θ′

₁

B

₁

(

P

,

X

)

]

+

[

θ′

₁

A

₂

(

P

,

X

)

θ

₁

−

2

θ′

₁

B

₂

(

P

,

X

)

]

+

[

θ

′

₂

A

₃

(P,

X)θ

₂

−

2

θ

′

₂

B

₃

(P,

X)

]

+

[

θ′

₂

A

₄

(

P

,

X

)

θ

₂

−

2

θ′

₂

B

₄

(

P

,

X

)

]

where C(X) =

∑

n 1 2 t

X

B1(P, X) is p × 1 vector with ith element

∑

= − 1 t 1 i t t i

X

X

; B2(P, X) is p × 1 vector with ith element

∑

∈ * − r C t i t t

X

X

;

B3(P, X) is p × 1 vector with ith element

∑

∈Cr −

t

i t

t

X

X

;B4(P, X) is p × 1 vector with element

∑

+ − n 1 t2 t t i

X

X

;

θ′

₁ = (α1, α2, … αp) ;

θ′

₂ = (β1, β2, …, βp)

A1(P, X) is p × p matrix with ith diagonal element is

∑

= − 1 t 1 t 2 i t

X

and ijth _{off-diagonal element is}

j t t 1 t t i

X

X

1 − = −

∑

.;

A2(P, X) is p × p matrix with ith diagonal element is

∑

∈ * − r C t 2 i t

X

and ijth _{off-diagonal element is} j t C t t i

X

X

* r − ∈ −

∑

;A3(P, X) is p × p matrix with ith diagonal element is

∑

∈Cr −

t 2

i t

X

and ijth _{off-diagonal element is} j t C t i t

X

X

r − ∈ −

∑

;A4(P, X) is p × p matrix with ith diagonal element is

∑

+ = − n t t 2 i t 1 2

X

and ijth _{off-diagonal element is} j t n 1 t t t i

X

X

2 − + =

∑

− ; P=t2-t1

C= {t1+1, t1+2……..,t2}; Cr is any subset of C with ‘r’ elements;Cr* is the compliment of Cr

Σ

r stands for the summation taken over all the (pCr) combination of (t1+1,….t2) selecting ‘r’ at a time of the

second term and remaining (p-r) of the first term.

3. THE POSTERIOR ANALYSIS

4

M1(P, X) = A3 (P, X) + A4 (P, X)

D(P, X) = B1 (P, X) + B2 (P, X)

D1(P, X) = B3 (P, X) + B4 (P, X)

After simplification, one can get, P(U/X) ∝

∑

= + − + − ⎟⎟ ⎠ ⎞ ⎜⎜ ⎝ ⎛ − P r b a n r Q e A 0 2 *' ) ( ) 1 ( 2 exp

σ

σ

α β _{… (8)} where Q* ' _{= [θ} 1 – M-1 (P, X) D(P, X)]′ M(P, X) [θ1 – M-1 (P, X) D(P, X)] + [θ2 –

M

₁−1(P, X) D1(P, X)]′ M1(P, X) . [θ2 –

M

₁−1(P, X) D1(P, X)] + C*(X) and C*_{(X) = [C(X) – D′(P, X)M}-1 _{(P,X) D(P, X) –} 1

D′

(P, X)

M

₁−1(P, X) D1(P, X)]

Eliminating θ1 and θ2 and σ2 from the above expression (8), one gets

P(t1, t2, α, β / X) ∝

∑

= β + α − P 0 r ) b a ( r

(

e

)

A

. 2 ) 3 p 2 n ( * 2 / 1 1 2 / 1

_|

_M

₍

_P

_,

_X

₎

_|

_[

_C

₍

_X

_)]

|

)

X

,

P

(

M

|

+ − − − − … (9)

The elimination of the parameters from (9) is analytically not possible since the joint posterior distribution of t1, t2, α and β is a complicated function of t1, t2, α and β. Therefore, one may have to resort to

numerical integration technique to determine the marginal posterior distribution of the parameter.

The first order autoregressive model is obtained by taking p=1 in (1) and in the joint posterior density (9) is reduced to the following, after simplification

P(t1, t2, α, β / X) ∝

∑

= P r 0 Ar

e

−(αa+βb)

|

V

|

−1/2

;

α, β > 0, 1 ≤ t1 < t2 ≤ n – 1 ... (10) where V =

⎥

⎥

⎥

⎦

⎤

⎢

⎢

⎢

⎣

⎡

+

+

C

)

2

n

(

A

0

0

C

)

2

n

(

A

3 2 and C = A1 – 3 2 2 2 2 1

A

B

A

B −

. REFERENCES

Bacon, D.W. and D.G. Watts (1971). Estimating the Transition between two intersecting lines. Biometrika, 58, 524-534.

Baufays and Rasson (1985). Variance changes in Autoregressive model, Time series Analysis: Theory and Practice, O.D. Anderson (Editor), North – Holland, Amsterdam.

Broemeling, L.D. (1985). Bayesian Analysis of Linear Models. Marcel Dekker Inc., New York and Basel. Chin Choy, J.H., and L.D. Broemeling (1980). Some Bayesian Inference for a Changing Linear Model:

Technometrics, 22(1), 71-78.

Ferreira, R.E. (1975). “A Bayesian Analsyis of a switching Regression Model: Known Number of Regimes,

Journal of the American Statistical Association, 70, 370-374.

Holbert, D. and L.D. Broemeling (1977). Bayesian Inference Related to Shifting Sequences and Two-Phase Regression. Communications in Statistics, 6, 265-275.

Moen, D.H., Salazar, D. and L.D. Broemeling (1985). Structural changes in multivariate regression models.

Communs. Statist. A, 14, 1757-1768.

Salazar, D., (1982). Structural Changes in Time Series Models, Journal of Econometrics, 19, 147-163. Smith and Cook (1990). Straight lines with a change point: a Bayesian analysis of some renal transplant data,

Appl. Statist., 29, 180-189.

Venkatesan, D and P. Arumugam (2007). Bayesian Analysis of Structural Changes in Autoregressive Models, American Journal of Mathematical and Management Sciences, 27, Nos. 1&2, 153-162.