3. Consider a Markov chain (Xn)n=0,1,2,... with state space S probability matrix = {1, 2, 3} and transition P = ( 1/5 3/5 1/5\ 0 1/2 1/2 3/10 7/10 0 The initial distribution is given by a = (1/2, 1/6, 1/3). Compute (a) P[X2 = k] for all k = 1, 2, 3; (b) E[X2]. Does the distribution of X2 computed in (a) depend on the initial distribution a? Does the expected value of X2 computed in (b) depend on the initial distribution a? Give a reason for both of your answers.

2. Let X₁, X₁,... be the Markov chain on state space {1,2,3,4} with transition matrix 1/2 1/2 0 0 1/7 0 3/7 3/7 1/3 1/3 1/3 2/3 1/6 1/6, 0 0 (a) Explain how you can tell this Markov chain has a limiting distribution and how you could compute it. Your answer should refer to the relevant Theorems in the notes. (b) Find the limiting distribution for this Markov chain. (c) Without doing any more calculations, what can you say about p1100) and p(100)?

2. Let X₁, X₁,... be the Markov chain on state space {1,2,3,4} with transition matrix 0 1/2 1/2 0 1/7 0 3/7 3/7 1/3 1/3 1/3 0 0 2/3 1/6 1/6, (a) Explain how you can tell this Markov chain has a limiting distribution and how you could compute it. Your answer should refer to the relevant Theorems

2. Let Xo, X₁,... be the Markov chain on state space {1,2,3,4} with transition matrix 1/2 1/2 0 0 1/7] 0 3/7 3/7 1/3 1/3 1/3 0 2/3 1/6 1/6, 0 (b) Find the limiting distribution for this Markov chain. (100) (100) ? Without doing any more calculations, what can you say about P₁,1 and P2,1

The random variables W1, W2,... are independent with common distribution k 1 2 3 4 Pr( W = k) 0.1 0.3 0.2 0.4 Let Xn max (W1,..., Wn) be the largest W observed to date. Determine the transition probability matrix for the Markov chain {Xn}.

Suppose that {Xn} is a Markov chain with state space S = {1, 2}, transition matrix （1/5 4/5 2/5 3/5）, and initial distribution P (X0 = 1) = 3/4 and P (X0 = 2) = 1/4. Compute the following:  P(X2 =2)

9. Let i be a transient state of a continuous-time Markov chain X with X (0) : = i. Show that the total time spent in state i has an exponential distribution.

Let X, be the Markov chain with state space Z and transition probability Pa,z+1 = P, Pa,2-1 = 1- p, where p > 1/2. Assume X, = 0. (a) Let Y = min{Xo, X1,...}. What is the distribution of Y? (b) For positive integer k, let T = min{n : X, = k} and let e(k) = E[TR]. Explain why e(k) = ke(1). (c) Find e(1). Hint: part (b) might be helpful. (d) Use (c) to give another proof that e(1) = o if p = 1/2.

4. Consider an aperiodic, irreducible Markov chain Xn (n = 0, 1, ...) with the N + 1 states 0, 1,..., N having stationary probabilities π¿ (i = 0, 1, ..., N). Suppose that the initial distribution of the Markov Chain is chosen according to the stationary probabilities. That is, P(X = i) = πi, i = 0, 1, ..., N. Define : Qij = P(X0 = j\X₁ = i), Q = [Iij](N+1)×(N+1), 0 ≤ i, j ≤ N. (2) The square of the matrix Q is written as Q² = [93] (N+1)x(N+1). Prove that q¡ = P(X0 = j|X2 = i). dj

A Markov chain X₁, X₁, X₂ ... on the states 0, 1, 2 has the transition probability matrix 1 2 0.2 0.7 P 10.2 0.2 0.6 2 0.6 0.1 0.3 and initial distribution Po = P(X₁ = 0) = 0.2, P₁ = P(Xo = 1) = 0.3, and P2 = P(Xo = 2) = 0.5. (1) Compute the two-step transition matrix; (2) What is P(X3 = 1|X₁ = 0)? (3) What is P(X3 = 0, X5 = 2|X₂ = 1)? (4) What is P(Xo = 2, X₂ = 0, X3 = 1)? 0 0 0.1