Chapter 5.3, Problem 21E

When a tennis player serves, he gets two chances to serve in bounds. If he fails to do so twice, he loses the point. If he attempts to serve an ace, he serves in bounds with probability 3/8.If he serves a lob, he serves in bounds with probability 7/8. If he serves an ace in bounds, he wins the point with probability 2/3. With an in-bounds lob, he wins the point with probability 1/3. If the cost is '+1' for each point lost and '-1' for each point won, the problem is to determine the optimal serving strategy to minimize the (long-run)expected average cost per point. (Hint: Let state 0 denote point over,two serves to go on next point; and let state 1 denote one serve left. (1). Formulate this problem as a Markov decision process by identifying the states and decisions and then finding the Cik. (2). Draw the corresponding state action diagram. (3). List all possible (stationary deterministic) policies. (4). For each policy, find the transition matrix and write an expression for the…

During each time period, a potential customer arrives at a restaurant with probability 1/2. If there are already two people at the restaurant (including the one being served), the potential customer leaves the restaurant immediately and never returns. However, if there is one person or less, he enters the restaurant and becomes an actual customer. The manager has two types of service configurations available. At the beginning of each period, a decision must be made on which configuration to use. If she uses her "slow" configuration at a cost of $3 and any customers are present during the period, one customer will be served and leave with probability 3/5. If she uses her "fast" configuration at a cost of $9 and any customers are present during the period, one customer will be served and leave with probability 4/5. The probability of more than one customer arriving or more than one customer being served in a period is zero. A profit of $50 is earned when a customer is served. The manager…

Every Saturday night a man plays poker at his home with the same group of friends. If he provides refreshments for the group (at an expected cost of $14) on any given Saturday night, the group will begin the following Saturday night in a good mood with probability 7/8 and in a bad mood with probability 1/8. However, if he fail to provide refreshments, the group will begin the following Saturday night in a good mood with probability 1/8 and in a bad mood with probability 7/8 regardless of their mood this Saturday. Furthermore, if the group begins the night in a bad mood and then he fails to provide refreshments, the group will gang up on him so that he incurs expected poker losses of $75. Under other circumstances he averages no gain or loss on his poker play. The man wishes to find the policy regarding when to provide refreshments that will minimize his (long-run) expected average cost per week. (1). Formulate this problem as a Markov decision process by identifying the states and…