Chapter 8.1, Problem 1E

During each time period, a potential customer arrives at a restaurant with probability 1/2. If there are already two people at the restaurant (including the one being served), the potential customer leaves the restaurant immediately and never returns. However, if there is one person or less, he enters the restaurant and becomes an actual customer. The manager has two types of service configurations available. At the beginning of each period, a decision must be made on which configuration to use. If she uses her "slow" configuration at a cost of $3 and any customers are present during the period, one customer will be served and leave with probability 3/5. If she uses her "fast" configuration at a cost of $9 and any customers are present during the period, one customer will be served and leave with probability 4/5. The probability of more than one customer arriving or more than one customer being served in a period is zero. A profit of $50 is earned when a customer is served. The manager…

Every Saturday night a man plays poker at his home with the same group of friends. If he provides refreshments for the group (at an expected cost of $14) on any given Saturday night, the group will begin the following Saturday night in a good mood with probability 7/8 and in a bad mood with probability 1/8. However, if he fail to provide refreshments, the group will begin the following Saturday night in a good mood with probability 1/8 and in a bad mood with probability 7/8 regardless of their mood this Saturday. Furthermore, if the group begins the night in a bad mood and then he fails to provide refreshments, the group will gang up on him so that he incurs expected poker losses of $75. Under other circumstances he averages no gain or loss on his poker play. The man wishes to find the policy regarding when to provide refreshments that will minimize his (long-run) expected average cost per week. (1). Formulate this problem as a Markov decision process by identifying the states and…

This year Amanda decides to invest in two different no-load mutual funds: the G Fund or the L Mutual Fund. At the end of each year, she liquidates her holdings, takes her profits, and then reinvests. The yearly profits of the mutual funds depend on where the market stood at the end of the preceding year. Recently the market has been oscillating around level 2 from one year end to the next, according to the probabilities given in the following transition matrix : L1 L2 L3 L1 0.2 0.4 0.4 L2 0.1 0.4 0.5 L3 0.3 0.3 0.4 Each year that the market moves up (down) 1 level, the G Fund has profits (losses) of $20k, while the L Fund has profits (losses) of $10k. If the market moves up (down) 2 level in a year, the G Fund has profits (losses) of $50k, while the L Fund has profits (losses) of only $20k. If the market does not change, there is no profit or loss for either fund. Amanda wishes to determine her optimal investment policy in order to maximize her (long-run) expected average profit per…