Markov Decision Processes: Monotonicity of Optimal Policy in Exponential and Quasi-Hyperbolic Discounting Parameters

Abstract

Intertemporal preferences of decision makers, i.e., the way they discount delayed utilities, impact their decisions. Empirical evidence suggests that individuals commonly have hyperbolic discounting preferences. This can result in time-inconsistent behavior, e.g., procrastination, which may be a barrier to adopting preventive behavior such as machine maintenance and patient adherence to treatment. In this paper, we theoretically compare the actions of individuals based on their discounting characteristics. We consider the Hyperbolic Discounting (HD) model, which is more representative of individual behavior than Exponential Discounting (ED). We formulate a discrete-time finite-horizon Markov decision process with Quasi-Hyperbolic Discounting (QHD), an analytically tractable function representing HD and present sufficient conditions that ensure the monotonicity of the optimal policy in the discounting parameters. We consider submodular maximization or supermodular maximization problems. Our paper is the first to investigate the monotonicity of the optimal policy in QHD parameters for these problems. Moreover, we compare the optimal actions under ED and QHD. We apply our results to the settings of machine maintenance, individual health behavior and inventory control. We provide numerical examples that show there might not be monotonicity if our sufficient conditions are not met. Also, we explore the discrepancy between the expected total exponentially-discounted rewards of the actions obtained from QHD and of the actions that are optimal under ED, and observe that this discrepancy is affected mainly by the present bias.