Are you sure you are using the correct model? Model Selection and Averaging of Impulse Responses
Impulse responses can be estimated to analyze the effects of a shock to a variable over time. Typically, (vector) autoregressive models are estimated and the impulse responses implied by the coefficients calculated. In general, however, there is no knowledge of the correct autoregressive order. In fact, when models are seen as approximations to the data generating process (DGP), all models are imperfect and there is no a priori difference in their validity. Hence, a lag length should be chosen
... h should be chosen by a sensible method, for instance an information criterion. In Monte Carlo simulations, this paper studies what characteristics influence the optimal autoregressive order when all models are only approximations to the DGP. It finds that the precise coefficients in the DGP, the sample size, and the impulse response horizon to be estimated all influence the mean squared error-minimizing lag length. Furthermore, it evaluates the performance of model selection and averaging methods for estimating impulse responses. Across the characteristics found to be relevant, averaging outperforms model selection, and in particular Mallows' Model Averaging and a smoothed Hannan-Quinn Information Criterion perform best. Finally, the study is extended to vector autoregressive models. In addition to the characteristics relevant in the univariate case, the optimal lag length also depends on which (cross) impulse response is to be estimated. Many issues remain for vector autoregressive models, however, and more work is necessary. 1 Are You Sure You Are Using the Correct Model? their policies on the economy. For this purpose, vector autoregressive models are oftentimes estimated to analyze the dynamics behind the underlying economic variables. To understand the effect that a shock to one variable has over time on the system of variables studied, impulse response functions can be estimated. Unfortunately, however, many issues remain in this estimation. This paper focuses on what order of an autoregressive model should be chosen to minimize the mean squared error of the resulting estimator for the impulse response. Furthermore, it compares the performance of various model selection and averaging techniques for estimating impulse responses in a Monte Carlo study. Lastly, it offers preliminary insights into the study of these issues for vector autoregressive models. In a brief paper, Hansen (2005) discusses challenges to model selection, focusing as an example on the model also used in this paper. He criticizes the common assumption that the true data generating process is among the candidate models for model selection, and advocates the use of selection methods specific to the purpose of the selected model. The Focused Information Criterion (FIC), developed by Claeskens and Hjort (2003) , is such a method, asymptotically selecting the model that minimizes the mean squared error of the estimator for the parameter of interest. Claeskens et al. (2007) justify and demonstrate its use in the setting of this paper. When no candidate model is the true DGP, however, selecting one such model that is known to be incorrect might be prone to a form of overfitting. As an alternative, estimates of different models can be pooled and averaged based on a prespecified rule, for instance using smoothed information criteria. Claeskens and Hjort (2008) provide a detailed theoretical treatment of information criteria (IC), in particular the FIC, and smoothed IC. In a simulation study of forecasting quality, Hansen (2008) evaluates further averaging techniques. The remainder of this paper is structured as follows. Section 2 presents some theoretical background on the time series models used and the calculation of the impulse response functions. Sections 3 and 4, respectively, offer an overview of the model selection and the averaging criteria employed. In section 5, the results of the Monte Carlo study are presented and their implications discussed. Section 6 concludes. The Model This section presents the models considered in this paper. While the true DGP in all simulations is a (vector) autoregressive moving-average model (ARMA), all candidate models are finite order (vector) autoregressive (AR) models. Section 2 provides theory for ARMA models, while section 2 extends this to the VAR setting. Section 2 explains the use of the impulse response function and gives computational details for (V)ARMA and (V)AR models.