Estimation Examples
##Example 1: simple model comparisons## In the section Estimation we have discussed an example where the estimation results of a model could depend on whether we use the EM algorithm or numerical optimization. We concluded, based on the vintage of data available at the middle of November of 1999, that the second alternative provides the model with a higher likelihood. Here, we will compare the resulting output of both approaches, which consider exactly the same model specification. Before tackling the subject of real-time simulations , we can already have an overview of the two models by answering questions such as the ones proposed below.
###Which method provides us with a better in-sample fit for GDP growth?### All the results available in the output tab can be compared across models using the graphical tools available in JDemetra+ (e.g. Tools>Container>Chart). Note that the data underlying all the graphs can also be copied and pasted to Excel, so users have some flexibility to analyse the results and perform exercises such as the one described here. We first open the two model tabs, and then we compare actual GDP growth with the signal extracted using the the two alternative estimation methods:
As mentioned earlier, Actual GDP growth is plotted at the middle of the quarter, so we have the wrong impression that it leads the signal, which is plotted for every month, and represents a weighted average of the factors. Still, it is very clear from the picture that the numerical procedure results on a signal that accounts for high frequency fluctuations of GDP growth, while the signal obtained with numerical optimization is smoother. The standard deviations of the difference between the actual data and the signal (i.e. the residual) are equal to 0.23 and 0.12, respectively. Those residuals can be further analysed by clicking on the branch "residuals" inside FIT.
It is left as an exercise to look at "Signal vs Data" in FIT to verify that in this example the model based on the EM algorithm obtains factors that account for a large proportion of the variance of GDP growth and survey data, while the model based on numerical optimization tuns out to yields a better fit for oil prices at the cost of accounting for a smaller fraction of GDP growth, as highlighted above.
As more and more data enters the model, both estimation procedures turn out to yield exactly the same parameter estimates, so both models turn out to be equivalent. However, that information was not revealed to us in 1999 with the hypothetical information set that we are using to estimate this model. In general, one should be aware of the following issues:
- We know that a good in-sample fit for GDP growth does not guarantee that the model will work out-of-sample.
- Two alternative parameter estimates may be translated into alternative correlation patterns in the data, which implies two different ways to read the economy.
##Example 2: Conditional forecasts ##
The two alternative parameter estimates corresponding to the same model (see example above) can imply alternative ways to read the economy. The exercise described below serves as an introduction to the two topics that will be discussed later on: Reading News and Real-Time Simulations
This problem of conditional forecasting is from a computational point of view equivalent to the problem of nowcasting, which will be treated in real-time simulations , but at this stage there are only two concepts that need to be defined:
- Unconditional forecast: E[GDP(t+h) | Info available in October 1999 ]
- Conditional forecast: E[GDP(t+h) | Info available in October 1999 + Surveys and Financial data until T ].
This idea can be understood as follows. Suppose that GDP and all hard data stops being published from 1999 onwards and we have to obtain a simple estimate of growth for the euro area on the basis of Surveys and financial data. Can we count on both models to extract the growth signal during the Great recession? In practice, we need to calculate the expected growth rate conditional on the surveys and financial variables available for the whole sample, and compare those results with the actual growth rates of the economy.
###How to calculate the conditional forecasts in this example?###
- Update in your excel file all the series you want to incorporate in your conditioning information set, e.g. realizations of surveys and financial data in our example, but it could also be done with assumptions regarding the evolution of those variables.
- Refresh your data (remember how to do it)
- Click on the green arrow of the processing tab (
estimationicon
) to re-run the model with the refreshed data. This action triggers a run of the kalman smoother, since the estimation options are, by default, unchecked after refreshing.
###Results###
It turns out that the model with the highest likelihood is also the one that whose conditional expectations for growth are most comparable to the reality.
####Understanding the results####
Can we decompose the difference between the unconditional forecast and the one conditional on the realization of surveys and financial variables. Such decomposition is needed to understand what are the indicators helping us to have a good estimation of the GDP growth rates.
Because of the dependence of all indicators on five common factors, understanding what are the most relevant variables in the conditioning information set is not straightforward. Luckily, such decomposition is given by the news analysis that will be introduced later on.
####Which model yields the best forecasts?####
We have compared the performance of two models at forecasting GDP growth over the Great Recession, and concluded by looking at a simple graph that one of the models is clearly superior. However, a more systematic evaluation of the forecasts would be required (real-time simulations ) if we aim to answer questions such as:
- Which model produces better GDP forecasts two months before the official release?
- How does forecasting uncertainty decrease when more and more information enters the information set?
- etc
[to be completed] ###The concept ### ###Example based on US data ###
[to be completed] ###Incorporating the calendar of release dates### ###Evaluation of point forecasts### ###Nowcasting uncertainty###