Reality Check Tests
When testing for Superior Predictive Ability (SPA), the question of interest is whether any alternative forecast is better than the benchmark forecast or, equivalently, whether the best alternative forecasting model is better than the benchmark.
In this sense Hansen, P. R. (2005) proposed a Superior Predictive Ability test which is more powerful and less sensitive to the inclusion of poor and irrelevant alternatives as the White, H. (2000) Reality Check.
The Hansen's Test of Superior Predictive Ability can be addressed by testing the null hypothesis that "the benchmark is not inferior to any alternative forecast". This testing problem is relevant for applied econometrics, because several ideas and specifications are often used before a model is selected.
We consider a situation where a decision must be made periods in advance and let
be a finite set of possible decision rules. Decisions are evaluated with a real valued loss function,
where
random variable that represents the aspects of the decision problem that are unknown at the time that the decision is made. We evaluate fore casts in terms of their expected loss,
.
Our leading example is the comparison of forecasts, so we often refer to as the k-th forecasting model. The first model,
, has a special role and is referred to as the benchmark.
The decision rule can represent a point forecast, an interval forecast, a density forecasts, or a trading rule for an investor [Hansen, P. R. (2005)].
The most common use of the Hansen, P. R. (2005) Superior Predictive Ability Test is considerate as a binary variable binary variable that instructs a trader to take either a short (
) (i.e., short position, is a directional trading or investment strategy where the investor sells shares of borrowed stock in the open market.) or a long (
) position in an asset at time
(i.e. long position is the buying of a security such as a stock, commodity or currency with the expectation the asset will rise in value.). The k-th trading rule yields the profit
, where
is the return on the asset in period t.
A trader who is currently using the rule, , might be interested to know whether an alternative rule has a larger expected profit than
. This can be formulated in our framework by setting
and
.
In this case, the benchmark could be which is the rule that is always "long in the market." This was the bench mark used by Sullivan et al. (1999, 2001) who evaluated the significance of technical trading rules and calendar effects in stock returns.
We are interested to know whether any of the models, k=1,...,m, are better than the benchmark in terms of expected loss. So we seek a test of the null hypothesis that the benchmark is not inferior to any of the alternatives. The variables that are key for our analysis are the relative performance variables, which are defined by:
.
So denotes the performance of model k relative to the benchmark at time t, and we stack these variables into the vector of relative performances,
. Provided that
is well defined, we can now formulate the null hypothesis of interest as:
It is important to note that Hansen, P. R. (2005) works under the assumption that model k is better than the benchmark if and only if . So we focus exclusively on the properties of
for t=1,...,n and abstract entirely from all aspects that relate to the construction of the
-variables.
The Hansen, P. R. (2005) Superior Predictive Ability Test assumes that is (strictly) stationary and second finite moment (to be possible to apply the Central Limit Theorem).
In this case the the asymptotic null distribution is:
At this point, all essential aspects of our framework are identical to those of White, H. (2000) Reality Check (RC). White, H. (2000) proceeded by constructing the RC from the test statistic:
where is the sample mean associated with the k-th model and
is consistent estimator of the variance and covariance matrix.
Hansen, P. R. (2005) suggests:
- Use the studentized test statistic:
- Invoke a null distribution that is based on
where:
where 1 is the indicator function and is some consistent estimator.
The Hansen, P. R. (2005) Superior Predictive Ability Test is usually implemented using the stationary bootstrap of Politis and Romano (1994). The stationary bootstrap of Politis and Romano (1994) is based on pseudo-time series of the original data. The pseudo time series for b=1,...,B are resamples of
where
is constructed by combining blocks of {1,..., n} with random lengths.
The leading case is that where the block length is chosen to be geometrically distributed with parameter , but the block length may be randomized differently, as discussed by Politis and Romano (1994). The number of bootstrap resamples B, should be chosen to be sufficiently large such that the results are not affected by the actual draws of
.
Here we follow the conventional setup of the stationary bootstrap and generate B resamples from two random B x n matrices U and V, where the elements? , are independent and uniformly distributed on (0,1]. The first element of each resample is defined by
where
is the smallest integer that is larger than or equal to x. For t=2,...,n the elements are given recursively by:
-
if
-
if
.
So with probability q, the t-th element is chosen uniformly on {1, ..., n} and with probability 1-q, the t-th element is chosen to be the integer that follows unless
in which case
. The block bootstrap is very similar to the stationary bootstrap, but instead of using blocks with random length, the block bootstrap combines blocks of equal length.
From the pseudo-time series, we calculate their sample averages:
that can be viewed as (asymptotically) independent draws from the distribution of , under the bootstrap distribution. So this provides an intermediate step to estimate the distribution of Hansen's test statistic.
Hansen, P. R. (2005) suggested construct the centered matrix with elements equal to:
for b=1,...,B and t=1,...,n. The can be:
-
.
-
.
-
.
for i=l,c or u. Hansen's test statistics under the null hypothesis by the empirical distribution we obtain from the bootstrap resamples . The p-values of the three tests for SPA are now simple to obtain. We calculate:
for b=1,...,B and the bootstrap p-value is given by:
where the null hypothesis should be rejected for small p-values.
Here an example how to use the hansen.spa function.
#Clean the R's workspace
rm(list=ls())
#Invoke the libraries
library(mlRFinance)
#Generate some fake relative performance variables (NxK)
D<-matrix(runif(10000,-1,+1),ncol=10,nrow = 10)
#Generate the relative performance benchmark
b<-runif(1000,-1,+1)
#Call the function
hansen.spa(Dmat=D,bVec=b,typeFunc=1,B=500,geomMean=20)We can use the same idea to test for the White, H. (2000) Reality Check:
#Clean the R's workspace
rm(list=ls())
#Invoke the libraries
library(mlRFinance)
#Generate some fake relative performance variables (NxK)
D<-matrix(runif(10000,-1,+1),ncol=10,nrow = 10)
#Generate the relative performance benchmark
b<-runif(1000,-1,+1)
#Call the function
white.spa(Dmat=D,bVec=b,typeFunc=1,B=500,geomMean=20)Whereas the frameworks of Diebold and Mariano (2012) and West (1996) involve tests for EPA, the testing problem in White, H. (2000) framework is a test for SPA. The distinction is important because the former leads to a simple null hypothesis, whereas the latter leads to a composite hypothesis.
The basic idea is compare the expected loss based on the model forecast and some benchmark model. The input must be two vectors representing the estimated loss for these two models.
The DM Statistic can be used with a wide variety of accuracy measures (in particular, the loss function need not be quadratic and need not even be symmetric), and forecast errors can be non-Gaussian, nonzero mean, serially correlated, and contemporaneously correlated.
The DM Statistic is given by:
where
and
where are the sample variance and auto-covariance, respectively. Usually, we use
. Under the null hypothesis
.
Harvey, Leybourne, and Newbold (1997) suggested a modification of the Diebold and Mariano (2012) test that leads to better small-sample properties.
This statistic is given by:
.
Under the null hypothesis this statistic follow a T-student with T-1 degrees of freedom.
Hansen, P. R. (2005) suggests that the Diebold and Mariano (2012) can be used to test multiple hypothesis since the analyst use some correction as, for example, the Bonferroni correction.
To use the DM functions we can simple do:
#Clean the R's workspace
rm(list=ls())
#Invoke the libraries
library(mlRFinance)
#Generate some fake loss results
model <-runif(1000,-1,1)
bench <-ruinf(1000,-1,1)
#Call the function
DM.epa(e.model=model, e.bench=bench)
#Corrected DM
DM.epa.corrected(e.model=model, e.bench=bench)