Understanding the trades/positions ... for my first backtest that left me without equity

[bravegag]

Hi all,

I ran my first backtest to end up without money haha! a classic! My prediction algo is not too bad with about 71% precision classification-wise, so I'm wondering whether I'm using the backtesting library incorrectly ... it looks to me like an alignment issue, I use the indicators at the current hour to predict price movements on the next hour.

This is an example snapshot of my backtest:

I see the true and predicted targets to match (first true and pred spike below which indicate a bull run of more than half a percent) but the actual trade result shows like two steps ahead. I am not understanding that and I am sure this has been discussed extensively but can't find the relevant discussions so here I go again.

My algo is set to predict 1h XRP price movements of at least 0.5% so half percent movement which is about the 80% and 20%, upper and lower price returns quantiles respectively. I then do this super simple Trading next implementation using my predictions. I maybe should , in addition to the classification performance, also manually compute the hit rate to compare with the backtest results.

def next(self):
    """"""
    # Prepare OCHL data
    high, low, close = self.data.High, self.data.Low, self.data.Close
    start_time = self.data.index[0]
    current_time = self.data.index[-1]

    # TODO: implement recurrent re-training of the models

    # run prediction and compute the forecast
    if current_time in self._y_pred.index:
        long_pred = int(self._y_pred.loc[current_time, 'bull'])
        long_pred = 1 if long_pred == 1 else 0
        short_pred = int(self._y_pred.loc[current_time, 'bear'])
        short_pred = 1 if short_pred == 1 else 0
    else:
        long_pred = 0
        short_pred = 0

    # 1 when long_pred is 1 and short_pred is 0, then
    # -1 when short_pred is 1 and long_pred is 0, zero otherwise
    forecast = long_pred - short_pred
    # update the forecast for the extra plot
    self.forecasts[-1] = forecast

    self.position.close()
    if forecast == 1:
        self.buy()
    elif forecast == -1:
        self.sell()

Oh I can't attach the full backtest, html is not supported ... bummer!

UPDATE

here are my bull and bear confusion matrices (not too bad)

the algo is able to predict 24 bull runs perfectly, and 103 false-positives (out of which some may be still positive some not)
[[944 103]
[382 24]]

the other algo is able to predict 50 bear runs perfectly, and 139 false-positives (out of which some may be still negative some not)
[[899 139]
[365 50]]

False-negatives aren't bad because I wouldn't enter a position (or missed opportunity) but also don't lose any money. However, I need to keep an eye on the false-positives as they mean I enter a trade/position and could be incorrect, for instance, a negative return while expecting a bull run.

UPDATE

It is definitely a misalignment, if I make my predictions / forecasts the true values I still lose all the money ... so need to find a way to align the data somehow.

[kernc]

Please try with v0.6.3 (just released) as it features multiple bug fixes and improvements!

[bravegag]

Hi @kernc thanks a lot for letting me know. i will check this WE and revert, LF seeing my prediction profitable :)

[bravegag]

At the end it worked, the issue was aligning date-time / index-wise the OCHL Strategty data to the predictions data. To be able to do that, I used as benchmark my labels (or a perfect market prediction) until the perfect market prediction was all green. From that point on I was able to align the ML predictions correctly to the backtesting's Strategy, then it worked.