Pipeline
The project runs in four stages, from raw data to a backtest.
STEP 1DataPrices, macro
STEP 2Feature engineeringLags, MA, RSI
STEP 3ModelingRidge / RF / XGBoost
STEP 4BacktestWalk-forward, Sharpe
Feature engineering
Every feature is lagged by one day, so the model only sees what was known at the previous close.
src/features.pypython
import pandas as pd
def build_features(df: pd.DataFrame, macro: pd.DataFrame) -> pd.DataFrame:
out = pd.DataFrame(index=df.index)
ret = df["close"].pct_change()
ma20 = df["close"].rolling(20).mean()
# shift(1): only use what was known at yesterday's close
out["ret_lag1"] = ret.shift(1)
out["ma20_gap"] = (df["close"] / ma20 - 1).shift(1)
out["rsi14"] = rsi(df["close"], window=14).shift(1)
# macro series: align to trading days, then forward-fill
out = out.join(macro[["vix", "pmi"]].shift(1), how="left").ffill()
return out.dropna()Model training
Models are compared with time-ordered cross-validation rather than random K-fold.
src/train.pypython
import xgboost as xgb
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
model = xgb.XGBRegressor(
n_estimators=500,
max_depth=4,
learning_rate=0.05,
subsample=0.8,
colsample_bytree=0.8,
)
for train_idx, test_idx in tscv.split(X):
model.fit(X.iloc[train_idx], y.iloc[train_idx])
preds = model.predict(X.iloc[test_idx])Feature importance
Short-term momentum and RSI carry most of the signal; macro features add a little on top.
Reproduce
Terminal
git clone https://github.com/[username]/sp500-return-prediction
cd sp500-return-prediction
pip install -r requirements.txt
python src/train.py --model xgboostFull notebook and source are on GitHub ↗.