Forecasting layer¶

This page covers the rolling-ridge and rolling-lasso predictors (v1 + the v2 kwarg-form extensions from EPIC-50a and EPIC-50b), and the gradient-boosted gbdt method.

The three are not the same shape. Ridge and lasso learn online: every bar pushes a training pair and refits, so the score at bar t reflects data through t-1. gbdt is trained offline in Python and fixed at load — it never learns during a run. That difference is why lookback is rejected for gbdt rather than merely unused: it is the rolling fit window, a boosted model has no fit, and accepting a number the run discards is what makes a config look tuned when it is not.

Rolling-fit linear predictors (RollingRidge, RollingLasso) exposed to .qe configs as predict_return(...). The motivation is to start data-driven strategy research from a known baseline — ridge / lasso regression — that any later ML approach has to beat to justify its complexity.

The function has two parse forms that produce the same machinery:

v1 positional — predict_return(lookback, alpha, f1, ...). Backward-compatible with every fixture in the repo. Forecasts next-bar close-to-close return, ridge-solved.
v2 kwarg (EPIC-50a + EPIC-50b) — predict_return(lookback = ..., features = [...], alpha = ... | lambda = ..., horizon = H, target = E, method = "ridge" | "lasso" | "gbdt", model = "<path>"). Unlocks these extensions:
- horizon = H — forecast H bars ahead instead of one.
- target = E — replace the hardcoded close-to-close return with any signal-context expression (e.g. volatility, multi-bar return, sign of return).
- method = "lasso" + lambda = L — switch the per-call-site solver to a coordinate-descent L1-regularized fit. Same span- taking API, sparse coefficients, sklearn-matching to 1e-6.
- method = "ridge" (the default) keeps existing fixtures and walk-forward results bit-for-bit identical.

The two forms cannot be mixed in a single call — pick one. v1 calls stay bit-for-bit identical; the binder asserts horizon = 1 kwarg ≡ v1 positional on the same features.

Quick start¶

backtest(
  data = yahoo("SPY", "1d", "2018-01-01", "2024-12-31"),
  strategy = signal(
    entry = predict_return(
              252, 1.0,
              lag_return(close, 1),
              lag_return(close, 5),
              rolling_zscore(close, 20),
              rsi(close, 14) / 100.0
            ) > 0.001,
    exit  = predict_return(
              252, 1.0,
              lag_return(close, 1),
              lag_return(close, 5),
              rolling_zscore(close, 20),
              rsi(close, 14) / 100.0
            ) < 0,
    symbol = "SPY",
  ),
  execution = execution(capital = 100_000),
  output    = output(results = "out/forecast.json"),
)

What this does, per bar:

Compute the four feature values from the bar's close.
If a prior bar's features are stashed, push (features_{t-1}, return_t) into the ridge — keeping the training pair causal — and refit.
Predict return_{t+1} from this bar's features.
Compare the prediction to the entry / exit thresholds, which becomes the strategy's signal for the next bar's fill.

The first lookback + feature_warmup bars produce NaN forecasts; the strategy stays flat through warm-up.

API¶

`predict_return(lookback, alpha, f1, f2, ..., fN)` — v1 positional¶

Arg	Type	Constraints
`lookback`	int literal	`> 0`, `>= n_features + 1`
`alpha`	double literal	`>= 0` (use `0` for plain OLS)
`f1..fN`	signal expr	at least one; at most 64

Returns a double — the next-bar forecast. Warm-up returns NaN; any feature being NaN this bar also returns NaN and breaks the causal training chain for one bar (better than corrupting the ring with garbage). Each call site allocates an independent RollingRidge instance; an entry and exit that both call predict_return(...) do not share state in v1 — they train in parallel on the same data and converge to similar coefficients.

`predict_return(lookback = ..., features = [...], method = ..., ...)` — kwarg form (EPIC-50a + EPIC-50b)¶

The kwarg form unlocks four extensions and accepts every original v1 feature unchanged:

Kwarg	Type	Default	Constraint
`lookback`	int literal	required for ridge / lasso; rejected for gbdt	`>= n_features + 1` (linear solvers only — gbdt has no fit window, so the under-determination guard is skipped)
`features`	array literal	— required	`[expr, expr, ...]`, 1 to 64 entries
`method`	string literal	`"ridge"`	`"ridge"`, `"lasso"` or `"gbdt"`
`model`	string literal	—	required iff `method = "gbdt"`, rejected otherwise; path to the offline-trained ensemble, loaded and validated at bind time, and its `n_features` must equal `features.size()`
`alpha`	double literal	required if `method = "ridge"`; rejected for lasso and gbdt	`>= 0`
`lambda`	double literal	required if `method = "lasso"`; rejected for ridge and gbdt	`> 0`
`horizon`	int literal	`1`	`>= 1`
`target`	signal-context expression	next-bar close-to-close return	causal (backward-looking)

method and model are the only two signal-context kwargs that accept a string literal — strings remain rejected everywhere else in the signal layer. Both are read once at bind time rather than evaluated per bar, which is what makes them safe exceptions rather than a widening of the rule. The binder gives a targeted error if a string sneaks into any other position.

Horizon (horizon = H) — forecast H bars ahead instead of one. At bar t, the predictor returns its forecast for target_{t+H}. Training pairs are (features_{t-H}, target_t) — the causal lag matches the forecast horizon, so a trace recorded with horizon = 5 pairs each y_hat with the realized target observed exactly five bars later. Warm-up grows by H - 1 bars (the ring must accumulate H past feature vectors before the first training pair can push).

Target (target = <expr>) — replace the v1 default ((close - prev_close) / prev_close) with any signal-context expression. Backward-looking forms (lag_return, rolling_return, rolling_zscore, etc.) are causal by construction. Try volatility prediction with target = realized_vol(close, 20), or longer-horizon return prediction with target = lag_return(close, 5) paired with horizon = 5.

This example named rolling_vol(close, 20) until EPIC-86. That is the std-dev of the price level in dollars, not of returns — a model trained on it predicts price level, not volatility. Use realized_vol. See Feature primitives below.

# Predict the 5-day-ahead close-to-close return from a few
# rolling features. Output is causal at bar t for bar t+5.
predict_return(
  lookback = 100,
  alpha    = 1.0,
  horizon  = 5,
  features = [
    lag_return(close, 1),
    rolling_zscore(close, 20),
    rsi(close, 14),
  ],
  target = lag_return(close, 5),
)

Method (method = "ridge" | "lasso" | "gbdt") — pick the solver. Ridge (the default) uses the closed-form (XᵀX + αI′)⁻¹Xᵀy via Cholesky. Lasso uses coordinate descent on the centered design matrix and matches sklearn.linear_model.Lasso(alpha = λ, fit_intercept = True) to ~1e-6 on the canonical 100×5 test design. The crucial behavioral difference is sparsity: at moderate lambda, lasso pushes irrelevant feature coefficients to exact zero, which both removes noise from the forecast and makes the learned model trivially inspectable. Use lasso when the feature set is wide or includes candidates you suspect are noise; use ridge when every feature has prior justification and you mostly want shrinkage.

# Same features, lasso-solved with a tighter penalty. Coefficients
# of features that aren't pulling weight collapse to exact zero.
predict_return(
  lookback = 252,
  method   = "lasso",
  lambda   = 0.01,
  features = [
    lag_return(close, 1),
    lag_return(close, 5),
    rolling_zscore(close, 20),
    rsi(close, 14) / 100.0,
  ],
)

gbdt is the odd one out and is documented as such. It does not fit anything at run time: the ensemble is trained offline in Python, serialised to JSON, and loaded once at bind time. push() and fit() on it are deliberate no-ops, so unlike ridge and lasso its score at bar t depends only on that bar's features and not on any history.

# No lookback, no alpha, no lambda — all three are refused here,
# because there is no fit window and nothing to regularise.
predict_return(
  method   = "gbdt",
  model    = "models/it30_h20.json",
  horizon  = 20,
  features = [
    lag_return(close, 1),
    lag_return(close, 20),
    rolling_zscore(close, 20),
    realized_vol(close, 20),
  ],
)

The model file is untrusted input on the path that decides trades, so it is validated rather than trusted: tree and node counts, depth, feature indices, non-finite thresholds and leaf values, and — the invariant that matters most — every child index must exceed its parent, because a backward edge is a cycle and a cycle silently drops a whole tree out of the ensemble. A malformed file is refused with the reason and the offending index named, never repaired.

The features it scores must be bit-identical to the ones it was trained on. That is what qe_factor --export-matrix exists for: the engine writes the training matrix at %.17g, and Python reads bytes off disk rather than recomputing a single feature in pandas. See the Limitations section below.

alpha and lambda are mutually exclusive — passing both, or passing lambda with method = "ridge" (or alpha with method = "lasso"), is rejected at bind time. lambda > 0 is required for lasso; for pure OLS use method = "ridge", alpha = 0 (which falls back to the Tikhonov-perturbed solve).

The new fields on ForecastTracePoint carry the horizon:

struct ForecastTracePoint {
    std::int64_t ts_ns;
    double       bar_index;     // bar at which `y_realized` was observed
    double       y_hat;         // forecast emitted `horizon` bars earlier
    double       y_realized;    // observed value of `target` at this bar
    std::size_t  horizon;       // == 1 in v1 traces; H in v2 traces
};

Feature primitives¶

Function	Output
`lag_return(price, k)`	`(p_t - p_{t-k}) / p_{t-k}`; NaN k bars
`rolling_vol(x, n)`	n-window sample std of `x` itself; NaN n-1 bars
`realized_vol(x, n)`	n-window sample std of `x`'s 1-bar returns; NaN n bars
`rolling_zscore(x, n)`	`(x_t - mean_n) / std_n`; 0 on constants
`price_level(x)`	identity; marks a deliberate price-level argument

All are per-call-site stateful, O(1) per push for lag_return, O(n) for the rolling stats — well under the per-bar budget for any reasonable window. price_level is pure and stateless.

Watch the units. rolling_vol returns the std-dev of whatever it is fed, so rolling_vol(close, 20) is in dollars and tracks price level (rank correlation +0.82 with close, +0.12 with true return volatility). realized_vol(close, 20) is the dimensionless per-bar return volatility, and is bit-identical to rolling_vol(lag_return(close, 1), 20). Neither is annualized. Feeding a dollar-valued feature into a ridge model alongside dimensionless ones also wrecks the regularization balance, so this is not only a semantics question. Passing a bare price variable to rolling_vol emits a config lint at load — see qe-language.md.

Walk-forward validation¶

The forecasting layer composes with the walk_forward(...) harness. Wrap a backtest(...) that uses predict_return in walk-forward windows; each test slice reports OOS metrics that are not in the predictor's training set:

walk_forward(
  base = backtest(
    data = yahoo("SPY", "1d", "2018-01-01", "2024-12-31"),
    strategy = signal(
      entry = predict_return(60, 1.0, lag_return(close, 1)) > 0.001,
      exit  = predict_return(60, 1.0, lag_return(close, 1)) < 0.0,
      symbol = "SPY",
    ),
    execution = execution(capital = 100_000),
  ),
  train_window = "365d",
  test_window  = "90d",
  step_window  = "90d",
)

Read the resulting results.json. The comparison is per window, not aggregated — walk_forward carries exactly n_windows and windows[], and each entry has both halves:

walk_forward.windows[i].in_sample_metrics.sharpe
walk_forward.windows[i].out_of_sample_metrics.sharpe

If in-sample sharpe is high across windows but the out-of-sample sharpe sits near zero, the predictor is fitting noise. This is the canonical overfit smell-test the forecasting layer is designed to support.

Inspecting forecasts in the dashboard¶

Opt in by setting record_forecast = true on execution(...):

execution(
  capital         = 100_000,
  commission_bps  = 1,
  record_forecast = true,
)

qe_run then drains the per-bar (y_hat, y_realized) trace from every predict_return(...) call site after the run and appends a forecasts[] block to results.json:

"forecasts": [
  {
    "id":         "entry#0",
    "lookback":   60,
    "alpha":      1.0,
    "n_features": 4,
    "metrics": {
      "rmse":                 0.0142,
      "directional_accuracy": 0.51,
      "r2_vs_naive":          0.03,
      "n":                    243
    },
    "series": [
      { "ts_ns": ..., "bar_index": ..., "y_hat": ..., "y_realized": ... },
      ...
    ]
  }
]

The F4 BCKT screen's bottom-right slot detects the block and switches into a scatter view: y_hat on x, y_realized on y, with a 45° identity line and a y=0 reference. The header badge restates n / directional accuracy / RMSE / R² vs naïve. Points above the identity line are bars where the model under-predicted the realized return; the top-right and bottom-left quadrants are direction-correct.

Cost note: each trace point is 24 bytes on disk and a few hundred bytes per render. For 60-bar smoke runs this is free; for 1M-bar minute runs you'll add ~25 MB to results.json. Leave the flag off unless you actually want the overlay.

Limitations of the v1 overlay: - Only the first call site is rendered (typical use: entry#0 and exit#0 are identical predictors, so the picture is the same). A selector across call sites is a follow-up. - walk_forward(...) doesn't currently drain predictor traces through its window-stitched runner — the overlay shows up on single-pass backtests only.

Limitations¶

Targets: target = ... (EPIC-50a) lets you predict volatility, multi-bar return, or any backward-looking expression. Forecast-of-classification (direction probability with a softmax head) is not supported by any method here — gbdt scores a regression leaf sum, not a class probability.
Methods: ridge (default), lasso (EPIC-50b) and gbdt ship today. Neural-net inference via ONNX was planned and deliberately dropped — see EPICS/done/EPIC-51-tree-and-nn-predictors.md. Bayesian regression is not planned.
gbdt does not train. The engine links no training library and never will: xgboost is deliberately absent from vcpkg.json, and only the traversal ships. Training happens offline, and the features it trains on must come from qe_factor --export-matrix rather than being recomputed in pandas — a feature that differs by 1e-9 between training and inference is a model scored on a distribution it never saw, and nothing in the backtest would say so.
No cross-sectional models. The predictor takes per-bar features from one symbol; cross-asset (e.g. residualize on a market factor) needs a future multi-symbol predict primitive.
No shared state across signal positions. An entry and exit expression that both call predict_return(...) instantiate independent ridges. Practical impact: ~2× compute for the same fit; no correctness issue.
No live coefficient inspection from the dashboard yet. The F4 BCKT screen renders the OOS curve when walk-forward is used; an overlay panel showing forecast-vs-realized is a follow-up.

Implementation pointers¶

Concern	File
Closed-form ridge solver	`include/qe/forecast/rolling_ridge.hpp`
Coordinate-descent lasso	`include/qe/forecast/rolling_lasso.hpp`
Boosted-tree loader + forward pass	`include/qe/forecast/gbdt.hpp` `src/forecast/gbdt.cpp`
Training-matrix export	`include/qe/io/feature_matrix.hpp` `src/io/feature_matrix.cpp`
Feature indicators	`include/qe/indicators/lag_return.hpp` `include/qe/indicators/rolling_stats.hpp` `include/qe/indicators/realized_vol.hpp`
DSL builtin registration	`src/dsl/env.cpp`
Per-bar dispatch	`src/dsl/evaluator.cpp` (case `PredictReturn`)
Warmup math	`src/dsl/analysis.cpp`
End-to-end smoke	`tests/fixtures/forecast_smoke.qe`

Pitfalls¶

lookback too small: with 5 features + intercept = 6 coefficients, lookback = 6 is the minimum but the design matrix will be rank-deficient on any colinear inputs; bump to at least 4 × n_features for sane fits. The binder rejects lookback < n_features + 1 to catch the obviously broken case.
64 features is a hard ceiling, not a guideline: the per-bar feature vector is built in a fixed-width stack array so the hot loop allocates nothing. The binder rejects a 65th feature outright (BoundAst::kMaxPredictFeatures). The ceiling used to be documented but unenforced, and a 65-feature call site bound cleanly and then wrote past the end of that array on the first bar.
Highly correlated features: the ridge is robust against collinearity at the LDLT level, but the learned slopes become noisy. Prefer a small set of orthogonal-ish features (lag_return at different horizons, a z-score, RSI normalized).
Refit cost on minute bars: at lookback 1000 and 5 features RollingRidge::fit() runs in ~250 µs. For 1M bars that's about 4 minutes wall time of pure refit — acceptable for one-shot research, painful inside a sweep × walk-forward grid. Drop the lookback or thin the features when that bites.
Lasso convergence: defaults are max_iter = 200, tol = 1e-6. For most signal-discovery tasks that's fine — once the ring slides by one sample, CD warm-starts from the previous fit and converges in 1–3 sweeps. If you're chasing tight numerical agreement with sklearn, the unit tests use max_iter = 200_000, tol = 1e-12; those knobs aren't exposed in the DSL because the signal layer doesn't need that precision.
Don't mix alpha and lambda: the binder rejects passing both, or passing lambda with method = "ridge", because they parameterize different objectives. The error message points to the offending kwarg.