# Data Pipeline Bug Analysis - Final Status

## Summary

After fixing all identified bugs, the feature count now matches (341), but the embeddings remain uncorrelated with the database 0_7 version.

**Latest Version**: v5
- Feature count: 341 ✓ (matches VAE input dim)
- Mean correlation with DB: 0.0050 (essentially zero)
- Status: All identified bugs fixed, but embeddings still differ

---

## Bugs Fixed

### 1. Market Classification (`FlagMarketInjector`) ✓ FIXED
- **Bug**: Used `instrument >= 600000` which misclassified 新三板 instruments
- **Fix**: Use string prefix matching with vocab_size=2 (not 3)
- **Impact**: 167 instruments corrected

### 2. ColumnRemover Missing `IsN` ✓ FIXED
- **Bug**: Only removed `IsZt, IsDt` but not `IsN`
- **Fix**: Added `IsN` to removal list
- **Impact**: Feature count alignment

### 3. RobustZScoreNorm Scope ✓ FIXED
- **Bug**: Applied normalization to all 341 features
- **Fix**: Only normalize 330 features (alpha158 + market_ext, both original + neutralized)
- **Impact**: Correct normalization scope

### 4. Wrong Data Sources for Market Flags ✓ FIXED
- **Bug**: Used `Limit, Stopping` (Float64) from kline_adjusted
- **Fix**: Load from correct sources:
  - kline_adjusted: `IsZt, IsDt, IsN, IsXD, IsXR, IsDR` (Boolean)
  - market_flag: `open_limit, close_limit, low_limit, high_stop` (Boolean, 4 cols)
- **Impact**: Correct boolean flag data

### 5. Feature Count Mismatch ✓ FIXED
- **Bug**: 344 features (3 extra)
- **Fix**: vocab_size=2 + 4 market_flag cols = 341 features
- **Impact**: VAE input dimension matches

---

## Correlation Results (v5)

| Metric | Value |
|--------|-------|
| Mean correlation (32 dims) | 0.0050 |
| Median correlation | 0.0079 |
| Min | -0.0420 |
| Max | 0.0372 |
| Overall (flattened) | 0.2225 |

**Conclusion**: Embeddings remain essentially uncorrelated with database.

---

## Possible Remaining Issues

1. **Different input data values**: The alpha158_0_7_beta Parquet files may contain different values than the original DolphinDB data used to train the VAE.

2. **Feature ordering mismatch**: The 330 RobustZScoreNorm parameters must be applied in the exact order:
   - [0:158] = alpha158 original
   - [158:316] = alpha158_ntrl
   - [316:323] = market_ext original (7 cols)
   - [323:330] = market_ext_ntrl (7 cols)

3. **Industry neutralization differences**: Our `IndusNtrlInjector` implementation may differ from qlib's.

4. **Missing transformations**: There may be additional preprocessing steps not captured in handler.yaml.

5. **VAE model mismatch**: The VAE model may have been trained with different data than what handler.yaml specifies.

---

## Recommended Next Steps

1. **Compare intermediate features**: Run both the qlib pipeline and our pipeline on the same input data and compare outputs at each step.

2. **Verify RobustZScoreNorm parameter order**: Check if our feature ordering matches the order used during VAE training.

3. **Compare predictions, not embeddings**: Instead of comparing VAE embeddings, compare the final d033 model predictions with the original 0_7 predictions.

4. **Check alpha158 data source**: Verify that `stg_1day_wind_alpha158_0_7_beta_1D` contains the same data as the original DolphinDB `stg_1day_wind_alpha158_0_7_beta` table.