# Result Report November (1)
## 1. TODOs

- [x] Compare autoregressive vs non-autoregressive
- [x] Rewrite dataloader for more input parameters (load forecast)
- [x] Explore more input parameters (load forecast)
- [x] Quantile Regression sampling fix
- [x] Quantile Regression exploration
- [x] Plots with good scaling (y-axis)
  
- [x] Some days in load forecast are missing, remove samples from dataset (Implemented a skip in the NRVDataset)
- [x] Quantile Regression nakijken
- [x] Test scores voor 96 values

- [x] Andere modellen (LSTM?)

- [x] Non autoregressive Quantile Regression
- [x] Fix debug plots for quantile regression -> predict quantiles and look if true value is below a quantile, if so 1 else 0 and average these over all samples
- [x] Full day debug plots for quantile regression
- [x] CPRS Metrics
- [x] Time as input parameter:
- [x] Probabilistic Baseline -> Quantiles on Training Data -> Breedte bekijken -> Gebruiken voor CPRS en plotjes
- [x] Day-ahead implicit net position

- [x] Faster sampling for quantile regression
- [x] Quantile plots for other model (Linear, GRU) (Check if better)
- [ ] Check example plots to see if metrics correspond with what seen on plots
- [x] Time step (96 values) to embedding layer
- [x] Mean of nrv per time step plotten (done for probabilistic baseline)
- [x] Convert back to MW on plots
- [x] Background model training

## 2. Autoregressive vs Non-Autoregressive

Training data: 2015 - 2022 \
Batch_size: 1024 \
Learning_rate: 0.0003 \
Early_stopping: 10 

### 2.1 Linear Model
Comparison: [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/compare-experiments;ids=3b512226b24c46199b584d7abc23bf96,097b3832eb5e4c5a8fa2e04887975c29/scalars/values?scalars=values)
<!-- table with 3 columns and rows: experiment, Train-MAE, Train-MSE, Test-MAE, Test-MSE -->
|  | Autoregressive | Non-Autoregressive | 
| --- | --- | --- |
| Experiment (ClearML) | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/3b512226b24c46199b584d7abc23bf96/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/097b3832eb5e4c5a8fa2e04887975c29/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) |
| Train-MAE | 68.0202865600586 | 94.83179473876953 |
| Train-MSE | 7861.2197265625 | 15977.8759765625 |
| Test-MAE | 78.05316925048828 | 104.11575317382812 |
| Test-MSE | 10882.755859375 | 21145.583984375 |

### 2.2 Non Linear Model

Hidden layers: 1 \
Hidden units: 512

Comparison: [Link](hhttps://clearml.victormylle.be/projects/135c055e64e54486a54055f33ca9a5af/compare-experiments;ids=57f4a09c6ce54296b6e034f2e0236420,b5f6862f900948c18a834a1a970c8710/scalars/values?scalars=values)

|  | Autoregressive | Non-Autoregressive |
| --- | --- | --- |
| Experiment (ClearML) | [Link](https://clearml.victormylle.be/projects/135c055e64e54486a54055f33ca9a5af/experiments/57f4a09c6ce54296b6e034f2e0236420/output/execution) | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/b5f6862f900948c18a834a1a970c8710/output/execution) |
| Train-MAE | 66.78179931640625 | 94.52633666992188 |
| Train-MSE | 7507.53955078125 | 15835.671875 |
| Test-MAE | 77.63729095458984 | 104.07229614257812 |
| Test-MSE | 10789.544921875 | 21090.9765625 |

Also tried with [3 hidden layers for Non-Autoregressive model](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/583d0bc1dfb7494d81cf356f6d003dbb/output/execution), test results didn't improve.

# 3. Quantile Regression

## 3.1 Sampling Fix
The model outputs values for the quantiles it is trained on. For example, the quantiles [0.025, 0.05, 0.1, 0.15, 0.85, 0.9, 0.95, 0.975] can be used. The values outputted by the model are like: [-0.23013, -0.19831, -0.15217, -0.13654, 0.011687, 0.015129, 0.043187, 0.047704]. Plotting these as CDF:

![Plotted output values](images/quantile_regression_sampling.png)\
*Plotted output values with cubic interpolation*

Samling from a uniform distribution, we can convert it to our distribution by using the inverse CDF. In python we can interpolate immediately by switching the x and y axis. This gives us the following code:
```
interp1d(quantiles, output_values, kind='quadratic', bounds_error=False, fill_value="extrapolate")
```
The mean of x amount of samples can be calculated.

## 3.2 Exploration
### 3.2.1 Linear Model
Learning Rate: 0.0003 \
Batch Size: 1024 \
Early Stopping: 10 \
Trining Data: 2015 - 2022

| Quantiles | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
| --- | --- | --- | --- | --- |
| 0.025, 0.1, 0.2, 0.3, 0.5, 0.6, 0.8, 0.85, 0.9, 0.975 | 68.07254628777868 | 7872.668472187121 | 78.11135584669907 | 10903.793883789216 |
| 0.025, 0.1, 0.15, 0.2, 0.5, 0.8, 0.85, 0.9, 0.975 | 68.0732244289865 | 7873.212834241974 | 78.1143230666738 | 10907.350919114313 |
| 0.025, 0.05, 0.1, 0.15, 0.85, 0.9, 0.95, 0.975 | 68.2798014428824 | 7936.7644114273935 | 78.50109206637464 | 11005.706457116454 |
| 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 | 68.06139224378171 | 7871.467571921973 | 78.10843456751378 | 10904.55519059502 |
| 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9 | 68.0350635204691 | 7868.661882749334 | 78.1139478264609 | 10905.562798801011 |
| 0.1, 0.2, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.8, 0.9 | 68.03191172170285 | 7868.483061240721 | 78.13204232055722 | 10908.837301340453 |

These are all very close

### 3.2.2 Non Linear Model
Learning Rate: 0.0003 \
Batch Size: 1024 \
Early Stopping: 10 \
Trining Data: 2015 - 2022 \
Hidden Layers: 3 \
Hidden Units: 1024

| Quantiles | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
| --- | --- | --- | --- | --- |
| 0.1, 0.2, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.8, 0.9 | 65.6542529596313 | 7392.5142575554955 | 77.55779692831604 | 10769.161724849037 |
| 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 | 65.68495924348356 | 7326.2239225611975 | 77.62433888969542 | 10789.003223366473 |

Also tried one with dropout of 0.3, results are better. This needs to be tested more (hyperparameter tuning is not the focus here):

| Quantiles | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
| --- | --- | --- | --- | --- |
| 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 | 63.710736942371994 | 6988.6436956508105 | 77.29499496466444 | 10706.484005597813 |


# More input parameters
## Non-Autoregressive
Learning Rate: 0.0003 \
Batch Size: 1024 \
Early Stopping: 15 \
Trining Data: 2015 - 2022 \
Hidden Layers: 3 \
Hidden Units: 1024

| Input Parameters | Experiment | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
| --- | --- | --- | --- | --- | --- |
| NRV History | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/cede0babadbb41d49dbcfd6ade78cd5e/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 94.40104675292969 | 15815.435546875 | 104.87157440185547 | 21358.115234375 |
| NRV History + Load History | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/aa7768bf0c054f55a55d99c15b3f0f25/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 92.76568603515625 | 15250.294921875 | 104.78298950195312 | 21043.908203125 |
| NRV History + Load History + Load Forecast | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/b213c7863f0f479f80bf612a9259c533/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 92.92375946044922 | 15294.857421875 | 104.52192687988281 | 20994.251953125 |

# Quantile Regression Debugging
[Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/50d0c7922faa4681816635c9cfa3eb72/info-output/debugImages?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=)


# Scores on full day prediction
Only history NRV as features

Learning Rate: 0.0003 \
Batch Size: 1024 \
Early Stopping: 15 \
Trining Data: 2015 - 2022 \
Hidden Layers: 3 \
Hidden Units: 1024

| Model | Experiment | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
| --- | --- | --- | --- | --- | --- |
| Non-Autoregressive | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/097b3832eb5e4c5a8fa2e04887975c29/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 94.52633666992188 | 15835.671875 | 104.07229614257812 | 21090.9765625 |
| Auto-Regressive | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/65748b5d30054d3381a76e2c5a73e107/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | - | - | 105.1736068725586 | 21376.697265625 |
| Quantile Regression | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/df5669968cf64c42ba7a97fc2d745b76/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | - | - | 105.53107468002209 | 21656.24950570062 |


# CRPS Metric
| Model | Experiment | Train-MAE | Train-MSE | Train-CRPS | Test-MAE | Test-MSE | Test-CRPS |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Probabilistic Baseline | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/599152a9e44d4ba6a5bcb603e5041b01/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | - | - | 72.78830217810247 | - | - | 75.9605281456783 |
| Non-Autoregressive Quantile | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/c50a62963dd649c387f1122ccee61d2f/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 98.43774474341907 | 17433.701092295152 | 70.63047790527344 | 104.28421422336042 | 20851.083458159148 | 74.81269836425781 |
| Auto-Regressive Quantile | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/8db35a590cfa46f081c7f4caf93d711d/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | - | - | - | 107.24027992397264 | 22016.697427833686 | 68.19192504882812 |