160 lines
12 KiB
Markdown
160 lines
12 KiB
Markdown
# Result Report November (1)
|
||
## 1. TODOs
|
||
|
||
- [x] Compare autoregressive vs non-autoregressive
|
||
- [x] Rewrite dataloader for more input parameters (load forecast)
|
||
- [x] Explore more input parameters (load forecast)
|
||
- [x] Quantile Regression sampling fix
|
||
- [x] Quantile Regression exploration
|
||
- [x] Plots with good scaling (y-axis)
|
||
|
||
- [x] Some days in load forecast are missing, remove samples from dataset (Implemented a skip in the NRVDataset)
|
||
- [x] Quantile Regression nakijken
|
||
- [x] Test scores voor 96 values
|
||
|
||
- [ ] (Optional) Andere modellen (LSTM?)
|
||
|
||
- [x] Non autoregressive Quantile Regression
|
||
- [x] Fix debug plots for quantile regression -> predict quantiles and look if true value is below a quantile, if so 1 else 0 and average these over all samples
|
||
- [ ] Full day debug plots for quantile regression
|
||
- [x] CPRS Metrics
|
||
- [x] Time as input parameter:
|
||
- [x] Probabilistic Baseline -> Quantiles on Training Data -> Breedte bekijken -> Gebruiken voor CPRS en plotjes
|
||
- [ ] Day-ahead implicit net position
|
||
|
||
- [x] Faster sampling for quantile regression
|
||
- [ ] Quantile plots for other model (Linear, GRU) (Check if better)
|
||
- [ ] Check example plots to see if metrics correspond with what seen on plots
|
||
- [x] Time step (96 values) to embedding layer
|
||
- [x] Mean of nrv per time step plotten (done for probabilistic baseline)
|
||
- [x] Convert back to MW on plots
|
||
|
||
## 2. Autoregressive vs Non-Autoregressive
|
||
|
||
Training data: 2015 - 2022 \
|
||
Batch_size: 1024 \
|
||
Learning_rate: 0.0003 \
|
||
Early_stopping: 10
|
||
|
||
### 2.1 Linear Model
|
||
Comparison: [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/compare-experiments;ids=3b512226b24c46199b584d7abc23bf96,097b3832eb5e4c5a8fa2e04887975c29/scalars/values?scalars=values)
|
||
<!-- table with 3 columns and rows: experiment, Train-MAE, Train-MSE, Test-MAE, Test-MSE -->
|
||
| | Autoregressive | Non-Autoregressive |
|
||
| --- | --- | --- |
|
||
| Experiment (ClearML) | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/3b512226b24c46199b584d7abc23bf96/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/097b3832eb5e4c5a8fa2e04887975c29/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) |
|
||
| Train-MAE | 68.0202865600586 | 94.83179473876953 |
|
||
| Train-MSE | 7861.2197265625 | 15977.8759765625 |
|
||
| Test-MAE | 78.05316925048828 | 104.11575317382812 |
|
||
| Test-MSE | 10882.755859375 | 21145.583984375 |
|
||
|
||
### 2.2 Non Linear Model
|
||
|
||
Hidden layers: 1 \
|
||
Hidden units: 512
|
||
|
||
Comparison: [Link](hhttps://clearml.victormylle.be/projects/135c055e64e54486a54055f33ca9a5af/compare-experiments;ids=57f4a09c6ce54296b6e034f2e0236420,b5f6862f900948c18a834a1a970c8710/scalars/values?scalars=values)
|
||
|
||
| | Autoregressive | Non-Autoregressive |
|
||
| --- | --- | --- |
|
||
| Experiment (ClearML) | [Link](https://clearml.victormylle.be/projects/135c055e64e54486a54055f33ca9a5af/experiments/57f4a09c6ce54296b6e034f2e0236420/output/execution) | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/b5f6862f900948c18a834a1a970c8710/output/execution) |
|
||
| Train-MAE | 66.78179931640625 | 94.52633666992188 |
|
||
| Train-MSE | 7507.53955078125 | 15835.671875 |
|
||
| Test-MAE | 77.63729095458984 | 104.07229614257812 |
|
||
| Test-MSE | 10789.544921875 | 21090.9765625 |
|
||
|
||
Also tried with [3 hidden layers for Non-Autoregressive model](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/583d0bc1dfb7494d81cf356f6d003dbb/output/execution), test results didn't improve.
|
||
|
||
# 3. Quantile Regression
|
||
|
||
## 3.1 Sampling Fix
|
||
The model outputs values for the quantiles it is trained on. For example, the quantiles [0.025, 0.05, 0.1, 0.15, 0.85, 0.9, 0.95, 0.975] can be used. The values outputted by the model are like: [-0.23013, -0.19831, -0.15217, -0.13654, 0.011687, 0.015129, 0.043187, 0.047704]. Plotting these as CDF:
|
||
|
||
\
|
||
*Plotted output values with cubic interpolation*
|
||
|
||
Samling from a uniform distribution, we can convert it to our distribution by using the inverse CDF. In python we can interpolate immediately by switching the x and y axis. This gives us the following code:
|
||
```
|
||
interp1d(quantiles, output_values, kind='quadratic', bounds_error=False, fill_value="extrapolate")
|
||
```
|
||
The mean of x amount of samples can be calculated.
|
||
|
||
## 3.2 Exploration
|
||
### 3.2.1 Linear Model
|
||
Learning Rate: 0.0003 \
|
||
Batch Size: 1024 \
|
||
Early Stopping: 10 \
|
||
Trining Data: 2015 - 2022
|
||
|
||
| Quantiles | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
|
||
| --- | --- | --- | --- | --- |
|
||
| 0.025, 0.1, 0.2, 0.3, 0.5, 0.6, 0.8, 0.85, 0.9, 0.975 | 68.07254628777868 | 7872.668472187121 | 78.11135584669907 | 10903.793883789216 |
|
||
| 0.025, 0.1, 0.15, 0.2, 0.5, 0.8, 0.85, 0.9, 0.975 | 68.0732244289865 | 7873.212834241974 | 78.1143230666738 | 10907.350919114313 |
|
||
| 0.025, 0.05, 0.1, 0.15, 0.85, 0.9, 0.95, 0.975 | 68.2798014428824 | 7936.7644114273935 | 78.50109206637464 | 11005.706457116454 |
|
||
| 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 | 68.06139224378171 | 7871.467571921973 | 78.10843456751378 | 10904.55519059502 |
|
||
| 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9 | 68.0350635204691 | 7868.661882749334 | 78.1139478264609 | 10905.562798801011 |
|
||
| 0.1, 0.2, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.8, 0.9 | 68.03191172170285 | 7868.483061240721 | 78.13204232055722 | 10908.837301340453 |
|
||
|
||
These are all very close
|
||
|
||
### 3.2.2 Non Linear Model
|
||
Learning Rate: 0.0003 \
|
||
Batch Size: 1024 \
|
||
Early Stopping: 10 \
|
||
Trining Data: 2015 - 2022 \
|
||
Hidden Layers: 3 \
|
||
Hidden Units: 1024
|
||
|
||
| Quantiles | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
|
||
| --- | --- | --- | --- | --- |
|
||
| 0.1, 0.2, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.8, 0.9 | 65.6542529596313 | 7392.5142575554955 | 77.55779692831604 | 10769.161724849037 |
|
||
| 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 | 65.68495924348356 | 7326.2239225611975 | 77.62433888969542 | 10789.003223366473 |
|
||
|
||
Also tried one with dropout of 0.3, results are better. This needs to be tested more (hyperparameter tuning is not the focus here):
|
||
|
||
| Quantiles | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
|
||
| --- | --- | --- | --- | --- |
|
||
| 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 | 63.710736942371994 | 6988.6436956508105 | 77.29499496466444 | 10706.484005597813 |
|
||
|
||
|
||
# More input parameters
|
||
## Non-Autoregressive
|
||
Learning Rate: 0.0003 \
|
||
Batch Size: 1024 \
|
||
Early Stopping: 15 \
|
||
Trining Data: 2015 - 2022 \
|
||
Hidden Layers: 3 \
|
||
Hidden Units: 1024
|
||
|
||
| Input Parameters | Experiment | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| NRV History | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/cede0babadbb41d49dbcfd6ade78cd5e/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 94.40104675292969 | 15815.435546875 | 104.87157440185547 | 21358.115234375 |
|
||
| NRV History + Load History | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/aa7768bf0c054f55a55d99c15b3f0f25/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 92.76568603515625 | 15250.294921875 | 104.78298950195312 | 21043.908203125 |
|
||
| NRV History + Load History + Load Forecast | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/b213c7863f0f479f80bf612a9259c533/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 92.92375946044922 | 15294.857421875 | 104.52192687988281 | 20994.251953125 |
|
||
|
||
# Quantile Regression Debugging
|
||
[Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/50d0c7922faa4681816635c9cfa3eb72/info-output/debugImages?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=)
|
||
|
||
|
||
# Scores on full day prediction
|
||
Only history NRV as features
|
||
|
||
Learning Rate: 0.0003 \
|
||
Batch Size: 1024 \
|
||
Early Stopping: 15 \
|
||
Trining Data: 2015 - 2022 \
|
||
Hidden Layers: 3 \
|
||
Hidden Units: 1024
|
||
|
||
| Model | Experiment | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| Non-Autoregressive | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/097b3832eb5e4c5a8fa2e04887975c29/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 94.52633666992188 | 15835.671875 | 104.07229614257812 | 21090.9765625 |
|
||
| Auto-Regressive | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/65748b5d30054d3381a76e2c5a73e107/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | - | - | 105.1736068725586 | 21376.697265625 |
|
||
| Quantile Regression | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/df5669968cf64c42ba7a97fc2d745b76/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | - | - | 105.53107468002209 | 21656.24950570062 |
|
||
|
||
|
||
# CRPS Metric
|
||
| Model | Experiment | Train-MAE | Train-MSE | Train-CRPS | Test-MAE | Test-MSE | Test-CRPS |
|
||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||
| Probabilistic Baseline | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/599152a9e44d4ba6a5bcb603e5041b01/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | - | - | 72.78830217810247 | - | - | 75.9605281456783 |
|
||
| Non-Autoregressive Quantile | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/c50a62963dd649c387f1122ccee61d2f/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | 98.43774474341907 | 17433.701092295152 | 70.63047790527344 | 104.28421422336042 | 20851.083458159148 | 74.81269836425781 |
|
||
| Auto-Regressive Quantile | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/8db35a590cfa46f081c7f4caf93d711d/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | - | - | - | 107.24027992397264 | 22016.697427833686 | 68.19192504882812 | |