102 lines
5.8 KiB
Markdown
102 lines
5.8 KiB
Markdown
# Result Report November (1)
|
||
## 1. TODOs
|
||
|
||
- [x] Compare autoregressive vs non-autoregressive
|
||
- [x] Rewrite dataloader for more input parameters (load forecast)
|
||
- [ ] Explore more input parameters (load forecast)
|
||
- [x] Quantile Regression sampling fix
|
||
- [x] Quantile Regression exploration
|
||
- [x] Plots with good scaling (y-axis)
|
||
|
||
- [x] Some days in load forecast are missing, remove samples from dataset (Implemented a skip in the NRVDataset)
|
||
- [ ] Quantile Regression nakijken
|
||
- [ ] Test scores voor 96 values
|
||
|
||
- [ ] (Optional) Andere modellen (LSTM?)
|
||
|
||
## 2. Autoregressive vs Non-Autoregressive
|
||
|
||
Training data: 2015 - 2022 \
|
||
Batch_size: 1024 \
|
||
Learning_rate: 0.0003 \
|
||
Early_stopping: 10
|
||
|
||
### 2.1 Linear Model
|
||
Comparison: [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/compare-experiments;ids=3b512226b24c46199b584d7abc23bf96,097b3832eb5e4c5a8fa2e04887975c29/scalars/values?scalars=values)
|
||
<!-- table with 3 columns and rows: experiment, Train-MAE, Train-MSE, Test-MAE, Test-MSE -->
|
||
| | Autoregressive | Non-Autoregressive |
|
||
| --- | --- | --- |
|
||
| Experiment (ClearML) | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/3b512226b24c46199b584d7abc23bf96/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/097b3832eb5e4c5a8fa2e04887975c29/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update&filter=) |
|
||
| Train-MAE | 68.0202865600586 | 94.83179473876953 |
|
||
| Train-MSE | 7861.2197265625 | 15977.8759765625 |
|
||
| Test-MAE | 78.05316925048828 | 104.11575317382812 |
|
||
| Test-MSE | 10882.755859375 | 21145.583984375 |
|
||
|
||
### 2.2 Non Linear Model
|
||
|
||
Hidden layers: 1 \
|
||
Hidden units: 512
|
||
|
||
Comparison: [Link](hhttps://clearml.victormylle.be/projects/135c055e64e54486a54055f33ca9a5af/compare-experiments;ids=57f4a09c6ce54296b6e034f2e0236420,b5f6862f900948c18a834a1a970c8710/scalars/values?scalars=values)
|
||
|
||
| | Autoregressive | Non-Autoregressive |
|
||
| --- | --- | --- |
|
||
| Experiment (ClearML) | [Link](https://clearml.victormylle.be/projects/135c055e64e54486a54055f33ca9a5af/experiments/57f4a09c6ce54296b6e034f2e0236420/output/execution) | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/b5f6862f900948c18a834a1a970c8710/output/execution) |
|
||
| Train-MAE | 66.78179931640625 | 94.52633666992188 |
|
||
| Train-MSE | 7507.53955078125 | 15835.671875 |
|
||
| Test-MAE | 77.63729095458984 | 104.07229614257812 |
|
||
| Test-MSE | 10789.544921875 | 21090.9765625 |
|
||
|
||
Also tried with [3 hidden layers for Non-Autoregressive model](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/583d0bc1dfb7494d81cf356f6d003dbb/output/execution), test results didn't improve.
|
||
|
||
# 3. Quantile Regression
|
||
|
||
## 3.1 Sampling Fix
|
||
The model outputs values for the quantiles it is trained on. For example, the quantiles [0.025, 0.05, 0.1, 0.15, 0.85, 0.9, 0.95, 0.975] can be used. The values outputted by the model are like: [-0.23013, -0.19831, -0.15217, -0.13654, 0.011687, 0.015129, 0.043187, 0.047704]. Plotting these as CDF:
|
||
|
||
\
|
||
*Plotted output values with cubic interpolation*
|
||
|
||
Samling from a uniform distribution, we can convert it to our distribution by using the inverse CDF. In python we can interpolate immediately by switching the x and y axis. This gives us the following code:
|
||
```
|
||
interp1d(quantiles, output_values, kind='quadratic', bounds_error=False, fill_value="extrapolate")
|
||
```
|
||
The mean of x amount of samples can be calculated.
|
||
|
||
## 3.2 Exploration
|
||
### 3.2.1 Linear Model
|
||
Learning Rate: 0.0003 \
|
||
Batch Size: 1024 \
|
||
Early Stopping: 10 \
|
||
Trining Data: 2015 - 2022
|
||
|
||
| Quantiles | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
|
||
| --- | --- | --- | --- | --- |
|
||
| 0.025, 0.1, 0.2, 0.3, 0.5, 0.6, 0.8, 0.85, 0.9, 0.975 | 68.07254628777868 | 7872.668472187121 | 78.11135584669907 | 10903.793883789216 |
|
||
| 0.025, 0.1, 0.15, 0.2, 0.5, 0.8, 0.85, 0.9, 0.975 | 68.0732244289865 | 7873.212834241974 | 78.1143230666738 | 10907.350919114313 |
|
||
| 0.025, 0.05, 0.1, 0.15, 0.85, 0.9, 0.95, 0.975 | 68.2798014428824 | 7936.7644114273935 | 78.50109206637464 | 11005.706457116454 |
|
||
| 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 | 68.06139224378171 | 7871.467571921973 | 78.10843456751378 | 10904.55519059502 |
|
||
| 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9 | 68.0350635204691 | 7868.661882749334 | 78.1139478264609 | 10905.562798801011 |
|
||
| 0.1, 0.2, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.8, 0.9 | 68.03191172170285 | 7868.483061240721 | 78.13204232055722 | 10908.837301340453 |
|
||
|
||
These are all very close
|
||
|
||
### 3.2.2 Non Linear Model
|
||
Learning Rate: 0.0003 \
|
||
Batch Size: 1024 \
|
||
Early Stopping: 10 \
|
||
Trining Data: 2015 - 2022 \
|
||
Hidden Layers: 3 \
|
||
Hidden Units: 1024
|
||
|
||
| Quantiles | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
|
||
| --- | --- | --- | --- | --- |
|
||
| 0.1, 0.2, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.8, 0.9 | 65.6542529596313 | 7392.5142575554955 | 77.55779692831604 | 10769.161724849037 |
|
||
| 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 | 65.68495924348356 | 7326.2239225611975 | 77.62433888969542 | 10789.003223366473 |
|
||
|
||
Also tried one with dropout of 0.3, results are better. This needs to be tested more (hyperparameter tuning is not the focus here):
|
||
|
||
| Quantiles | Train-MAE | Train-MSE | Test-MAE | Test-MSE |
|
||
| --- | --- | --- | --- | --- |
|
||
| 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 | 63.710736942371994 | 6988.6436956508105 | 77.29499496466444 | 10706.484005597813 |
|