203 lines
14 KiB
Markdown
203 lines
14 KiB
Markdown
# Tasks
|
||
- [ ] Quantiles zelf breder maken na fitten, literatuur bekijken (overconfident voor ondergrens) (Validation set januari 2023)
|
||
Eerst literatuur bekijken ofdat probleem al voorkomt
|
||
|
||
- [ ] compare reconstructed prices with real imbalance prices (on figure and metrics on whole test set)
|
||
|
||
- [ ] Baseline policy (Eche imbalanceprijzen):
|
||
- Batterij:
|
||
- 2 MWh
|
||
- 1 MW power (charging / discharging)
|
||
- Training data -> 2 tresholds (1 voor charging, 1 voor discharging) (Volledig chargen / dischargen)
|
||
- 2 Mwh -> 8 kwartieren
|
||
|
||
- 1000 generaties voor 1 dag -> 1000 prijzen reconstrueren -> tresholds bepalen (logische tresholds) -> 2 tresholds voor 1 dag en toepassen op volledige test set
|
||
|
||
-> resultaten: # charge cycle
|
||
|
||
- [ ] Learning rate non autoregressive
|
||
|
||
- [x] !!! Historgram quantile plot volledige dag
|
||
- [x] CRPS en MAE, over 96 wanneer wordt het slecht
|
||
- [x] crps loss fixen
|
||
- [x] More input parameters
|
||
- [x] Non autoregressive (baseline for linear)
|
||
|
||
(
|
||
- NRV to prices (bid ladder from yesterday) -> Elia
|
||
- Simple policy
|
||
)
|
||
|
||
- [ ] Diffusion vs Variable Selection Network
|
||
|
||
# Different Model Architectures (For Quantile Regression)
|
||
## Baseline
|
||
The baseline just calculates the values for the given quantiles using the training data.
|
||
|
||
|
||
|
||
*Mean NRV for whole day*
|
||
|
||
|
||
|
||
| train_CRPSLoss | test_CRPSLoss |
|
||
|---|---|
|
||
| 70.70666326392862 | 76.03928650729421 |
|
||
|
||
# Auto Regressive Models
|
||
### Linear Model
|
||
#### Example summary of the Linear Model
|
||
```
|
||
==========================================================================================
|
||
Layer (type:depth-idx) Output Shape Param #
|
||
==========================================================================================
|
||
Sequential [1024, 13] --
|
||
├─TimeEmbedding: 1-1 [1024, 195] --
|
||
│ └─Embedding: 2-1 [1024, 2] 192
|
||
├─LinearRegression: 1-2 [1024, 13] --
|
||
│ └─Linear: 2-2 [1024, 13] 2,548
|
||
==========================================================================================
|
||
Total params: 2,740
|
||
Trainable params: 2,740
|
||
Non-trainable params: 0
|
||
Total mult-adds (M): 2.81
|
||
==========================================================================================
|
||
Input size (MB): 0.79
|
||
Forward/backward pass size (MB): 0.12
|
||
Params size (MB): 0.01
|
||
Estimated Total Size (MB): 0.93
|
||
==========================================================================================
|
||
```
|
||
|
||
| Experiment | Quarter | Day Of Week | Load | Wind | Net | test_L1Loss | test_CRPSLoss |
|
||
|---|---|---|---|---|---|---|---|
|
||
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/42f3795e5863464b8aa25bcdac75cf08/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | True | True | 101.63938813784577 | 68.4846434120529 |
|
||
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/848565e544e34748bf80ac7eca379a50/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | False | False | 105.50163727880451 | 73.50646092092921 |
|
||
|
||
|
||
#### Quantiles Histogram Debug Plots
|
||
|
||
<p float="middle" width=100%>
|
||
<img src="december_images/Quantile_Performance_Comparison_Training_Set_00000014.jpeg" width="49%" />
|
||
<img src="december_images/Quantile_Performance_Comparison_Test_Set_00000014.jpeg" width="49%" />
|
||
</p>
|
||
|
||
### Non Linear Model
|
||
|
||
#### Example summary of the Non Linear Model
|
||
```
|
||
==========================================================================================
|
||
Layer (type:depth-idx) Output Shape Param #
|
||
==========================================================================================
|
||
Sequential [1024, 13] --
|
||
├─TimeEmbedding: 1-1 [1024, 96] --
|
||
├─NonLinearRegression: 1-2 [1024, 13] --
|
||
│ └─ModuleList: 2-9 -- (recursive)
|
||
│ │ └─Linear: 3-1 [1024, 512] 49,664
|
||
│ └─ReLU: 2-2 [1024, 512] --
|
||
│ └─ModuleList: 2-9 -- (recursive)
|
||
│ │ └─Dropout: 3-2 [1024, 512] --
|
||
│ └─ReLU: 2-4 [1024, 512] --
|
||
│ └─ModuleList: 2-9 -- (recursive)
|
||
│ │ └─Linear: 3-3 [1024, 512] 262,656
|
||
│ └─ReLU: 2-6 [1024, 512] --
|
||
│ └─ModuleList: 2-9 -- (recursive)
|
||
│ │ └─Dropout: 3-4 [1024, 512] --
|
||
│ └─ReLU: 2-8 [1024, 512] --
|
||
│ └─ModuleList: 2-9 -- (recursive)
|
||
│ │ └─Linear: 3-5 [1024, 13] 6,669
|
||
==========================================================================================
|
||
Total params: 318,989
|
||
Trainable params: 318,989
|
||
Non-trainable params: 0
|
||
Total mult-adds (M): 326.64
|
||
==========================================================================================
|
||
Input size (MB): 0.39
|
||
Forward/backward pass size (MB): 8.50
|
||
Params size (MB): 1.28
|
||
Estimated Total Size (MB): 10.16
|
||
==========================================================================================
|
||
```
|
||
|
||
| Experiment | Quarter | Day Of Week | Load | Wind | Net | test_L1Loss | test_CRPSLoss |
|
||
|---|---|---|---|---|---|---|---|
|
||
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/6b50442e1cec4bf9b3bd5a34077b4217/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | False | False | False | 103.6061492655297 | 67.56606704206783 |
|
||
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/9d30a4c6d63d44d18dff2496d1ec67f1/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | False | False | 103.6867771851146 | 68.6250203464861 |
|
||
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/45aa03c6e9f447859b1f8915576002c4/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | True | True | 102.03053663410438 | 68.968302991015 |
|
||
|
||
|
||
#### Quantiles Histogram Debug Plots
|
||
|
||
<p float="middle" width=100%>
|
||
<img src="december_images/Quantile_Performance_Comparison_Training Set_00000009.jpeg" width="49%" />
|
||
<img src="december_images/Quantile_Performance_Comparison_Test Set_00000009.jpeg" width="49%" />
|
||
</p>
|
||
|
||
### LSTM/GRU Model
|
||
Input shape: (batch_size, sequence_length, input_features) \
|
||
If load forecast is used as input, each time step gets the load forecast for the next time step as feature.
|
||
|
||
Example:
|
||
We have time steps:
|
||
|
||
| | 1 | 2 | 3 | 4 | 5 |
|
||
|---|---|---|---|---|---|
|
||
| NRV | 0.1 | 0.2 | 0.15 | 0.3 | 0.4 |
|
||
| Load forecast | 0.4 | 0.23 | 0.48 | 0.2 | 0.1 |
|
||
|
||
If we want to predict the NRV for time step 5 using the information we have beforehand, we can use the NRV from the previous time steps. We can however also use the load forecast of time step 5. To incorporate this information as input, we need to move the load forecast one time step back. This means, that the input for time step 5 is given with the NRV of time step 4.
|
||
|
||
If the time is also wanted as input, we add this as a feature for every timestep aswell.
|
||
|
||
#### Example summary of the LSTM/GRU Model
|
||
```
|
||
==========================================================================================
|
||
Layer (type:depth-idx) Output Shape Param #
|
||
==========================================================================================
|
||
Sequential [512, 13] --
|
||
├─TimeEmbedding: 1-1 [512, 96, 5] --
|
||
│ └─Embedding: 2-1 [512, 96, 4] 384
|
||
├─GRUModel: 1-2 [512, 13] --
|
||
│ └─GRU: 2-2 [512, 96, 512] 3,949,056
|
||
│ └─Linear: 2-3 [512, 13] 6,669
|
||
==========================================================================================
|
||
Total params: 3,956,109
|
||
Trainable params: 3,956,109
|
||
Non-trainable params: 0
|
||
Total mult-adds (G): 194.11
|
||
==========================================================================================
|
||
Input size (MB): 0.39
|
||
Forward/backward pass size (MB): 202.95
|
||
Params size (MB): 15.82
|
||
Estimated Total Size (MB): 219.17
|
||
==========================================================================================
|
||
```
|
||
|
||
| Experiment | Quarter | DoW | Load | Wind | Net | test_L1Loss | test_CRPSLoss |
|
||
|---|---|---|---|---|---|---|---|
|
||
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/21ea327daf4b48a7967376743d6054ad/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | True | True | 104.26072751090527 | 66.05181515234297 |
|
||
|
||
|
||
#### Quantiles Histogram Debug Plots
|
||
|
||
<p float="middle" width=100%>
|
||
<img src="december_images/Quantile_Performance_Comparison_Training_Set_00000013.jpeg" width="49%" />
|
||
<img src="december_images/Quantile_Performance_Comparison_Test_Set_00000013.jpeg" width="49%" />
|
||
</p>
|
||
|
||
### Results for now
|
||
| Model | test_L1Loss | test_CRPSLoss |
|
||
|---|---|---|
|
||
| Linear Model | 101.63938813784577 | 68.4846434120529 |
|
||
| Non Linear Model | 102.03053663410438 | 68.968302991015 |
|
||
| LSTM/GRU Model | 104.26072751090527 | 66.05181515234297 |
|
||
|
||
|
||
|
||
# Non Auto-Regressive Models
|
||
|
||
| Model | Experiment | test_L1Loss | test_CRPSLoss |
|
||
|---|---|---| ---|
|
||
| Linear Model | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/8f00fa7713f94d2ca376ce0eabc2742f/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | 104.89022124436754 | 69.04472427024562 |
|
||
| Non Linear Model | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/f2ed5c3b54cd4d859d72bc55e53ff646/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | 103.250159162178 | 74.21675055952228 |
|