VictorMylle/Thesis
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fixed losses and multiple other things

Browse Source
This commit is contained in:
Victor Mylle
2023-12-12 11:02:42 +00:00
parent d3bf04d68c
commit c06cc10aa6
12 changed files with 5093936 additions and 122 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
Result-Reports/December.md
View File

@@ -1,3 +1,35 @@
# Tasks
- [ ] Quantiles zelf breder maken na fitten, literatuur bekijken (overconfident voor ondergrens) (Validation set januari 2023)
Eerst literatuur bekijken ofdat probleem al voorkomt
- [ ] compare reconstructed prices with real imbalance prices (on figure and metrics on whole test set)
- [ ] Baseline policy (Eche imbalanceprijzen):
- Batterij:
- 2 MWh
- 1 MW power (charging / discharging)
- Training data -> 2 tresholds (1 voor charging, 1 voor discharging) (Volledig chargen / dischargen)
- 2 Mwh -> 8 kwartieren
- 1000 generaties voor 1 dag -> 1000 prijzen reconstrueren -> tresholds bepalen (logische tresholds) -> 2 tresholds voor 1 dag en toepassen op volledige test set
-> resultaten: # charge cycle
- [ ] Learning rate non autoregressive
- [x] !!! Historgram quantile plot volledige dag
- [x] CRPS en MAE, over 96 wanneer wordt het slecht
- [x] crps loss fixen
- [x] More input parameters
- [x] Non autoregressive (baseline for linear)
(
- NRV to prices (bid ladder from yesterday) -> Elia
- Simple policy
)
- [ ] Diffusion vs Variable Selection Network
# Different Model Architectures (For Quantile Regression)
## Baseline
The baseline just calculates the values for the given quantiles using the training data.
@@ -10,7 +42,7 @@ The baseline just calculates the values for the given quantiles using the traini
| train_CRPSLoss | test_CRPSLoss |
|---|---|
| 74.1899447775193 | 79.26462867583763 |
| 70.70666326392862 | 76.03928650729421 |
# Auto Regressive Models
### Linear Model
@@ -37,12 +69,11 @@ Estimated Total Size (MB): 0.93
==========================================================================================
```
| Experiment | Quarter | Load forecast | Load History | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/c7a0f30439ba4ef5bac28cc8337318ce/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | False |False |False | 105.62005737808495 | 78.6946345109206 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/a35aa8e60ef94999af909134d2285afc/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | False | False | 104.97209199411934 | 78.15958404541016 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/c67a9b2c2c6f42278dbca527dcc283b0/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | False | 104.98653461048444 | 78.18278430058406 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/0f1f2bec9bc94beca9b749d5f1708190/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | 104.82491272720578 | 77.90755403958835 |
| Experiment | Quarter | Day Of Week | Load | Wind | Net  | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/42f3795e5863464b8aa25bcdac75cf08/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | True | True | 101.63938813784577 | 68.4846434120529 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/848565e544e34748bf80ac7eca379a50/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | False | False | 105.50163727880451 | 73.50646092092921 |
#### Quantiles Histogram Debug Plots
@@ -88,12 +119,12 @@ Estimated Total Size (MB): 10.16
==========================================================================================
```
| Experiment | Quarter | Load forecast | Load History | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/b15e452a19d941cdb6a59562e42765c7/hyper-params/configuration/model?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | False | False | False | 105.75275872112196 | 79.5905984731821 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/57834a557d104c078245e09506270fc2/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | False | False | 104.9115321283131 | 78.9574656853309 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/6e0916d84fa94a74874add62fbba3c92/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | False | 104.05637291829032 | 78.49674870417668 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/1182d39a984b478c9301aafb4a81ff1b/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | 103.89383283348461 | 77.7099763430082 |
| Experiment | Quarter | Day Of Week | Load | Wind | Net  | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/6b50442e1cec4bf9b3bd5a34077b4217/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | False | False | False | 103.6061492655297 | 67.56606704206783 | 
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/9d30a4c6d63d44d18dff2496d1ec67f1/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | False | False | 103.6867771851146 | 68.6250203464861 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/45aa03c6e9f447859b1f8915576002c4/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | True | True | 102.03053663410438 | 68.968302991015 |
#### Quantiles Histogram Debug Plots
@@ -142,11 +173,10 @@ Estimated Total Size (MB): 219.17
==========================================================================================
```
| Experiment | Quarter | Load forecast | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/da13772831694537a8a676f873f0577b/info-output/metrics/scalar) | False | False | 104.91248365620233 | 80.52249167947208 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/1c3bea2d3ce2494498fd2f188c23ae93/info-output/metrics/scalar) | True | False | 104.01024075423138 | 79.42769390928979 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/38ee177cdd4741dbb9668c8902b03acc/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | 103.57896084611653 | 79.2824327805463 |
| Experiment | Quarter | DoW | Load | Wind | Net | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/21ea327daf4b48a7967376743d6054ad/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | True | True | 104.26072751090527 | 66.05181515234297 |
#### Quantiles Histogram Debug Plots
@@ -158,22 +188,15 @@ Estimated Total Size (MB): 219.17
### Results for now
| Model | test_L1Loss | test_CRPSLoss |
|---|---|---|
| Linear Model | 104.82491272720578 | 77.90755403958835 |
| Non Linear Model | 103.89383283348461 | 77.7099763430082 |
| LSTM/GRU Model | 103.57896084611653 | 79.2824327805463 |
| Linear Model | 101.63938813784577 | 68.4846434120529 |
| Non Linear Model | 102.03053663410438 | 68.968302991015 |
| LSTM/GRU Model | 104.26072751090527 | 66.05181515234297 |
# Tasks
- [ ] Quantiles zelf breder maken na fitten, literatuur bekijken (overconfident voor ondergrens)
- [x] !!! Historgram quantile plot volledige dag
- [ ] CRPS en MAE, over 96 wanneer wordt het slecht
- [x] crps loss fixen
- [ ] More input parameters
- [ ] Non autoregressive (baseline for linear)
(
- NRV to prices (bid ladder from yesterday) -> Elia
- Simple policy
)
# Non Auto-Regressive Models
- [ ] Diffusion vs Variable Selection Network
| Model | Experiment | test_L1Loss | test_CRPSLoss |
|---|---|---| ---|
| Linear Model | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/8f00fa7713f94d2ca376ce0eabc2742f/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | 104.89022124436754 | 69.04472427024562 |
| Non Linear Model | [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/f2ed5c3b54cd4d859d72bc55e53ff646/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | 103.250159162178 | 74.21675055952228 |

2
Result-Reports/November_1.md
View File

@@ -20,7 +20,7 @@
- [x] CPRS Metrics
- [x] Time as input parameter:
- [x] Probabilistic Baseline -> Quantiles on Training Data -> Breedte bekijken -> Gebruiken voor CPRS en plotjes
- [ ] Day-ahead implicit net position
- [x] Day-ahead implicit net position
- [x] Faster sampling for quantile regression
- [x] Quantile plots for other model (Linear, GRU) (Check if better)