VictorMylle/Thesis
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fixed small summary with model architectures until now

Browse Source
This commit is contained in:
Victor Mylle
2023-11-30 21:53:35 +00:00
parent eba10c8f83
commit 120b6aa5bd
23 changed files with 402 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files

163
Result-Reports/December.md Normal file
View File

@@ -0,0 +1,163 @@
# Different Model Architectures (For Quantile Regression)
## Baseline
The baseline just calculates the values for the given quantiles using the training data.
![Mean NRV](december_images/mean_nrv.png)
*Mean NRV for whole day*
![Predicted quantiles](december_images/probabilistic_baseline_quantiles.png)
| train_CRPSLoss | test_CRPSLoss |
|---|---|
| 74.1899447775193 | 79.26462867583763 |
# Auto Regressive Models
### Linear Model
#### Example summary of the Linear Model
```
==========================================================================================
Layer (type:depth-idx) Output Shape Param #
==========================================================================================
Sequential [1024, 13] --
├─TimeEmbedding: 1-1 [1024, 195] --
│ └─Embedding: 2-1 [1024, 2] 192
├─LinearRegression: 1-2 [1024, 13] --
│ └─Linear: 2-2 [1024, 13] 2,548
==========================================================================================
Total params: 2,740
Trainable params: 2,740
Non-trainable params: 0
Total mult-adds (M): 2.81
==========================================================================================
Input size (MB): 0.79
Forward/backward pass size (MB): 0.12
Params size (MB): 0.01
Estimated Total Size (MB): 0.93
==========================================================================================
```
| Experiment | Quarter | Load forecast | Load History | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/c7a0f30439ba4ef5bac28cc8337318ce/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | False |False |False | 105.62005737808495 | 78.6946345109206 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/a35aa8e60ef94999af909134d2285afc/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | False | False | 104.97209199411934 | 78.15958404541016 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/c67a9b2c2c6f42278dbca527dcc283b0/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | False | 104.98653461048444 | 78.18278430058406 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/0f1f2bec9bc94beca9b749d5f1708190/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | 104.82491272720578 | 77.90755403958835 |
#### Quantiles Histogram Debug Plots
<p float="middle" width=100%>
<img src="december_images/Quantile_Performance_Comparison_Training_Set_00000014.jpeg" width="49%" />
<img src="december_images/Quantile_Performance_Comparison_Test_Set_00000014.jpeg" width="49%" />
</p>
### Non Linear Model
#### Example summary of the Non Linear Model
```
==========================================================================================
Layer (type:depth-idx) Output Shape Param #
==========================================================================================
Sequential [1024, 13] --
├─TimeEmbedding: 1-1 [1024, 96] --
├─NonLinearRegression: 1-2 [1024, 13] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Linear: 3-1 [1024, 512] 49,664
│ └─ReLU: 2-2 [1024, 512] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Dropout: 3-2 [1024, 512] --
│ └─ReLU: 2-4 [1024, 512] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Linear: 3-3 [1024, 512] 262,656
│ └─ReLU: 2-6 [1024, 512] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Dropout: 3-4 [1024, 512] --
│ └─ReLU: 2-8 [1024, 512] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Linear: 3-5 [1024, 13] 6,669
==========================================================================================
Total params: 318,989
Trainable params: 318,989
Non-trainable params: 0
Total mult-adds (M): 326.64
==========================================================================================
Input size (MB): 0.39
Forward/backward pass size (MB): 8.50
Params size (MB): 1.28
Estimated Total Size (MB): 10.16
==========================================================================================
```
| Experiment | Quarter | Load forecast | Load History | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/b15e452a19d941cdb6a59562e42765c7/hyper-params/configuration/model?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | False | False | False | 105.75275872112196 | 79.5905984731821 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/57834a557d104c078245e09506270fc2/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | False | False | 104.9115321283131 | 78.9574656853309 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/6e0916d84fa94a74874add62fbba3c92/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | False | 104.05637291829032 | 78.49674870417668 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/1182d39a984b478c9301aafb4a81ff1b/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | 103.89383283348461 | 77.7099763430082 |
#### Quantiles Histogram Debug Plots
<p float="middle" width=100%>
<img src="december_images/Quantile_Performance_Comparison_Training Set_00000009.jpeg" width="49%" />
<img src="december_images/Quantile_Performance_Comparison_Test Set_00000009.jpeg" width="49%" />
</p>
### LSTM/GRU Model
Input shape: (batch_size, sequence_length, input_features) \
If load forecast is used as input, each time step gets the load forecast for the next time step as feature.
Example:
We have time steps:
| | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| NRV | 0.1 | 0.2 | 0.15 | 0.3 | 0.4 |
| Load forecast | 0.4 | 0.23 | 0.48 | 0.2 | 0.1 |
If we want to predict the NRV for time step 5 using the information we have beforehand, we can use the NRV from the previous time steps. We can however also use the load forecast of time step 5. To incorporate this information as input, we need to move the load forecast one time step back. This means, that the input for time step 5 is given with the NRV of time step 4.
If the time is also wanted as input, we add this as a feature for every timestep aswell.
#### Example summary of the LSTM/GRU Model
```
==========================================================================================
Layer (type:depth-idx) Output Shape Param #
==========================================================================================
Sequential [512, 13] --
├─TimeEmbedding: 1-1 [512, 96, 5] --
│ └─Embedding: 2-1 [512, 96, 4] 384
├─GRUModel: 1-2 [512, 13] --
│ └─GRU: 2-2 [512, 96, 512] 3,949,056
│ └─Linear: 2-3 [512, 13] 6,669
==========================================================================================
Total params: 3,956,109
Trainable params: 3,956,109
Non-trainable params: 0
Total mult-adds (G): 194.11
==========================================================================================
Input size (MB): 0.39
Forward/backward pass size (MB): 202.95
Params size (MB): 15.82
Estimated Total Size (MB): 219.17
==========================================================================================
```
| Experiment | Quarter | Load forecast | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/da13772831694537a8a676f873f0577b/info-output/metrics/scalar) | False | False | 104.91248365620233 | 80.52249167947208 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/1c3bea2d3ce2494498fd2f188c23ae93/info-output/metrics/scalar) | True | False | 104.01024075423138 | 79.42769390928979 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/38ee177cdd4741dbb9668c8902b03acc/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | 103.57896084611653 | 79.2824327805463 |
#### Quantiles Histogram Debug Plots
<p float="middle" width=100%>
<img src="december_images/Quantile_Performance_Comparison_Training_Set_00000013.jpeg" width="49%" />
<img src="december_images/Quantile_Performance_Comparison_Test_Set_00000013.jpeg" width="49%" />
</p>
### Results for now
| Model | test_L1Loss | test_CRPSLoss |
|---|---|---|
| Linear Model | 104.82491272720578 | 77.90755403958835 |
| Non Linear Model | 103.89383283348461 | 77.7099763430082 |
| LSTM/GRU Model | 103.57896084611653 | 79.2824327805463 |