VictorMylle/Thesis
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fixed small summary with model architectures until now

Browse Source
This commit is contained in:
Victor Mylle
2023-11-30 21:53:35 +00:00
parent eba10c8f83
commit 120b6aa5bd
23 changed files with 402 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files

163
Result-Reports/December.md Normal file
View File

@@ -0,0 +1,163 @@
# Different Model Architectures (For Quantile Regression)
## Baseline
The baseline just calculates the values for the given quantiles using the training data.
![Mean NRV](december_images/mean_nrv.png)
*Mean NRV for whole day*
![Predicted quantiles](december_images/probabilistic_baseline_quantiles.png)
| train_CRPSLoss | test_CRPSLoss |
|---|---|
| 74.1899447775193 | 79.26462867583763 |
# Auto Regressive Models
### Linear Model
#### Example summary of the Linear Model
```
==========================================================================================
Layer (type:depth-idx) Output Shape Param #
==========================================================================================
Sequential [1024, 13] --
├─TimeEmbedding: 1-1 [1024, 195] --
│ └─Embedding: 2-1 [1024, 2] 192
├─LinearRegression: 1-2 [1024, 13] --
│ └─Linear: 2-2 [1024, 13] 2,548
==========================================================================================
Total params: 2,740
Trainable params: 2,740
Non-trainable params: 0
Total mult-adds (M): 2.81
==========================================================================================
Input size (MB): 0.79
Forward/backward pass size (MB): 0.12
Params size (MB): 0.01
Estimated Total Size (MB): 0.93
==========================================================================================
```
| Experiment | Quarter | Load forecast | Load History | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/c7a0f30439ba4ef5bac28cc8337318ce/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | False |False |False | 105.62005737808495 | 78.6946345109206 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/a35aa8e60ef94999af909134d2285afc/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | False | False | 104.97209199411934 | 78.15958404541016 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/c67a9b2c2c6f42278dbca527dcc283b0/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | False | 104.98653461048444 | 78.18278430058406 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/0f1f2bec9bc94beca9b749d5f1708190/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | 104.82491272720578 | 77.90755403958835 |
#### Quantiles Histogram Debug Plots
<p float="middle" width=100%>
<img src="december_images/Quantile_Performance_Comparison_Training_Set_00000014.jpeg" width="49%" />
<img src="december_images/Quantile_Performance_Comparison_Test_Set_00000014.jpeg" width="49%" />
</p>
### Non Linear Model
#### Example summary of the Non Linear Model
```
==========================================================================================
Layer (type:depth-idx) Output Shape Param #
==========================================================================================
Sequential [1024, 13] --
├─TimeEmbedding: 1-1 [1024, 96] --
├─NonLinearRegression: 1-2 [1024, 13] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Linear: 3-1 [1024, 512] 49,664
│ └─ReLU: 2-2 [1024, 512] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Dropout: 3-2 [1024, 512] --
│ └─ReLU: 2-4 [1024, 512] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Linear: 3-3 [1024, 512] 262,656
│ └─ReLU: 2-6 [1024, 512] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Dropout: 3-4 [1024, 512] --
│ └─ReLU: 2-8 [1024, 512] --
│ └─ModuleList: 2-9 -- (recursive)
│ │ └─Linear: 3-5 [1024, 13] 6,669
==========================================================================================
Total params: 318,989
Trainable params: 318,989
Non-trainable params: 0
Total mult-adds (M): 326.64
==========================================================================================
Input size (MB): 0.39
Forward/backward pass size (MB): 8.50
Params size (MB): 1.28
Estimated Total Size (MB): 10.16
==========================================================================================
```
| Experiment | Quarter | Load forecast | Load History | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/b15e452a19d941cdb6a59562e42765c7/hyper-params/configuration/model?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | False | False | False | 105.75275872112196 | 79.5905984731821 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/57834a557d104c078245e09506270fc2/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | False | False | 104.9115321283131 | 78.9574656853309 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/6e0916d84fa94a74874add62fbba3c92/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | False | 104.05637291829032 | 78.49674870417668 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/1182d39a984b478c9301aafb4a81ff1b/execution?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | True | 103.89383283348461 | 77.7099763430082 |
#### Quantiles Histogram Debug Plots
<p float="middle" width=100%>
<img src="december_images/Quantile_Performance_Comparison_Training Set_00000009.jpeg" width="49%" />
<img src="december_images/Quantile_Performance_Comparison_Test Set_00000009.jpeg" width="49%" />
</p>
### LSTM/GRU Model
Input shape: (batch_size, sequence_length, input_features) \
If load forecast is used as input, each time step gets the load forecast for the next time step as feature.
Example:
We have time steps:
| | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| NRV | 0.1 | 0.2 | 0.15 | 0.3 | 0.4 |
| Load forecast | 0.4 | 0.23 | 0.48 | 0.2 | 0.1 |
If we want to predict the NRV for time step 5 using the information we have beforehand, we can use the NRV from the previous time steps. We can however also use the load forecast of time step 5. To incorporate this information as input, we need to move the load forecast one time step back. This means, that the input for time step 5 is given with the NRV of time step 4.
If the time is also wanted as input, we add this as a feature for every timestep aswell.
#### Example summary of the LSTM/GRU Model
```
==========================================================================================
Layer (type:depth-idx) Output Shape Param #
==========================================================================================
Sequential [512, 13] --
├─TimeEmbedding: 1-1 [512, 96, 5] --
│ └─Embedding: 2-1 [512, 96, 4] 384
├─GRUModel: 1-2 [512, 13] --
│ └─GRU: 2-2 [512, 96, 512] 3,949,056
│ └─Linear: 2-3 [512, 13] 6,669
==========================================================================================
Total params: 3,956,109
Trainable params: 3,956,109
Non-trainable params: 0
Total mult-adds (G): 194.11
==========================================================================================
Input size (MB): 0.39
Forward/backward pass size (MB): 202.95
Params size (MB): 15.82
Estimated Total Size (MB): 219.17
==========================================================================================
```
| Experiment | Quarter | Load forecast | test_L1Loss | test_CRPSLoss |
|---|---|---|---|---|
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/da13772831694537a8a676f873f0577b/info-output/metrics/scalar) | False | False | 104.91248365620233 | 80.52249167947208 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/1c3bea2d3ce2494498fd2f188c23ae93/info-output/metrics/scalar) | True | False | 104.01024075423138 | 79.42769390928979 |
| [Link](https://clearml.victormylle.be/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/38ee177cdd4741dbb9668c8902b03acc/info-output/metrics/scalar?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&columns=m.1a899a19b54957e02a21c3a1d82577ad.0970ca62a85af2722008c5220e9d8a9e.value.Summary%2Ftest_CRPSLoss.lastreported&columns=m.293da6b015ca6a65992dcf7a53fa0237.098f6bcd4621d373cade4e832627b4f6.min_value.PinballLoss.test&order=-last_update&filter=) | True | True | 103.57896084611653 | 79.2824327805463 |
#### Quantiles Histogram Debug Plots
<p float="middle" width=100%>
<img src="december_images/Quantile_Performance_Comparison_Training_Set_00000013.jpeg" width="49%" />
<img src="december_images/Quantile_Performance_Comparison_Test_Set_00000013.jpeg" width="49%" />
</p>
### Results for now
| Model | test_L1Loss | test_CRPSLoss |
|---|---|---|
| Linear Model | 104.82491272720578 | 77.90755403958835 |
| Non Linear Model | 103.89383283348461 | 77.7099763430082 |
| LSTM/GRU Model | 103.57896084611653 | 79.2824327805463 |

5
Result-Reports/November_1.md
View File

@@ -12,7 +12,7 @@
- [x] Quantile Regression nakijken - [x] Quantile Regression nakijken
- [x] Test scores voor 96 values - [x] Test scores voor 96 values
- [ ] (Optional) Andere modellen (LSTM?) - [x] Andere modellen (LSTM?)
- [x] Non autoregressive Quantile Regression - [x] Non autoregressive Quantile Regression
- [x] Fix debug plots for quantile regression -> predict quantiles and look if true value is below a quantile, if so 1 else 0 and average these over all samples - [x] Fix debug plots for quantile regression -> predict quantiles and look if true value is below a quantile, if so 1 else 0 and average these over all samples
@@ -23,11 +23,12 @@
- [ ] Day-ahead implicit net position - [ ] Day-ahead implicit net position
- [x] Faster sampling for quantile regression - [x] Faster sampling for quantile regression
- [ ] Quantile plots for other model (Linear, GRU) (Check if better) - [x] Quantile plots for other model (Linear, GRU) (Check if better)
- [ ] Check example plots to see if metrics correspond with what seen on plots - [ ] Check example plots to see if metrics correspond with what seen on plots
- [x] Time step (96 values) to embedding layer - [x] Time step (96 values) to embedding layer
- [x] Mean of nrv per time step plotten (done for probabilistic baseline) - [x] Mean of nrv per time step plotten (done for probabilistic baseline)
- [x] Convert back to MW on plots - [x] Convert back to MW on plots
- [x] Background model training
## 2. Autoregressive vs Non-Autoregressive ## 2. Autoregressive vs Non-Autoregressive

BIN
Result-Reports/december_images/Quantile_Performance_Comparison_Test Set_00000009.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 78 KiB

BIN
Result-Reports/december_images/Quantile_Performance_Comparison_Test_Set_00000013.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 77 KiB

BIN
Result-Reports/december_images/Quantile_Performance_Comparison_Test_Set_00000014.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 78 KiB

BIN
Result-Reports/december_images/Quantile_Performance_Comparison_Training Set_00000009.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 79 KiB

BIN
Result-Reports/december_images/Quantile_Performance_Comparison_Training_Set_00000013.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 76 KiB

BIN
Result-Reports/december_images/Quantile_Performance_Comparison_Training_Set_00000014.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 78 KiB

BIN
Result-Reports/december_images/mean_nrv.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 72 KiB

BIN
Result-Reports/december_images/probabilistic_baseline_quantiles.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 184 KiB

1
requirements.txt
View File

@@ -10,3 +10,4 @@ prettytable
clearml clearml
properscoring properscoring
nbconvert nbconvert
torchinfo

101
src/data/dataset.py
View File

@@ -15,19 +15,18 @@ class NrvDataset(Dataset):
lstm: bool = False, lstm: bool = False,
): ):
self.data_config = data_config self.data_config = data_config
self.dataframe = dataframe
self.full_day_skip = full_day_skip self.full_day_skip = full_day_skip
self.lstm = lstm self.lstm = lstm
# reset dataframe index # reset dataframe index
self.dataframe.reset_index(drop=True, inplace=True) dataframe.reset_index(drop=True, inplace=True)
self.sequence_length = sequence_length self.sequence_length = sequence_length
self.predict_sequence_length = predict_sequence_length self.predict_sequence_length = predict_sequence_length
self.samples_to_skip = self.skip_samples() self.samples_to_skip = self.skip_samples(dataframe=dataframe)
total_indices = set( total_indices = set(
range(len(self.dataframe) - self.sequence_length - self.predict_sequence_length) range(len(dataframe) - self.sequence_length - self.predict_sequence_length)
) )
self.valid_indices = sorted(list(total_indices - set(self.samples_to_skip))) self.valid_indices = sorted(list(total_indices - set(self.samples_to_skip)))
@@ -48,20 +47,28 @@ class NrvDataset(Dataset):
self.forecast_features.append("wind_gen_forecast") self.forecast_features.append("wind_gen_forecast")
# add time feature to dataframe # add time feature to dataframe
time_feature = np.array([0] * len(self.dataframe)) time_feature = np.array([0] * len(dataframe))
if self.data_config.QUARTER: if self.data_config.QUARTER:
time_feature += self.dataframe["quarter"] time_feature += dataframe["quarter"]
if self.data_config.DAY_OF_WEEK: if self.data_config.DAY_OF_WEEK:
d_w = self.dataframe["day_of_week"] d_w = dataframe["day_of_week"]
if self.data_config.QUARTER: if self.data_config.QUARTER:
d_w *= 96 d_w *= 96
time_feature += d_w time_feature += d_w
self.dataframe["time_feature"] = time_feature # if not all zero:
if time_feature.any():
self.time_feature = torch.tensor(time_feature).float().reshape(-1)
else:
self.time_feature = None
def skip_samples(self): self.nrv = torch.tensor(dataframe["nrv"].values).float().reshape(-1)
nan_rows = self.dataframe[self.dataframe.isnull().any(axis=1)]
self.history_features, self.forecast_features = self.preprocess_data(dataframe)
def skip_samples(self, dataframe):
nan_rows = dataframe[dataframe.isnull().any(axis=1)]
nan_indices = nan_rows.index nan_indices = nan_rows.index
skip_indices = [ skip_indices = [
list( list(
@@ -79,18 +86,22 @@ class NrvDataset(Dataset):
# add indices that are not the start of a day (00:15) to the skip indices (use datetime column) # add indices that are not the start of a day (00:15) to the skip indices (use datetime column)
# get indices of all 00:15 timestamps # get indices of all 00:15 timestamps
if self.full_day_skip: if self.full_day_skip:
start_of_day_indices = self.dataframe[ start_of_day_indices = dataframe[
self.dataframe["datetime"].dt.time != pd.Timestamp("00:15:00").time() dataframe["datetime"].dt.time != pd.Timestamp("00:15:00").time()
].index ].index
skip_indices.extend(start_of_day_indices) skip_indices.extend(start_of_day_indices)
skip_indices = list(set(skip_indices)) skip_indices = list(set(skip_indices))
return skip_indices return skip_indices
def preprocess_data(self, dataframe):
return torch.tensor(dataframe[self.history_features].values).float(), torch.tensor(dataframe[self.forecast_features].values).float()
def __len__(self): def __len__(self):
return len(self.valid_indices) return len(self.valid_indices)
def _get__all_data(self, idx: int): def _get_all_data(self, idx: int):
history_df = self.dataframe.iloc[idx : idx + self.sequence_length] history_df = self.dataframe.iloc[idx : idx + self.sequence_length]
forecast_df = self.dataframe.iloc[ forecast_df = self.dataframe.iloc[
idx + self.sequence_length : idx + self.sequence_length + self.predict_sequence_length idx + self.sequence_length : idx + self.sequence_length + self.predict_sequence_length
@@ -100,38 +111,46 @@ class NrvDataset(Dataset):
def __getitem__(self, idx): def __getitem__(self, idx):
actual_idx = self.valid_indices[idx] actual_idx = self.valid_indices[idx]
history_df, forecast_df = self._get__all_data(actual_idx)
# get nrv history features # get nrv history features
nrv_features = torch.tensor(history_df[["nrv"]].values).reshape(-1) nrv_features = self.nrv[actual_idx : actual_idx + self.sequence_length]
# get history featues history_features = self.history_features[actual_idx : actual_idx + self.sequence_length, :]
history_features = history_df[self.history_features].values forecast_features = self.forecast_features[actual_idx + self.sequence_length : actual_idx + self.sequence_length + self.predict_sequence_length, :]
# combine the history features to one tensor (first one feature, then the next one, etc.) if self.time_feature is not None:
history_features = torch.tensor(history_features) time_features = self.time_feature[actual_idx : actual_idx + self.sequence_length]
else:
# get forecast features time_features = None
forecast_features = forecast_df[self.forecast_features].values
forecast_features = torch.tensor(forecast_features)
# add last time feature of the history
time_feature = history_df["time_feature"].iloc[-1]
## all features ## all features
if not self.lstm: if not self.lstm:
all_features = torch.cat( all_features_list = [nrv_features]
[nrv_features, history_features.reshape(-1), forecast_features.reshape(-1), torch.tensor([time_feature])], dim=0
) if history_features.numel() > 0:
all_features_list.append(history_features.reshape(-1))
if forecast_features.numel() > 0:
all_features_list.append(forecast_features.reshape(-1))
if time_features is not None:
all_features_list.append(torch.tensor([time_features[-1]]))
all_features = torch.cat(all_features_list, dim=0)
else: else:
time_features = torch.tensor(history_df["time_feature"].values).reshape(-1, 1) all_features_list = [nrv_features.unsqueeze(1)]
# combine (96, ) and (96, 2) to (96, 3)
all_features = torch.cat( if self.forecast_features.numel() > 0:
[nrv_features.unsqueeze(1), time_features], dim=1 history_forecast_features = self.forecast_features[actual_idx + 1 : actual_idx + self.sequence_length + 1, :]
) all_features_list.append(history_forecast_features)
if time_features is not None:
all_features_list.append(time_features.unsqueeze(-1))
all_features = torch.cat(all_features_list, dim=1)
# Target sequence, flattened if necessary # Target sequence, flattened if necessary
nrv_target = forecast_df["nrv"].values nrv_target = self.nrv[actual_idx + self.sequence_length : actual_idx + self.sequence_length + self.predict_sequence_length]
# check if nan values are present # check if nan values are present
if torch.isnan(all_features).any(): if torch.isnan(all_features).any():
@@ -141,16 +160,18 @@ class NrvDataset(Dataset):
# all features and target to float # all features and target to float
all_features = all_features.float() all_features = all_features.float()
# to tens&éazzaéaz"ezéors
nrv_target = torch.tensor(nrv_target).float()
return all_features, nrv_target, idx return all_features, nrv_target, idx
def random_day_autoregressive(self, idx: int): def random_day_autoregressive(self, idx: int):
all_features, nrv_target, _ = self.__getitem__(idx) all_features, nrv_target, _ = self.__getitem__(idx)
# remove the first 96 values of the features (the nrv history) # remove the first 96 values of the features (the nrv history)
all_features = all_features[self.sequence_length :] if not self.lstm:
all_features = all_features[self.sequence_length :]
else:
# last time step
all_features = all_features[-1, :]
all_features = all_features.unsqueeze(0)
return all_features, nrv_target return all_features, nrv_target

8
src/models/linear_regression.py
View File

@@ -1,11 +1,17 @@
import torch import torch
import numpy as np
class LinearRegression(torch.nn.Module): class LinearRegression(torch.nn.Module):
def __init__(self, inputSize, output_size): def __init__(self, inputSize, output_size):
super(LinearRegression, self).__init__() super(LinearRegression, self).__init__()
self.inputSize = inputSize self.inputSize = inputSize
self.output_size = output_size self.output_size = output_size
self.linear = torch.nn.Linear(inputSize, output_size)
# dimension multiplication without first one
dim = inputSize[1:]
dim = [int(x) for x in dim]
dim = np.prod(dim)
self.linear = torch.nn.Linear(dim, output_size)
def forward(self, x): def forward(self, x):
x = torch.squeeze(x, -1) x = torch.squeeze(x, -1)

6
src/models/lstm_model.py
View File

@@ -37,9 +37,9 @@ class GRUModel(torch.nn.Module):
def forward(self, x): def forward(self, x):
# Forward pass through the GRU layers # Forward pass through the GRU layers
_, hidden_state = self.gru(x) x, _ = self.gru(x)
x = x[:, -1, :]
# Use the hidden state from the last time step for the output # Use the hidden state from the last time step for the output
output = self.linear(hidden_state[-1]) output = self.linear(x)
return output return output

2
src/models/non_linear_regression.py
View File

@@ -13,7 +13,7 @@ class NonLinearRegression(torch.nn.Module):
# add linear layers with relu # add linear layers with relu
self.layers = torch.nn.ModuleList() self.layers = torch.nn.ModuleList()
self.layers.append(torch.nn.Linear(inputSize, hiddenSize)) self.layers.append(torch.nn.Linear(inputSize[-1], hiddenSize))
self.layers.append(torch.nn.Dropout(dropout)) self.layers.append(torch.nn.Dropout(dropout))
for _ in range(numLayers - 2): for _ in range(numLayers - 2):
self.layers.append(torch.nn.Linear(hiddenSize, hiddenSize)) self.layers.append(torch.nn.Linear(hiddenSize, hiddenSize))

4
src/models/time_embedding_layer.py
View File

@@ -10,6 +10,8 @@ class TimeEmbedding(nn.Module):
def forward(self, x): def forward(self, x):
# Extract the last 'time_features' from the input # Extract the last 'time_features' from the input
if self.time_features == 0:
return x
time_feature = x[..., -1] # Use ellipsis to access the last dimension time_feature = x[..., -1] # Use ellipsis to access the last dimension
# convert to int # convert to int
time_feature = time_feature.int() time_feature = time_feature.int()
@@ -20,6 +22,8 @@ class TimeEmbedding(nn.Module):
def output_dim(self, input_dim): def output_dim(self, input_dim):
if self.time_features == 0:
return input_dim
# Create a list from the input dimension # Create a list from the input dimension
input_dim_list = list(input_dim) input_dim_list = list(input_dim)
# Modify the last dimension # Modify the last dimension

152
src/notebooks/training.ipynb
View File

@@ -42,15 +42,15 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 3, "execution_count": 5,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"#### Data Processor ####\n", "#### Data Processor ####\n",
"data_config = DataConfig()\n", "data_config = DataConfig()\n",
"data_config.NRV_HISTORY = True\n", "data_config.NRV_HISTORY = True\n",
"data_config.LOAD_HISTORY = False\n", "data_config.LOAD_HISTORY = True\n",
"data_config.LOAD_FORECAST = False\n", "data_config.LOAD_FORECAST = True\n",
"\n", "\n",
"data_config.WIND_FORECAST = False\n", "data_config.WIND_FORECAST = False\n",
"data_config.WIND_HISTORY = False\n", "data_config.WIND_HISTORY = False\n",
@@ -72,16 +72,32 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 11,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
{
"ename": "KeyboardInterrupt",
"evalue": "Interrupted by user",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/workspaces/Thesis/src/notebooks/training.ipynb Cell 5\u001b[0m line \u001b[0;36m7\n\u001b[1;32m <a href='vscode-notebook-cell://dev-container%2B7b22686f737450617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f546865736973222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f5468657369732f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d@ssh-remote%2Bvictormylle.be/workspaces/Thesis/src/notebooks/training.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2'>3</a>\u001b[0m trainer \u001b[39m=\u001b[39m ProbabilisticBaselineTrainer(\n\u001b[1;32m <a href='vscode-notebook-cell://dev-container%2B7b22686f737450617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f546865736973222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f5468657369732f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d@ssh-remote%2Bvictormylle.be/workspaces/Thesis/src/notebooks/training.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3'>4</a>\u001b[0m quantiles\u001b[39m=\u001b[39mquantiles, data_processor\u001b[39m=\u001b[39mdata_processor, clearml_helper\u001b[39m=\u001b[39mclearml_helper\n\u001b[1;32m <a href='vscode-notebook-cell://dev-container%2B7b22686f737450617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f546865736973222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f5468657369732f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d@ssh-remote%2Bvictormylle.be/workspaces/Thesis/src/notebooks/training.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4'>5</a>\u001b[0m )\n\u001b[1;32m <a href='vscode-notebook-cell://dev-container%2B7b22686f737450617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f546865736973222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f5468657369732f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d@ssh-remote%2Bvictormylle.be/workspaces/Thesis/src/notebooks/training.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5'>6</a>\u001b[0m trainer\u001b[39m.\u001b[39madd_metrics_to_track([CRPSLoss()])\n\u001b[0;32m----> <a href='vscode-notebook-cell://dev-container%2B7b22686f737450617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f546865736973222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f766963746f726d796c6c652f53656144726976652f4d79204c69627261726965732f4750552d7365727665722f5468657369732f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d@ssh-remote%2Bvictormylle.be/workspaces/Thesis/src/notebooks/training.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=6'>7</a>\u001b[0m trainer\u001b[39m.\u001b[39;49mtrain()\n",
"File \u001b[0;32m/workspaces/Thesis/src/notebooks/../../src/trainers/probabilistic_baseline.py:43\u001b[0m, in \u001b[0;36mProbabilisticBaselineTrainer.train\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 42\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mtrain\u001b[39m(\u001b[39mself\u001b[39m):\n\u001b[0;32m---> 43\u001b[0m task \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49minit_clearml_task()\n\u001b[1;32m 44\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m 45\u001b[0m time_steps \u001b[39m=\u001b[39m [[] \u001b[39mfor\u001b[39;00m _ \u001b[39min\u001b[39;00m \u001b[39mrange\u001b[39m(\u001b[39m96\u001b[39m)]\n",
"File \u001b[0;32m/workspaces/Thesis/src/notebooks/../../src/trainers/probabilistic_baseline.py:28\u001b[0m, in \u001b[0;36mProbabilisticBaselineTrainer.init_clearml_task\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 25\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mclearml_helper:\n\u001b[1;32m 26\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mNone\u001b[39;00m\n\u001b[0;32m---> 28\u001b[0m task_name \u001b[39m=\u001b[39m \u001b[39minput\u001b[39;49m(\u001b[39m\"\u001b[39;49m\u001b[39mEnter a task name: \u001b[39;49m\u001b[39m\"\u001b[39;49m)\n\u001b[1;32m 29\u001b[0m \u001b[39mif\u001b[39;00m task_name \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39m\"\u001b[39m:\n\u001b[1;32m 30\u001b[0m task_name \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mUntitled Task\u001b[39m\u001b[39m\"\u001b[39m\n",
"File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/ipykernel/kernelbase.py:1251\u001b[0m, in \u001b[0;36mKernel.raw_input\u001b[0;34m(self, prompt)\u001b[0m\n\u001b[1;32m 1249\u001b[0m msg \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mraw_input was called, but this frontend does not support input requests.\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m 1250\u001b[0m \u001b[39mraise\u001b[39;00m StdinNotImplementedError(msg)\n\u001b[0;32m-> 1251\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_input_request(\n\u001b[1;32m 1252\u001b[0m \u001b[39mstr\u001b[39;49m(prompt),\n\u001b[1;32m 1253\u001b[0m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_parent_ident[\u001b[39m\"\u001b[39;49m\u001b[39mshell\u001b[39;49m\u001b[39m\"\u001b[39;49m],\n\u001b[1;32m 1254\u001b[0m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mget_parent(\u001b[39m\"\u001b[39;49m\u001b[39mshell\u001b[39;49m\u001b[39m\"\u001b[39;49m),\n\u001b[1;32m 1255\u001b[0m password\u001b[39m=\u001b[39;49m\u001b[39mFalse\u001b[39;49;00m,\n\u001b[1;32m 1256\u001b[0m )\n",
"File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/ipykernel/kernelbase.py:1295\u001b[0m, in \u001b[0;36mKernel._input_request\u001b[0;34m(self, prompt, ident, parent, password)\u001b[0m\n\u001b[1;32m 1292\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mKeyboardInterrupt\u001b[39;00m:\n\u001b[1;32m 1293\u001b[0m \u001b[39m# re-raise KeyboardInterrupt, to truncate traceback\u001b[39;00m\n\u001b[1;32m 1294\u001b[0m msg \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mInterrupted by user\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m-> 1295\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mKeyboardInterrupt\u001b[39;00m(msg) \u001b[39mfrom\u001b[39;00m \u001b[39mNone\u001b[39;00m\n\u001b[1;32m 1296\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mException\u001b[39;00m:\n\u001b[1;32m 1297\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mlog\u001b[39m.\u001b[39mwarning(\u001b[39m\"\u001b[39m\u001b[39mInvalid Message:\u001b[39m\u001b[39m\"\u001b[39m, exc_info\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m)\n",
"\u001b[0;31mKeyboardInterrupt\u001b[0m: Interrupted by user"
]
}
],
"source": [ "source": [
"data_processor.set_full_day_skip(True)\n", "data_processor.set_full_day_skip(True)\n",
"quantiles = [0.01, 0.05, 0.1, 0.15, 0.4, 0.5, 0.6, 0.85, 0.9, 0.95, 0.99]\n", "quantiles = [0.01, 0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 0.99]\n",
"trainer = ProbabilisticBaselineTrainer(\n", "trainer = ProbabilisticBaselineTrainer(\n",
" quantiles=quantiles, data_processor=data_processor, clearml_helper=clearml_helper\n", " quantiles=quantiles, data_processor=data_processor, clearml_helper=clearml_helper\n",
")\n", ")\n",
"trainer.add_metrics_to_track([CRPSLoss(quantiles=quantiles)])\n", "trainer.add_metrics_to_track([CRPSLoss()])\n",
"trainer.train()" "trainer.train()"
] ]
}, },
@@ -100,7 +116,7 @@
"source": [ "source": [
"#### Hyperparameters ####\n", "#### Hyperparameters ####\n",
"inputDim = data_processor.get_input_size()\n", "inputDim = data_processor.get_input_size()\n",
"learningRate = 0.00005\n", "learningRate = 0.0001\n",
"epochs = 150\n", "epochs = 150\n",
"\n", "\n",
"# model = LinearRegression(inputDim, 96)\n", "# model = LinearRegression(inputDim, 96)\n",
@@ -133,8 +149,8 @@
"learningRate = 0.0003\n", "learningRate = 0.0003\n",
"epochs = 50\n", "epochs = 50\n",
"\n", "\n",
"# model = LinearRegression(inputDim, 1)\n", "model = LinearRegression(inputDim, 1)\n",
"model = NonLinearRegression(inputDim, 1, hiddenSize=1024, numLayers=5)\n", "# model = NonLinearRegression(inputDim, 1, hiddenSize=1024, numLayers=5)\n",
"optimizer = torch.optim.Adam(model.parameters(), lr=learningRate)\n", "optimizer = torch.optim.Adam(model.parameters(), lr=learningRate)\n",
"\n", "\n",
"#### Data Processor ####\n", "#### Data Processor ####\n",
@@ -167,13 +183,15 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 4, "execution_count": 6,
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
"name": "stdout", "name": "stdout",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"ClearML Task: created new task id=1182d39a984b478c9301aafb4a81ff1b\n",
"ClearML results page: http://192.168.1.182:8080/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/1182d39a984b478c9301aafb4a81ff1b/output/log\n",
"96\n" "96\n"
] ]
}, },
@@ -181,72 +199,66 @@
"name": "stderr", "name": "stderr",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"/workspaces/Thesis/src/notebooks/../../src/trainers/quantile_trainer.py:68: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", "Connecting multiple input models with the same name: `checkpoint`. This might result in the wrong model being used when executing remotely\n"
" quantiles_tensor = torch.tensor(quantiles)\n",
"/workspaces/Thesis/src/notebooks/../../src/losses/pinball_loss.py:8: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
" self.quantiles_tensor = torch.tensor(quantiles, dtype=torch.float32)\n",
"InsecureRequestWarning: Certificate verification is disabled! Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n"
] ]
}, },
{ {
"name": "stdout", "name": "stdout",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"ClearML Task: created new task id=4652507a84f5435fb6bd98c645d15f24\n", "Early stopping triggered\n"
"ClearML results page: http://192.168.1.182:8080/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/4652507a84f5435fb6bd98c645d15f24/output/log\n",
"2023-11-26 22:15:47,860 - clearml.Task - INFO - Storing jupyter notebook directly as code\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Switching to remote execution, output log page http://192.168.1.182:8080/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/4652507a84f5435fb6bd98c645d15f24/output/log\n"
]
},
{
"ename": "",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001b[1;31mThe Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click <a href='https://aka.ms/vscodeJupyterKernelCrash'>here</a> for more info. View Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
] ]
} }
], ],
"source": [ "source": [
"task = clearml_helper.get_task(task_name=\"Autoregressive Quantile Regression + Quarter + Load Forecast + Load History\")\n",
"data_config = task.connect(data_config, name=\"data_features\")\n",
"\n",
"#### Hyperparameters ####\n", "#### Hyperparameters ####\n",
"data_processor.set_output_size(1)\n", "data_processor.set_output_size(1)\n",
"inputDim = data_processor.get_input_size()\n", "inputDim = data_processor.get_input_size()\n",
"learningRate = 0.0001\n", "epochs = 300\n",
"epochs = 100\n",
"\n", "\n",
"# quantiles = torch.tensor([0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99]).to(\"cuda\")\n", "quantiles = task.get_parameter(\"general/quantiles\", cast=True)\n",
"quantiles = torch.tensor(\n", "if quantiles is None:\n",
" [0.01, 0.05, 0.1, 0.15, 0.3, 0.4, 0.5, 0.6, 0.7, 0.85, 0.9, 0.95, 0.99]\n", " quantiles = [0.01, 0.05, 0.1, 0.15, 0.3, 0.4, 0.5, 0.6, 0.7, 0.85, 0.9, 0.95, 0.99]\n",
").to(\"cuda\")\n", " task.set_parameter(\"general/quantiles\", quantiles)\n",
"else:\n",
" if isinstance(quantiles, str):\n",
" quantiles = eval(quantiles)\n",
"\n", "\n",
"# model = LinearRegression(inputDim, len(quantiles))\n", "model_parameters = {\n",
"time_embedding = TimeEmbedding(data_processor.get_time_feature_size(), 4)\n", " \"learning_rate\": 0.0001,\n",
"non_linear_regression_model = NonLinearRegression(time_embedding.output_dim(inputDim), len(quantiles), hiddenSize=1024, numLayers=5)\n", " \"hidden_size\": 512,\n",
" \"num_layers\": 3,\n",
" \"dropout\": 0.2,\n",
" \"time_feature_embedding\": 2,\n",
"}\n",
"\n",
"model_parameters = task.connect(model_parameters, name=\"model_parameters\")\n",
"\n",
"time_embedding = TimeEmbedding(data_processor.get_time_feature_size(), model_parameters[\"time_feature_embedding\"])\n",
"# linear_regression = LinearRegression(time_embedding.output_dim(inputDim), len(quantiles))\n",
"non_linear_regression_model = NonLinearRegression(time_embedding.output_dim(inputDim), len(quantiles), hiddenSize=model_parameters[\"hidden_size\"], numLayers=model_parameters[\"num_layers\"], dropout=model_parameters[\"dropout\"])\n",
"model = nn.Sequential(time_embedding, non_linear_regression_model)\n", "model = nn.Sequential(time_embedding, non_linear_regression_model)\n",
"optimizer = torch.optim.Adam(model.parameters(), lr=learningRate)\n", "optimizer = torch.optim.Adam(model.parameters(), lr=model_parameters[\"learning_rate\"])\n",
"\n", "\n",
"#### Trainer ####\n", "#### Trainer ####\n",
"trainer = AutoRegressiveQuantileTrainer(\n", "trainer = AutoRegressiveQuantileTrainer(\n",
" model,\n", " model,\n",
" inputDim,\n",
" optimizer,\n", " optimizer,\n",
" data_processor,\n", " data_processor,\n",
" quantiles,\n", " quantiles,\n",
" \"cuda\",\n", " \"cuda\",\n",
" debug=True,\n", " debug=False,\n",
" clearml_helper=clearml_helper,\n",
")\n", ")\n",
"trainer.add_metrics_to_track(\n", "trainer.add_metrics_to_track(\n",
" [PinballLoss(quantiles), MSELoss(), L1Loss(), CRPSLoss(quantiles)]\n", " [PinballLoss(quantiles), MSELoss(), L1Loss(), CRPSLoss()]\n",
")\n", ")\n",
"trainer.early_stopping(patience=10)\n", "trainer.early_stopping(patience=10)\n",
"trainer.plot_every(5)\n", "trainer.plot_every(5)\n",
"trainer.train(epochs=epochs, remotely=True)" "trainer.train(task=task, epochs=epochs, remotely=False)"
] ]
}, },
{ {
@@ -258,49 +270,9 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 3, "execution_count": null,
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [],
{
"name": "stderr",
"output_type": "stream",
"text": [
"/workspaces/Thesis/src/notebooks/../../src/trainers/quantile_trainer.py:335: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
" quantiles_tensor = torch.tensor(quantiles)\n",
"/workspaces/Thesis/src/notebooks/../../src/losses/pinball_loss.py:22: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
" self.quantiles_tensor = torch.tensor(quantiles, dtype=torch.float32)\n",
"InsecureRequestWarning: Certificate verification is disabled! Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"ClearML Task: created new task id=0c748cf6ec0f4c748cc35be78ae4c6c1\n",
"ClearML results page: http://192.168.1.182:8080/projects/2e46d4af6f1e4c399cf9f5aa30bc8795/experiments/0c748cf6ec0f4c748cc35be78ae4c6c1/output/log\n",
"2023-11-26 16:15:07,490 - clearml.Task - INFO - Storing jupyter notebook directly as code\n",
"2023-11-26 16:15:09,255 - clearml.model - WARNING - 500 model found when searching for `file:///workspaces/Thesis/src/notebooks/checkpoint.pt`\n",
"2023-11-26 16:15:09,256 - clearml.model - WARNING - Selected model `Autoregressive Quantile Regression (quarter + day of week)` (id=bc0cb0d7fc614e2e8b0edf5b85348646)\n",
"2023-11-26 16:15:09,265 - clearml.frameworks - INFO - Found existing registered model id=bc0cb0d7fc614e2e8b0edf5b85348646 [/workspaces/Thesis/src/notebooks/checkpoint.pt] reusing it.\n",
"2023-11-26 16:15:09,958 - clearml.Task - INFO - Completed model upload to http://192.168.1.182:8081/Thesis/NrvForecast/Untitled%20Task.0c748cf6ec0f4c748cc35be78ae4c6c1/models/checkpoint.pt\n",
"2023-11-26 16:15:10,998 - clearml.Task - INFO - Completed model upload to http://192.168.1.182:8081/Thesis/NrvForecast/Untitled%20Task.0c748cf6ec0f4c748cc35be78ae4c6c1/models/checkpoint.pt\n",
"2023-11-26 16:15:12,118 - clearml.Task - INFO - Completed model upload to http://192.168.1.182:8081/Thesis/NrvForecast/Untitled%20Task.0c748cf6ec0f4c748cc35be78ae4c6c1/models/checkpoint.pt\n",
"2023-11-26 16:15:13,152 - clearml.Task - INFO - Completed model upload to http://192.168.1.182:8081/Thesis/NrvForecast/Untitled%20Task.0c748cf6ec0f4c748cc35be78ae4c6c1/models/checkpoint.pt\n",
"2023-11-26 16:15:14,540 - clearml.Task - INFO - Completed model upload to http://192.168.1.182:8081/Thesis/NrvForecast/Untitled%20Task.0c748cf6ec0f4c748cc35be78ae4c6c1/models/checkpoint.pt\n",
"Early stopping triggered\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/workspaces/Thesis/src/notebooks/../../src/trainers/quantile_trainer.py:366: UserWarning:\n",
"\n",
"Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1682343967769/work/torch/csrc/utils/tensor_new.cpp:245.)\n",
"\n"
]
}
],
"source": [ "source": [
"### Data Processor ###\n", "### Data Processor ###\n",
"data_processor.set_full_day_skip(True)\n", "data_processor.set_full_day_skip(True)\n",

2
src/trainers/autoregressive_trainer.py
View File

@@ -15,6 +15,7 @@ class AutoRegressiveTrainer(Trainer):
def __init__( def __init__(
self, self,
model: torch.nn.Module, model: torch.nn.Module,
input_dim: tuple,
optimizer: torch.optim.Optimizer, optimizer: torch.optim.Optimizer,
criterion: torch.nn.Module, criterion: torch.nn.Module,
data_processor: DataProcessor, data_processor: DataProcessor,
@@ -23,6 +24,7 @@ class AutoRegressiveTrainer(Trainer):
): ):
super().__init__( super().__init__(
model=model, model=model,
input_dim=input_dim,
optimizer=optimizer, optimizer=optimizer,
criterion=criterion, criterion=criterion,
data_processor=data_processor, data_processor=data_processor,

10
src/trainers/probabilistic_baseline.py
View File

@@ -48,7 +48,7 @@ class ProbabilisticBaselineTrainer(Trainer):
predict_sequence_length=96 predict_sequence_length=96
) )
for inputs, _ in train_loader: for inputs, _, _ in train_loader:
for i in range(96): for i in range(96):
time_steps[i].extend(inputs[:, i].numpy()) time_steps[i].extend(inputs[:, i].numpy())
@@ -80,7 +80,7 @@ class ProbabilisticBaselineTrainer(Trainer):
raise raise
def log_final_metrics(self, task, dataloader, quantile_values, train: bool = True): def log_final_metrics(self, task, dataloader, quantile_values, train: bool = True):
metric = CRPSLoss(self.quantiles) metric = CRPSLoss()
crps_values = [] crps_values = []
crps_inversed_values = [] crps_inversed_values = []
@@ -147,6 +147,9 @@ class ProbabilisticBaselineTrainer(Trainer):
def plot_quantiles(self, quantile_values): def plot_quantiles(self, quantile_values):
fig = go.Figure() fig = go.Figure()
# inverse transform quantile_values
quantile_values = self.data_processor.inverse_transform(quantile_values)
for i, q in enumerate(self.quantiles): for i, q in enumerate(self.quantiles):
values_for_quantile = quantile_values[:, i] values_for_quantile = quantile_values[:, i]
fig.add_trace( fig.add_trace(
@@ -159,7 +162,8 @@ class ProbabilisticBaselineTrainer(Trainer):
) )
fig.update_layout(title="Quantile Values") fig.update_layout(title="Quantile Values")
fig.update_yaxes(range=[-1, 1])
fig.update_layout(height=600)
return fig return fig

46
src/trainers/quantile_trainer.py
View File

@@ -60,6 +60,7 @@ class AutoRegressiveQuantileTrainer(AutoRegressiveTrainer):
def __init__( def __init__(
self, self,
model: torch.nn.Module, model: torch.nn.Module,
input_dim: tuple,
optimizer: torch.optim.Optimizer, optimizer: torch.optim.Optimizer,
data_processor: DataProcessor, data_processor: DataProcessor,
quantiles: list, quantiles: list,
@@ -72,6 +73,7 @@ class AutoRegressiveQuantileTrainer(AutoRegressiveTrainer):
criterion = PinballLoss(quantiles=quantiles) criterion = PinballLoss(quantiles=quantiles)
super().__init__( super().__init__(
model=model, model=model,
input_dim=input_dim,
optimizer=optimizer, optimizer=optimizer,
criterion=criterion, criterion=criterion,
data_processor=data_processor, data_processor=data_processor,
@@ -192,7 +194,10 @@ class AutoRegressiveQuantileTrainer(AutoRegressiveTrainer):
prev_features = prev_features.to(self.device) prev_features = prev_features.to(self.device)
targets = targets.to(self.device) targets = targets.to(self.device)
initial_sequence = prev_features[:, :96] if len(list(prev_features.shape)) == 2:
initial_sequence = prev_features[:, :96]
else:
initial_sequence = prev_features[:, :, 0]
target_full = targets[:, 0].unsqueeze(1) # (batch_size, 1) target_full = targets[:, 0].unsqueeze(1) # (batch_size, 1)
with torch.no_grad(): with torch.no_grad():
@@ -206,22 +211,37 @@ class AutoRegressiveQuantileTrainer(AutoRegressiveTrainer):
predictions_full = new_predictions_full.unsqueeze(1) predictions_full = new_predictions_full.unsqueeze(1)
for i in range(sequence_length - 1): for i in range(sequence_length - 1):
new_features = torch.cat( if len(list(prev_features.shape)) == 2:
(prev_features[:, 1:96], samples), dim=1 new_features = torch.cat(
) # (batch_size, 96) (prev_features[:, 1:96], samples), dim=1
) # (batch_size, 96)
new_features = new_features.float() new_features = new_features.float()
other_features, new_targets = dataset.get_batch_autoregressive( other_features, new_targets = dataset.get_batch_autoregressive(
np.array(idx_batch) + i + 1 np.array(idx_batch) + i + 1
) # (batch_size, new_features) ) # (batch_size, new_features)
if other_features is not None:
prev_features = torch.cat(
(new_features.to(self.device), other_features.to(self.device)), dim=1
) # (batch_size, 96 + new_features)
else:
prev_features = new_features
if other_features is not None:
prev_features = torch.cat(
(new_features.to(self.device), other_features.to(self.device)), dim=1
) # (batch_size, 96 + new_features)
else: else:
prev_features = new_features other_features, new_targets = dataset.get_batch_autoregressive(
np.array(idx_batch) + i + 1
) # (batch_size, 1, new_features)
# change the other_features nrv based on the samples
other_features[:, 0, 0] = samples.squeeze(-1)
# make sure on same device
other_features = other_features.to(self.device)
prev_features = prev_features.to(self.device)
prev_features = torch.cat(
(prev_features[:, 1:, :], other_features), dim=1
) # (batch_size, 96, new_features)
target_full = torch.cat( target_full = torch.cat(
(target_full, new_targets.to(self.device)), dim=1 (target_full, new_targets.to(self.device)), dim=1

6
src/trainers/trainer.py
View File

@@ -6,18 +6,20 @@ import plotly.graph_objects as go
import numpy as np import numpy as np
from plotly.subplots import make_subplots from plotly.subplots import make_subplots
from clearml.config import running_remotely from clearml.config import running_remotely
from torchinfo import summary
class Trainer: class Trainer:
def __init__( def __init__(
self, self,
model: torch.nn.Module, model: torch.nn.Module,
input_dim: tuple,
optimizer: torch.optim.Optimizer, optimizer: torch.optim.Optimizer,
criterion: torch.nn.Module, criterion: torch.nn.Module,
data_processor: DataProcessor, data_processor: DataProcessor,
device: torch.device, device: torch.device,
debug: bool = True, debug: bool = True,
): ):
self.input_dim = input_dim
self.model = model self.model = model
self.optimizer = optimizer self.optimizer = optimizer
self.criterion = criterion self.criterion = criterion
@@ -70,6 +72,8 @@ class Trainer:
task.add_tags(self.optimizer.__class__.__name__) task.add_tags(self.optimizer.__class__.__name__)
task.add_tags(self.__class__.__name__) task.add_tags(self.__class__.__name__)
task.set_configuration_object("model", str(summary(self.model, self.input_dim)))
self.optimizer.name = self.optimizer.__class__.__name__ self.optimizer.name = self.optimizer.__class__.__name__
self.criterion.name = self.criterion.__class__.__name__ self.criterion.name = self.criterion.__class__.__name__

43
src/training_scripts/autoregressive_quantiles.py
View File

@@ -17,19 +17,18 @@ from src.models.time_embedding_layer import TimeEmbedding
#### ClearML #### #### ClearML ####
clearml_helper = ClearMLHelper(project_name="Thesis/NrvForecast") clearml_helper = ClearMLHelper(project_name="Thesis/NrvForecast")
task = clearml_helper.get_task(task_name="Autoregressive Quantile Regression") task = clearml_helper.get_task(task_name="Autoregressive Quantile Regression: GRU + Quarter + Load Forecast")
#### Data Processor #### #### Data Processor ####
data_config = DataConfig() data_config = DataConfig()
data_config.NRV_HISTORY = True data_config.NRV_HISTORY = True
data_config.LOAD_HISTORY = True
data_config.LOAD_FORECAST = True data_config.LOAD_FORECAST = True
data_config.QUARTER = True data_config.QUARTER = True
data_config.DAY_OF_WEEK = True data_config.DAY_OF_WEEK = False
# data_config = task.connect(data_config, name="data_features") data_config = task.connect(data_config, name="data_features")
data_processor = DataProcessor(data_config, path="", lstm=True) data_processor = DataProcessor(data_config, path="", lstm=True)
data_processor.set_batch_size(512) data_processor.set_batch_size(512)
@@ -39,37 +38,49 @@ data_processor.set_full_day_skip(False)
#### Hyperparameters #### #### Hyperparameters ####
data_processor.set_output_size(1) data_processor.set_output_size(1)
inputDim = data_processor.get_input_size() inputDim = data_processor.get_input_size()
learningRate = 0.001 epochs = 400
epochs = 100
print("Input dim: ", inputDim)
# add parameters to clearml # add parameters to clearml
quantiles = task.get_parameter("general/quantiles", cast=True) quantiles = task.get_parameter("general/quantiles", cast=True)
# make sure it is a list
if quantiles is None: if quantiles is None:
quantiles = [0.01, 0.05, 0.1, 0.15, 0.3, 0.4, 0.5, 0.6, 0.7, 0.85, 0.9, 0.95, 0.99] quantiles = [0.01, 0.05, 0.1, 0.15, 0.3, 0.4, 0.5, 0.6, 0.7, 0.85, 0.9, 0.95, 0.99]
task.set_parameter("general/quantiles", quantiles) task.set_parameter("general/quantiles", quantiles)
else:
# if string, convert to list "[0.01, 0.05, 0.1, 0.15, 0.3, 0.4, 0.5, 0.6, 0.7, 0.85, 0.9, 0.95, 0.99]""
if isinstance(quantiles, str):
quantiles = eval(quantiles)
model_parameters = {
"learning_rate": 0.0001,
"hidden_size": 512,
"num_layers": 2,
"dropout": 0.2,
"time_feature_embedding": 4,
}
model_parameters = task.connect(model_parameters, name="model_parameters")
time_embedding = TimeEmbedding(data_processor.get_time_feature_size(), model_parameters["time_feature_embedding"])
lstm_model = GRUModel(time_embedding.output_dim(inputDim), len(quantiles), hidden_size=model_parameters["hidden_size"], num_layers=model_parameters["num_layers"], dropout=model_parameters["dropout"])
# model = LinearRegression(inputDim, len(quantiles))
time_embedding = TimeEmbedding(data_processor.get_time_feature_size(), 4)
# non_linear_regression_model = NonLinearRegression(time_embedding.output_dim(inputDim), len(quantiles), hiddenSize=1024, numLayers=5)
lstm_model = GRUModel(time_embedding.output_dim(inputDim), len(quantiles), hidden_size=512, num_layers=2)
model = nn.Sequential(time_embedding, lstm_model) model = nn.Sequential(time_embedding, lstm_model)
optimizer = torch.optim.Adam(model.parameters(), lr=learningRate) optimizer = torch.optim.Adam(model.parameters(), lr=model_parameters["learning_rate"])
#### Trainer #### #### Trainer ####
trainer = AutoRegressiveQuantileTrainer( trainer = AutoRegressiveQuantileTrainer(
model, model,
inputDim,
optimizer, optimizer,
data_processor, data_processor,
quantiles, quantiles,
"cuda", "cuda",
debug=True, debug=False,
) )
trainer.add_metrics_to_track( trainer.add_metrics_to_track(
[PinballLoss(quantiles), MSELoss(), L1Loss(), CRPSLoss()] [PinballLoss(quantiles), MSELoss(), L1Loss(), CRPSLoss()]
) )
trainer.early_stopping(patience=10) trainer.early_stopping(patience=30)
trainer.plot_every(100) trainer.plot_every(5)
trainer.train(task=task, epochs=epochs, remotely=True) trainer.train(task=task, epochs=epochs, remotely=True)

32
src/training_scripts/hyperparameter_optimizer.py
View File

@@ -60,11 +60,16 @@ quantile_lists = [
quantiles_range = DiscreteParameterRange("general/quantiles", values=quantile_lists) quantiles_range = DiscreteParameterRange("general/quantiles", values=quantile_lists)
#### Data Config #### #### Data Config ####
quarter_range = DiscreteParameterRange("data_features/quarter", values=[True, False]) quarter_range = DiscreteParameterRange("data_features/quarter", values=[True])
day_of_week_range = DiscreteParameterRange("data_features/day_of_week", values=[True, False]) day_of_week_range = DiscreteParameterRange("data_features/day_of_week", values=[True])
load_forecast_range = DiscreteParameterRange("data_features/load_forecast", values=[True, False]) load_forecast_range = DiscreteParameterRange("data_features/load_forecast", values=[True])
load_history_range = DiscreteParameterRange("data_features/load_history", values=[True, False])
learning_rate = DiscreteParameterRange("model_parameters/learning_rate", values=[0.00001, 0.00005, 0.0001, 0.0005, 0.001])
hidden_size = DiscreteParameterRange("model_parameters/hidden_size", values=[64, 128, 256, 512, 1024, 2048])
num_layers = DiscreteParameterRange("model_parameters/num_layers", values=[1, 2, 3, 4, 5, 6])
dropout = DiscreteParameterRange("model_parameters/dropout", values=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
time_feature_embedding = DiscreteParameterRange("model_parameters/time_feature_embedding", values=[1,2,3,4,5,6])
### OPTIMIZER OBJECT ### ### OPTIMIZER OBJECT ###
optimizer = HyperParameterOptimizer( optimizer = HyperParameterOptimizer(
@@ -75,24 +80,27 @@ optimizer = HyperParameterOptimizer(
execution_queue=execution_queue, execution_queue=execution_queue,
max_number_of_concurrent_tasks=1, max_number_of_concurrent_tasks=1,
optimizer_class=aSearchStrategy, optimizer_class=aSearchStrategy,
max_iteration_per_job=50, max_iteration_per_job=300,
# save_top_k_tasks_only=3, # save_top_k_tasks_only=3,
pool_period_min=0.2, pool_period_min=1,
total_max_jobs=15, total_max_jobs=40,
hyper_parameters=[ hyper_parameters=[
quantiles_range,
quarter_range, quarter_range,
day_of_week_range, day_of_week_range,
load_forecast_range, load_forecast_range,
load_history_range learning_rate,
hidden_size,
num_layers,
dropout,
time_feature_embedding
] ]
) )
task.execute_remotely(queue_name="hypertuning", exit_process=True) task.execute_remotely(queue_name="hypertuning", exit_process=True)
optimizer.set_report_period(0.2) optimizer.set_report_period(1)
def job_complete_callback( def job_complete_callback(
job_id, # type: str job_id, # type: str
@@ -106,9 +114,9 @@ def job_complete_callback(
print('WOOT WOOT we broke the record! Objective reached {}'.format(objective_value)) print('WOOT WOOT we broke the record! Objective reached {}'.format(objective_value))
optimizer.start(job_complete_callback=job_complete_callback) optimizer.start(job_complete_callback=job_complete_callback)
optimizer.set_time_limit(in_minutes=120.0) optimizer.set_time_limit(in_minutes=120.0*8)
optimizer.wait() optimizer.wait()
top_exp = optimizer.get_top_experiments(top_k=3) top_exp = optimizer.get_top_experiments(top_k=5)
print([t.id for t in top_exp]) print([t.id for t in top_exp])
# make sure background optimization stopped # make sure background optimization stopped
optimizer.stop() optimizer.stop()