Data

Tools

Indexes

Applications

Experiments

Open Source Science

271-anomaly-transformer-time-series-anomaly-detection-with-association-discrepancy

https://github.com/szu-advtech-2023/271-anomaly-transformer-time-series-anomaly-detection-with-association-discrepancy

Science Score: 10.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (5.1%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

Basic Info
  • Host: GitHub
  • Owner: SZU-AdvTech-2023
  • License: mit
  • Language: Python
  • Default Branch: main
  • Size: 397 MB
Statistics
  • Stars: 0
  • Watchers: 0
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Created over 2 years ago · Last pushed over 2 years ago
Metadata Files
Readme License Citation

README.md

0. Intro

0.1

[TOC]

0.2

  • paper: https://openreview.net/forum?id=LzQQ89U1qm_

  • codes: https://github.com/thuml/Anomaly-Transformer

  • datasets(): https://drive.google.com/drive/folders/1gisthCoE-RrKJ0j3KPV7xiibhHWT9qRm?usp=sharing

  • datasets()

  • my review:

1. Startup

https://github.com/thuml/Anomaly-TransformerGoogle Cloud

GTX 1660 6GB

CPUIntel i7-10700 2.90GHz

16GB DDR4

Ubuntu 20.04.1 5.15.0-89-generic ()

CUDArelease 11.5

bash Thu Nov 30 16:24:15 2023 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce GTX 1660 Off | 00000000:01:00.0 On | N/A | | 77% 78C P0 96W / 120W | 5636MiB / 6144MiB | 99% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+

Pytorch 1.8.0

conda 22.9.0condapython 3.6Anomaly-Transformercondapytorchconda

bash conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge

pipimport torch

bash pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

CUDAcudatoolkitCUDApytorch.

2.

SDMPSMMSLSMAPSWaT

2.1 SMD

SMD.sh

bash (Anomaly-Transformer) username@username-ubuntu:/media/username/folder/Dev/Anomaly-Transformer$ bash ./scripts/SMD.sh ./scripts/SMD.sh: line 2: $'\r': command not found Traceback (most recent call last): File "main.py", line 7, in <module> from solver import Solver File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 9, in <module> from data_factory.data_loader import get_loader_segment File "/media/username/folder/Dev/Anomaly-Transformer/data_factory/data_loader.py", line 11, in <module> import pandas as pd ModuleNotFoundError: No module named 'pandas' Traceback (most recent call last): File "main.py", line 7, in <module> from solver import Solver File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 9, in <module> from data_factory.data_loader import get_loader_segment File "/media/username/folder/Dev/Anomaly-Transformer/data_factory/data_loader.py", line 11, in <module> import pandas as pd ModuleNotFoundError: No module named 'pandas'

****packagepackage

  • sklearn: pip install scikit-learn
  • pandas : pip install pandas

SMD.sh

```bash (Anomaly-Transformer) username@username-ubuntu:/media/username/folder/Dev/Anomaly-Transformer$ bash ./scripts/SMD.sh ./scripts/SMD.sh: line 2: $'\r': command not found ------------ Options ------------- anormlyratio: 0.5 batchsize: 256 datapath: dataset/SMD dataset: SMD inputc: 38 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 10 outputc: 38 pretrainedmodel: None winsize: 100 -------------- End ---------------- Traceback (most recent call last): File "main.py", line 52, in main(config) File "main.py", line 18, in main solver = Solver(vars(config)) File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 74, in init dataset=self.dataset) File "/media/username/folder/Dev/Anomaly-Transformer/datafactory/dataloader.py", line 204, in getloadersegment dataset = SMDSegLoader(datapath, winsize, step, mode) File "/media/username/folder/Dev/Anomaly-Transformer/datafactory/dataloader.py", line 166, in init data = np.load(datapath + "/SMDtrain.npy") File "/home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/numpy/lib/npyio.py", line 416, in load fid = stack.entercontext(open(osfspath(file), "rb")) FileNotFoundError: [Errno 2] No such file or directory: 'dataset/SMD/SMDtrain.npy' ------------ Options ------------- anormlyratio: 0.5 batchsize: 256 datapath: dataset/SMD dataset: SMD inputc: 38 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 38 pretrainedmodel: 20 winsize: 100 -------------- End ---------------- Traceback (most recent call last): File "main.py", line 52, in main(config) File "main.py", line 18, in main solver = Solver(vars(config)) File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 74, in _init__ dataset=self.dataset) File "/media/username/folder/Dev/Anomaly-Transformer/datafactory/dataloader.py", line 204, in getloadersegment dataset = SMDSegLoader(datapath, winsize, step, mode) File "/media/username/folder/Dev/Anomaly-Transformer/datafactory/dataloader.py", line 166, in init data = np.load(datapath + "/SMDtrain.npy") File "/home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/numpy/lib/npyio.py", line 416, in load fid = stack.entercontext(open(osfspath(file), "rb")) FileNotFoundError: [Errno 2] No such file or directory: 'dataset/SMD/SMD_train.npy'

```

****'dataset/SMD/SMD_train.npy'(Tsinghua Cloud or Google Cloud)

Anomoly_Transformer/ dataset/ SMD/ SMD_test.npy SMD_train.npy ...... PSM/ test.csv train.csv ...... MSL/ MSL_test.npy ...... SMAP/ SMAP_test.npy ...... ......

SMD.sh

```bash (Anomaly-Transformer) username@username-ubuntu:/media/username/folder/Dev/Anomaly-Transformer$ bash ./scripts/SMD.sh ./scripts/SMD.sh: line 2: $'\r': command not found ------------ Options ------------- anormlyratio: 0.5 batchsize: 256 datapath: dataset/SMD dataset: SMD inputc: 38 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 10 outputc: 38 pretrainedmodel: None winsize: 100 -------------- End ---------------- ======================TRAIN MODE====================== Traceback (most recent call last): File "main.py", line 52, in main(config) File "main.py", line 21, in main solver.train() File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 161, in train self.winsize)).detach())) + torch.mean( File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 13, in myklloss res = p * (torch.log(p + 0.0001) - torch.log(q + 0.0001)) RuntimeError: CUDA out of memory. Tried to allocate 80.00 MiB (GPU 0; 5.79 GiB total capacity; 3.97 GiB already allocated; 49.75 MiB free; 4.11 GiB reserved in total by PyTorch) ------------ Options ------------- anormlyratio: 0.5 batchsize: 256 datapath: dataset/SMD dataset: SMD inputc: 38 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 38 pretrainedmodel: 20 winsize: 100 -------------- End ---------------- Traceback (most recent call last): File "main.py", line 52, in main(config) File "main.py", line 23, in main solver.test() File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 210, in test os.path.join(str(self.modelsavepath), str(self.dataset) + 'checkpoint.pth'))) File "/home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/serialization.py", line 579, in load with openfilelike(f, 'rb') as openedfile: File "/home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/serialization.py", line 230, in openfilelike return _openfile(nameorbuffer, mode) File "/home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/serialization.py", line 211, in init super(openfile, self).init(open(name, mode)) FileNotFoundError: [Errno 2] No such file or directory: 'checkpoints/SMD_checkpoint.pth'

```

  • 1CUDA out of memory: RuntimeError: CUDA out of memory. Tried to allocate 80.00 MiB (GPU 0; 5.79 GiB total capacity; 3.97 GiB already allocated; 49.75 MiB free; 4.11 GiB reserved in total by PyTorch)CUDA
  • 2checkpointcheckpointtest

CUDA

  • 1batch_size
  • batch_size256128./scripts/SMD.sh
  • RuntimeError: CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 5.79 GiB total capacity; 3.89 GiB already allocated; 82.94 MiB free; 4.05 GiB reserved in total by PyTorch)
  • batch_size64./scripts/SMD.sh
  • batch_size32./scripts/SMD.sh
  • epoch

bash ======================TRAIN MODE====================== speed: 0.1335s/iter; left time: 283.1503s speed: 0.1289s/iter; left time: 260.5845s Epoch: 1 cost time: 29.139591455459595 Epoch: 1, Steps: 222 | Train Loss: -40.3103769 Vali Loss: -46.1086967 Validation loss decreased (inf --> -46.108697). Saving model ... Updating learning rate to 0.0001 speed: 0.2505s/iter; left time: 475.7060s speed: 0.1302s/iter; left time: 234.2822s Epoch: 2 cost time: 28.97248649597168

epochtest

bash Threshold : 0.06388568006455485 pred: (708400,) gt: (708400,) pred: (708400,) gt: (708400,) Accuracy : 0.9926, Precision : 0.8927, Recall : 0.9329, F-score : 0.9124

bash (Anomaly-Transformer) username@username-ubuntu:/media/username/folder/Dev/Anomaly-Transformer$ bash ./scripts/SMD.sh ------------ Options ------------- anormly_ratio: 0.5 batch_size: 32 data_path: dataset/SMD dataset: SMD input_c: 38 k: 3 lr: 0.0001 mode: train model_save_path: checkpoints num_epochs: 10 output_c: 38 pretrained_model: None win_size: 100 -------------- End ---------------- ======================TRAIN MODE====================== speed: 0.1335s/iter; left time: 283.1503s speed: 0.1289s/iter; left time: 260.5845s Epoch: 1 cost time: 29.139591455459595 Epoch: 1, Steps: 222 | Train Loss: -40.3103769 Vali Loss: -46.1086967 Validation loss decreased (inf --> -46.108697). Saving model ... Updating learning rate to 0.0001 speed: 0.2505s/iter; left time: 475.7060s speed: 0.1302s/iter; left time: 234.2822s Epoch: 2 cost time: 28.97248649597168 Epoch: 2, Steps: 222 | Train Loss: -47.4852449 Vali Loss: -46.8629997 EarlyStopping counter: 1 out of 3 Updating learning rate to 5e-05 speed: 0.2555s/iter; left time: 428.5185s speed: 0.1307s/iter; left time: 206.1918s Epoch: 3 cost time: 29.593196392059326 Epoch: 3, Steps: 222 | Train Loss: -47.8205990 Vali Loss: -47.0798451 EarlyStopping counter: 2 out of 3 Updating learning rate to 2.5e-05 speed: 0.2540s/iter; left time: 369.4981s speed: 0.1327s/iter; left time: 179.8330s Epoch: 4 cost time: 29.744439840316772 Epoch: 4, Steps: 222 | Train Loss: -47.9206608 Vali Loss: -47.1366013 EarlyStopping counter: 3 out of 3 Early stopping ------------ Options ------------- anormly_ratio: 0.5 batch_size: 32 data_path: dataset/SMD dataset: SMD input_c: 38 k: 3 lr: 0.0001 mode: test model_save_path: checkpoints num_epochs: 10 output_c: 38 pretrained_model: 20 win_size: 100 -------------- End ---------------- ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.06388568006455485 pred: (708400,) gt: (708400,) pred: (708400,) gt: (708400,) Accuracy : 0.9926, Precision : 0.8927, Recall : 0.9329, F-score : 0.9124

2.2 PSM

PSM.sh

bash ======================TEST MODE====================== Threshold : 0.0011754722148179996 pred: (87800,) gt: (87800,) pred: (87800,) gt: (87800,) Accuracy : 0.9882, Precision : 0.9697, Recall : 0.9883, F-score : 0.9789

```bash

nomaly-Transformer$ bash ./scripts/PSM.sh ------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/PSM dataset: PSM inputc: 25 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 25 pretrainedmodel: None winsize: 100 -------------- End ---------------- test: (87841, 25) train: (132481, 25) test: (87841, 25) train: (132481, 25) test: (87841, 25) train: (132481, 25) test: (87841, 25) train: (132481, 25) ======================TRAIN MODE====================== speed: 0.1336s/iter; left time: 1644.4129s speed: 0.1275s/iter; left time: 1556.9812s speed: 0.1276s/iter; left time: 1545.0969s speed: 0.1277s/iter; left time: 1534.3159s speed: 0.1279s/iter; left time: 1524.0642s speed: 0.1279s/iter; left time: 1510.8930s speed: 0.1279s/iter; left time: 1498.2409s speed: 0.1279s/iter; left time: 1485.3983s speed: 0.1279s/iter; left time: 1472.8507s speed: 0.1280s/iter; left time: 1460.4547s speed: 0.1280s/iter; left time: 1448.4515s speed: 0.1282s/iter; left time: 1437.4174s speed: 0.1284s/iter; left time: 1427.1299s speed: 0.1284s/iter; left time: 1414.0394s speed: 0.1284s/iter; left time: 1401.1101s speed: 0.1285s/iter; left time: 1389.8639s speed: 0.1284s/iter; left time: 1375.8026s speed: 0.1283s/iter; left time: 1361.4072s speed: 0.1284s/iter; left time: 1349.9602s speed: 0.1284s/iter; left time: 1336.6942s speed: 0.1282s/iter; left time: 1322.4420s speed: 0.1283s/iter; left time: 1310.0931s speed: 0.1283s/iter; left time: 1297.5508s speed: 0.1282s/iter; left time: 1283.9510s speed: 0.1283s/iter; left time: 1271.7995s speed: 0.1283s/iter; left time: 1259.0883s speed: 0.1283s/iter; left time: 1245.8020s speed: 0.1283s/iter; left time: 1233.0175s speed: 0.1283s/iter; left time: 1220.1547s speed: 0.1283s/iter; left time: 1207.7776s speed: 0.1284s/iter; left time: 1195.7177s speed: 0.1282s/iter; left time: 1181.4120s speed: 0.1283s/iter; left time: 1168.8951s speed: 0.1282s/iter; left time: 1155.4520s speed: 0.1283s/iter; left time: 1143.4009s speed: 0.1284s/iter; left time: 1131.5084s speed: 0.1283s/iter; left time: 1117.7446s speed: 0.1282s/iter; left time: 1104.4219s speed: 0.1282s/iter; left time: 1091.2835s speed: 0.1283s/iter; left time: 1078.9449s speed: 0.1282s/iter; left time: 1065.9970s Epoch: 1 cost time: 531.1504812240601 Epoch: 1, Steps: 4137 | Train Loss: -48.0091480 Vali Loss: -48.8543076 Validation loss decreased (inf --> -48.854308). Saving model ... Updating learning rate to 0.0001 speed: 1.2588s/iter; left time: 10290.9493s speed: 0.1282s/iter; left time: 1035.4373s speed: 0.1282s/iter; left time: 1022.6818s speed: 0.1283s/iter; left time: 1010.5991s speed: 0.1282s/iter; left time: 996.7476s speed: 0.1283s/iter; left time: 984.5289s speed: 0.1282s/iter; left time: 971.1445s speed: 0.1282s/iter; left time: 958.5275s speed: 0.1282s/iter; left time: 945.7043s speed: 0.1283s/iter; left time: 933.1298s speed: 0.1282s/iter; left time: 919.9409s speed: 0.1282s/iter; left time: 907.2530s speed: 0.1282s/iter; left time: 894.4075s speed: 0.1282s/iter; left time: 881.5417s speed: 0.1283s/iter; left time: 869.0542s speed: 0.1283s/iter; left time: 856.4396s speed: 0.1284s/iter; left time: 844.1700s speed: 0.1283s/iter; left time: 830.4890s speed: 0.1282s/iter; left time: 816.9863s speed: 0.1283s/iter; left time: 804.9645s speed: 0.1282s/iter; left time: 791.7809s speed: 0.1283s/iter; left time: 779.5495s speed: 0.1283s/iter; left time: 766.7960s speed: 0.1282s/iter; left time: 753.4139s speed: 0.1282s/iter; left time: 740.1944s speed: 0.1282s/iter; left time: 727.6711s speed: 0.1284s/iter; left time: 715.8365s speed: 0.1282s/iter; left time: 701.6651s speed: 0.1283s/iter; left time: 689.5141s speed: 0.1282s/iter; left time: 676.0763s speed: 0.1282s/iter; left time: 663.4497s speed: 0.1282s/iter; left time: 650.6223s speed: 0.1284s/iter; left time: 638.5416s speed: 0.1282s/iter; left time: 625.1153s speed: 0.1283s/iter; left time: 612.6931s speed: 0.1283s/iter; left time: 599.7292s speed: 0.1282s/iter; left time: 586.6867s speed: 0.1284s/iter; left time: 574.6260s speed: 0.1283s/iter; left time: 561.4819s speed: 0.1283s/iter; left time: 548.4647s speed: 0.1283s/iter; left time: 535.5813s Epoch: 2 cost time: 530.542858839035 Epoch: 2, Steps: 4137 | Train Loss: -48.9527894 Vali Loss: -48.9326362 EarlyStopping counter: 1 out of 3 Updating learning rate to 5e-05 speed: 1.2538s/iter; left time: 5062.9567s speed: 0.1284s/iter; left time: 505.7279s speed: 0.1284s/iter; left time: 492.9298s speed: 0.1283s/iter; left time: 479.5802s speed: 0.1282s/iter; left time: 466.2639s speed: 0.1283s/iter; left time: 453.8794s speed: 0.1284s/iter; left time: 441.3263s speed: 0.1282s/iter; left time: 428.0605s speed: 0.1284s/iter; left time: 415.8170s speed: 0.1283s/iter; left time: 402.4540s speed: 0.1282s/iter; left time: 389.4098s speed: 0.1283s/iter; left time: 376.9801s speed: 0.1283s/iter; left time: 364.0838s speed: 0.1283s/iter; left time: 351.2112s speed: 0.1282s/iter; left time: 338.1965s speed: 0.1283s/iter; left time: 325.5066s speed: 0.1282s/iter; left time: 312.6431s speed: 0.1284s/iter; left time: 300.1481s speed: 0.1283s/iter; left time: 287.0474s speed: 0.1284s/iter; left time: 274.4572s speed: 0.1282s/iter; left time: 261.2532s speed: 0.1282s/iter; left time: 248.4272s speed: 0.1282s/iter; left time: 235.6939s speed: 0.1282s/iter; left time: 222.7621s speed: 0.1282s/iter; left time: 209.9875s speed: 0.1282s/iter; left time: 197.1853s speed: 0.1282s/iter; left time: 184.3661s speed: 0.1283s/iter; left time: 171.6811s speed: 0.1285s/iter; left time: 159.0394s speed: 0.1283s/iter; left time: 145.9588s speed: 0.1283s/iter; left time: 133.1463s speed: 0.1283s/iter; left time: 120.3105s speed: 0.1282s/iter; left time: 107.4170s speed: 0.1283s/iter; left time: 94.6591s speed: 0.1282s/iter; left time: 81.8221s speed: 0.1282s/iter; left time: 68.9838s speed: 0.1282s/iter; left time: 56.1650s speed: 0.1282s/iter; left time: 43.3404s speed: 0.1283s/iter; left time: 30.5237s speed: 0.1282s/iter; left time: 17.6921s speed: 0.1282s/iter; left time: 4.8703s Epoch: 3 cost time: 530.5573189258575 Epoch: 3, Steps: 4137 | Train Loss: -48.9824078 Vali Loss: -48.9623636 EarlyStopping counter: 2 out of 3 Updating learning rate to 2.5e-05 ------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/PSM dataset: PSM inputc: 25 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 25 pretrainedmodel: 20 winsize: 100 -------------- End ---------------- test: (87841, 25) train: (132481, 25) test: (87841, 25) train: (132481, 25) test: (87841, 25) train: (132481, 25) test: (87841, 25) train: (132481, 25) ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/reduction.py:42: UserWarning: sizeaverage and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.0011754722148179996 pred: (87800,) gt: (87800,) pred: (87800,) gt: (87800,) Accuracy : 0.9882, Precision : 0.9697, Recall : 0.9883, F-score : 0.9789

```

2.3 MSL

MSL.sh

```bash (Anomaly-Transformer) username@username-ubuntu:/media/username/folder/Dev/Anomaly-Transformer$ bash ./scripts/MSL.sh ------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/MSL dataset: MSL inputc: 55 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 55 pretrainedmodel: None winsize: 100 -------------- End ---------------- test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) Traceback (most recent call last): File "main.py", line 52, in main(config) File "main.py", line 18, in main solver = Solver(vars(config)) File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 85, in init self.buildmodel() File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 90, in buildmodel self.model = AnomalyTransformer(winsize=self.winsize, encin=self.inputc, cout=self.outputc, elayers=3) File "/media/username/folder/Dev/Anomaly-Transformer/model/AnomalyTransformer.py", line 77, in _init__ ) for l in range(elayers) File "/media/username/folder/Dev/Anomaly-Transformer/model/AnomalyTransformer.py", line 77, in ) for l in range(elayers) File "/media/username/folder/Dev/Anomaly-Transformer/model/attn.py", line 29, in init self.distances = torch.zeros((windowsize, windowsize)).cuda() File "/home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/cuda/init.py", line 170, in lazyinit torch.C.cudainit() RuntimeError: No CUDA GPUs are available ------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/MSL dataset: MSL inputc: 55 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 55 pretrainedmodel: 20 winsize: 100 -------------- End ---------------- test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) Traceback (most recent call last): File "main.py", line 52, in main(config) File "main.py", line 18, in main solver = Solver(vars(config)) File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 85, in _init__ self.buildmodel() File "/media/username/folder/Dev/Anomaly-Transformer/solver.py", line 90, in buildmodel self.model = AnomalyTransformer(winsize=self.winsize, encin=self.inputc, cout=self.outputc, elayers=3) File "/media/username/folder/Dev/Anomaly-Transformer/model/AnomalyTransformer.py", line 77, in _init__ ) for l in range(elayers) File "/media/username/folder/Dev/Anomaly-Transformer/model/AnomalyTransformer.py", line 77, in ) for l in range(elayers) File "/media/username/folder/Dev/Anomaly-Transformer/model/attn.py", line 29, in init self.distances = torch.zeros((windowsize, windowsize)).cuda() File "/home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/cuda/init.py", line 170, in lazyinit torch.C.cuda_init() RuntimeError: No CUDA GPUs are available

```

RuntimeError: No CUDA GPUs are availableGPUGPU

```bash (Anomaly-Transformer) username@username-ubuntu:/media/username/folder/Dev/Anomaly-Transformer$ python Python 3.6.13 |Anaconda, Inc.| (default, Jun 4 2021, 14:25:59) [GCC 7.5.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import torch print(torch.cuda.devicecount()) 1 print(torch.cuda.isavailable()) True ```

MSL.shMSL.sh

bash export CUDA_VISIBLE_DEVICES=7

70MSL.sh...(70)

MSL.sh

GPU

bash ======================TEST MODE====================== Threshold : 0.0012788161612115718 pred: (73700,) gt: (73700,) pred: (73700,) gt: (73700,) Accuracy : 0.9863, Precision : 0.9186, Recall : 0.9545, F-score : 0.9362

bash (Anomaly-Transformer) username@username-ubuntu:/media/username/folder/Dev/Anomaly-Transformer$ bash ./scripts/MSL.sh ------------ Options ------------- anormly_ratio: 1.0 batch_size: 32 data_path: dataset/MSL dataset: MSL input_c: 55 k: 3 lr: 0.0001 mode: train model_save_path: checkpoints num_epochs: 3 output_c: 55 pretrained_model: None win_size: 100 -------------- End ---------------- test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) ======================TRAIN MODE====================== speed: 0.1424s/iter; left time: 763.5606s speed: 0.1358s/iter; left time: 714.4375s speed: 0.1387s/iter; left time: 716.0470s speed: 0.1354s/iter; left time: 685.1058s speed: 0.1357s/iter; left time: 673.0052s speed: 0.1402s/iter; left time: 681.4136s speed: 0.1380s/iter; left time: 656.8551s speed: 0.1355s/iter; left time: 631.5301s speed: 0.1360s/iter; left time: 620.2718s speed: 0.1350s/iter; left time: 602.2681s speed: 0.1344s/iter; left time: 586.1590s speed: 0.1352s/iter; left time: 576.0738s speed: 0.1372s/iter; left time: 570.8950s speed: 0.1358s/iter; left time: 551.6753s speed: 0.1326s/iter; left time: 525.1665s speed: 0.1336s/iter; left time: 515.8068s speed: 0.1349s/iter; left time: 507.2338s speed: 0.1348s/iter; left time: 493.3304s Epoch: 1 cost time: 247.81738114356995 Epoch: 1, Steps: 1820 | Train Loss: -47.0458832 Vali Loss: -46.7697310 Validation loss decreased (inf --> -46.769731). Saving model ... Updating learning rate to 0.0001 speed: 1.1043s/iter; left time: 3910.1910s speed: 0.1326s/iter; left time: 456.2046s speed: 0.1330s/iter; left time: 444.4217s speed: 0.1336s/iter; left time: 433.0193s speed: 0.1396s/iter; left time: 438.4048s speed: 0.1384s/iter; left time: 420.9290s speed: 0.1358s/iter; left time: 399.4554s speed: 0.1363s/iter; left time: 387.1776s speed: 0.1354s/iter; left time: 371.1777s speed: 0.1354s/iter; left time: 357.5956s speed: 0.1351s/iter; left time: 343.2376s speed: 0.1355s/iter; left time: 330.8024s speed: 0.1362s/iter; left time: 318.7658s speed: 0.1367s/iter; left time: 306.3689s speed: 0.1363s/iter; left time: 291.7184s speed: 0.1358s/iter; left time: 277.2149s speed: 0.1362s/iter; left time: 264.4203s speed: 0.1352s/iter; left time: 248.9006s Epoch: 2 cost time: 246.60186314582825 Epoch: 2, Steps: 1820 | Train Loss: -48.5221037 Vali Loss: -47.3841785 EarlyStopping counter: 1 out of 3 Updating learning rate to 5e-05 speed: 1.1290s/iter; left time: 1942.9595s speed: 0.1394s/iter; left time: 225.9686s speed: 0.1351s/iter; left time: 205.5129s speed: 0.1406s/iter; left time: 199.7962s speed: 0.1332s/iter; left time: 175.9856s speed: 0.1326s/iter; left time: 161.8460s speed: 0.1314s/iter; left time: 147.2958s speed: 0.1334s/iter; left time: 136.1576s speed: 0.1319s/iter; left time: 121.5173s speed: 0.1389s/iter; left time: 114.0600s speed: 0.1306s/iter; left time: 94.1768s speed: 0.1396s/iter; left time: 86.6974s speed: 0.1352s/iter; left time: 70.4401s speed: 0.1373s/iter; left time: 57.8087s speed: 0.1379s/iter; left time: 44.2689s speed: 0.1322s/iter; left time: 29.2235s speed: 0.1308s/iter; left time: 15.8220s speed: 0.1308s/iter; left time: 2.7468s Epoch: 3 cost time: 245.10069799423218 Epoch: 3, Steps: 1820 | Train Loss: -48.7357392 Vali Loss: -47.5481951 EarlyStopping counter: 2 out of 3 Updating learning rate to 2.5e-05 ------------ Options ------------- anormly_ratio: 1.0 batch_size: 32 data_path: dataset/MSL dataset: MSL input_c: 55 k: 3 lr: 0.0001 mode: test model_save_path: checkpoints num_epochs: 10 output_c: 55 pretrained_model: 20 win_size: 100 -------------- End ---------------- test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) test: (73729, 55) train: (58317, 55) ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.0012788161612115718 pred: (73700,) gt: (73700,) pred: (73700,) gt: (73700,) Accuracy : 0.9863, Precision : 0.9186, Recall : 0.9545, F-score : 0.9362

2.4 SMAP

SMAP.sh

GPU

```bash ======================TEST MODE====================== Threshold : 0.0005670388956787038 pred: (427600,) gt: (427600,) pred: (427600,) gt: (427600,) Accuracy : 0.9906, Precision : 0.9360, Recall : 0.9943, F-score : 0.9642

```

bash (Anomaly-Transformer) username@username-ubuntu:/media/username/folder/Dev/Anomaly-Transformer$ bash ./scripts/SMAP.sh ------------ Options ------------- anormly_ratio: 1.0 batch_size: 32 data_path: dataset/SMAP dataset: SMAP input_c: 25 k: 3 lr: 0.0001 mode: train model_save_path: checkpoints num_epochs: 3 output_c: 25 pretrained_model: None win_size: 100 -------------- End ---------------- test: (427617, 25) train: (135183, 25) test: (427617, 25) train: (135183, 25) test: (427617, 25) train: (135183, 25) test: (427617, 25) train: (135183, 25) ======================TRAIN MODE====================== speed: 0.1343s/iter; left time: 1687.4161s speed: 0.1316s/iter; left time: 1641.0434s speed: 0.1304s/iter; left time: 1612.4711s speed: 0.1304s/iter; left time: 1599.8954s speed: 0.1315s/iter; left time: 1600.4497s speed: 0.1304s/iter; left time: 1573.3904s speed: 0.1306s/iter; left time: 1563.0526s speed: 0.1313s/iter; left time: 1557.9986s speed: 0.1308s/iter; left time: 1539.4388s speed: 0.1302s/iter; left time: 1518.6118s speed: 0.1312s/iter; left time: 1518.1523s speed: 0.1305s/iter; left time: 1495.9211s speed: 0.1306s/iter; left time: 1484.8276s speed: 0.1306s/iter; left time: 1471.1267s speed: 0.1297s/iter; left time: 1447.8968s speed: 0.1304s/iter; left time: 1442.7176s speed: 0.1299s/iter; left time: 1424.9521s speed: 0.1303s/iter; left time: 1415.9097s speed: 0.1309s/iter; left time: 1409.4994s speed: 0.1311s/iter; left time: 1398.0175s speed: 0.1318s/iter; left time: 1392.9594s speed: 0.1302s/iter; left time: 1362.8479s speed: 0.1371s/iter; left time: 1421.0785s speed: 0.1292s/iter; left time: 1326.9539s speed: 0.1303s/iter; left time: 1324.7232s speed: 0.1308s/iter; left time: 1316.6065s speed: 0.1309s/iter; left time: 1304.8822s speed: 0.1306s/iter; left time: 1289.0181s speed: 0.1322s/iter; left time: 1291.2752s speed: 0.1315s/iter; left time: 1271.0320s speed: 0.1302s/iter; left time: 1245.4013s speed: 0.1310s/iter; left time: 1240.1241s speed: 0.1309s/iter; left time: 1225.9448s speed: 0.1300s/iter; left time: 1204.4843s speed: 0.1308s/iter; left time: 1198.7496s speed: 0.1329s/iter; left time: 1205.0089s speed: 0.1319s/iter; left time: 1183.1681s speed: 0.1301s/iter; left time: 1153.9812s speed: 0.1295s/iter; left time: 1135.2198s speed: 0.1307s/iter; left time: 1132.5606s speed: 0.1312s/iter; left time: 1124.1417s speed: 0.1296s/iter; left time: 1097.1383s Epoch: 1 cost time: 553.0207221508026 Epoch: 1, Steps: 4222 | Train Loss: -47.6426614 Vali Loss: -48.1685601 Validation loss decreased (inf --> -48.168560). Saving model ... Updating learning rate to 0.0001 speed: 5.5265s/iter; left time: 46118.9601s speed: 0.1295s/iter; left time: 1067.8902s speed: 0.1297s/iter; left time: 1056.0485s speed: 0.1296s/iter; left time: 1042.8939s speed: 0.1328s/iter; left time: 1055.0172s speed: 0.1347s/iter; left time: 1056.7791s speed: 0.1300s/iter; left time: 1006.6005s speed: 0.1293s/iter; left time: 988.4494s speed: 0.1292s/iter; left time: 975.1445s speed: 0.1294s/iter; left time: 963.6841s speed: 0.1292s/iter; left time: 948.7126s speed: 0.1292s/iter; left time: 936.1060s speed: 0.1291s/iter; left time: 922.7592s speed: 0.1291s/iter; left time: 909.7682s speed: 0.1291s/iter; left time: 896.7182s speed: 0.1290s/iter; left time: 882.9915s speed: 0.1291s/iter; left time: 870.9842s speed: 0.1289s/iter; left time: 856.8340s speed: 0.1289s/iter; left time: 843.7893s speed: 0.1291s/iter; left time: 831.9244s speed: 0.1292s/iter; left time: 819.9622s speed: 0.1297s/iter; left time: 809.7022s speed: 0.1293s/iter; left time: 794.2553s speed: 0.1292s/iter; left time: 781.1323s speed: 0.1292s/iter; left time: 767.8188s speed: 0.1293s/iter; left time: 755.7132s speed: 0.1292s/iter; left time: 742.2035s speed: 0.1293s/iter; left time: 729.9284s speed: 0.1294s/iter; left time: 717.5859s speed: 0.1293s/iter; left time: 703.9006s speed: 0.1292s/iter; left time: 690.3800s speed: 0.1291s/iter; left time: 677.3184s speed: 0.1293s/iter; left time: 665.2619s speed: 0.1292s/iter; left time: 651.7931s speed: 0.1292s/iter; left time: 638.8412s speed: 0.1293s/iter; left time: 626.6065s speed: 0.1292s/iter; left time: 612.8525s speed: 0.1291s/iter; left time: 599.8719s speed: 0.1292s/iter; left time: 587.1467s speed: 0.1292s/iter; left time: 574.4164s speed: 0.1293s/iter; left time: 561.6398s speed: 0.1291s/iter; left time: 548.2016s Epoch: 2 cost time: 546.9801330566406 Epoch: 2, Steps: 4222 | Train Loss: -48.5213919 Vali Loss: -48.2957534 EarlyStopping counter: 1 out of 3 Updating learning rate to 5e-05 speed: 5.4772s/iter; left time: 22582.6544s speed: 0.1305s/iter; left time: 524.9768s speed: 0.1301s/iter; left time: 510.4672s speed: 0.1292s/iter; left time: 494.0773s speed: 0.1291s/iter; left time: 480.7389s speed: 0.1293s/iter; left time: 468.5919s speed: 0.1292s/iter; left time: 455.1257s speed: 0.1292s/iter; left time: 442.3313s speed: 0.1293s/iter; left time: 429.8049s speed: 0.1293s/iter; left time: 416.6019s speed: 0.1291s/iter; left time: 403.3154s speed: 0.1292s/iter; left time: 390.4252s speed: 0.1291s/iter; left time: 377.3882s speed: 0.1292s/iter; left time: 364.6556s speed: 0.1293s/iter; left time: 352.1070s speed: 0.1291s/iter; left time: 338.6508s speed: 0.1292s/iter; left time: 325.8527s speed: 0.1291s/iter; left time: 312.7774s speed: 0.1292s/iter; left time: 300.1695s speed: 0.1291s/iter; left time: 286.9356s speed: 0.1291s/iter; left time: 274.0536s speed: 0.1299s/iter; left time: 262.8002s speed: 0.1324s/iter; left time: 254.5154s speed: 0.1298s/iter; left time: 236.6313s speed: 0.1328s/iter; left time: 228.8171s speed: 0.1327s/iter; left time: 215.4497s speed: 0.1304s/iter; left time: 198.6513s speed: 0.1295s/iter; left time: 184.3195s speed: 0.1299s/iter; left time: 171.8900s speed: 0.1292s/iter; left time: 157.9532s speed: 0.1290s/iter; left time: 144.8993s speed: 0.1292s/iter; left time: 132.2070s speed: 0.1293s/iter; left time: 119.3879s speed: 0.1291s/iter; left time: 106.2423s speed: 0.1291s/iter; left time: 93.3420s speed: 0.1291s/iter; left time: 80.4567s speed: 0.1292s/iter; left time: 67.5827s speed: 0.1292s/iter; left time: 54.6509s speed: 0.1293s/iter; left time: 41.7594s speed: 0.1292s/iter; left time: 28.8161s speed: 0.1292s/iter; left time: 15.8970s speed: 0.1291s/iter; left time: 2.9699s Epoch: 3 cost time: 547.1292362213135 Epoch: 3, Steps: 4222 | Train Loss: -48.6120459 Vali Loss: -48.3690009 EarlyStopping counter: 2 out of 3 Updating learning rate to 2.5e-05 ------------ Options ------------- anormly_ratio: 1.0 batch_size: 32 data_path: dataset/SMAP dataset: SMAP input_c: 25 k: 3 lr: 0.0001 mode: test model_save_path: checkpoints num_epochs: 10 output_c: 25 pretrained_model: 20 win_size: 100 -------------- End ---------------- test: (427617, 25) train: (135183, 25) test: (427617, 25) train: (135183, 25) test: (427617, 25) train: (135183, 25) test: (427617, 25) train: (135183, 25) ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.0005670388956787038 pred: (427600,) gt: (427600,) pred: (427600,) gt: (427600,) Accuracy : 0.9906, Precision : 0.9360, Recall : 0.9943, F-score : 0.9642

2.5 SWaT

iTrustSWaT

KTable 132015SWaTGoogle DriveSWAT/SWaT.A1&A2_Dec 2015/Physical/SWaT_dataset_Attack_v0.xlsxSWaT_dataset_Normal_v1.xlsx

```python

1. (P1,P2)csv

2. Pythonnumpynpy

import numpy as np import pandas as pd

swattrainpd = pd.readcsv('./dataset/SWaT/SWaTDatasetNormalv1.csv') swattestpd = pd.readcsv('./dataset/SWaT/SWaTDatasetAttackv0.csv')

print(swattrainpd.shape) print(swattestpd.shape) print(swattestpd['Normal/Attack'].unique()) print(swattestpd.head()) """ (495000, 53) (449919, 53) ['Normal' 'Attack' 'A ttack'] Timestamp FIT101 LIT101 ... P602 P603 Normal/Attack 0 28/12/2015 10:00:00 AM 2.427057 522.8467 ... 1 1 Normal 1 28/12/2015 10:00:01 AM 2.446274 522.8860 ... 1 1 Normal 2 28/12/2015 10:00:02 AM 2.489191 522.8467 ... 1 1 Normal 3 28/12/2015 10:00:03 AM 2.534350 522.9645 ... 1 1 Normal 4 28/12/2015 10:00:04 AM 2.569260 523.4748 ... 1 1 Normal

[5 rows x 53 columns] """

swattestpd = swattestpd.replace('Normal',0).replace('Attack',1).replace('A ttack',1) swattestlabelnp = swattestpd.iloc[:,52].values swattestnp = swattestpd.drop([' Timestamp','Normal/Attack'], axis=1).values swattrainnp = swattrain_pd.drop([' Timestamp','Normal/Attack'], axis=1).values

print(swattrainnp.shape) print(swattestnp.shape) print(swattestlabel_np.shape) """ (495000, 51) (449919, 51) (449919,) """

np.save('./dataset/SWaT/swattestlabel.npy',swattestlabelnp) np.save('./dataset/SWaT/swattrain.npy',swattrainnp) np.save('./dataset/SWaT/swattest.npy',swattest_np) ```

./scripts/SWaT.sh./scripts/Start.sh

```bash export CUDAVISIBLEDEVICES=0

python main.py --anormlyratio 0.5 --numepochs 3 --batchsize 32 --mode train --dataset SWaT --datapath dataset/SWaT --inputc 51 --outputc 51 python main.py --anormlyratio 0.1 --numepochs 10 --batchsize 32 --mode test --dataset SWaT --datapath dataset/SWaT --inputc 51 --outputc 51 --pretrained_model 10 ```

SWaTdataloder./data_factory/data_loader.pySwatSegLoaderget_loader_segment

```python ''' Loader for SWaT dataset ''' class SwatSegLoader(object): def init(self, datapath, winsize, step, mode="train"): self.mode = mode self.step = step self.winsize = winsize self.scaler = StandardScaler() data = np.load(datapath + "/swattrain.npy") self.scaler.fit(data) data = self.scaler.transform(data) testdata = np.load(datapath + "/swattest.npy") self.test = self.scaler.transform(testdata)

    self.train = data
    self.val = self.test
    self.test_labels = np.load(data_path + "/swat_test_label.npy")
    print("test:", self.test.shape)
    print("train:", self.train.shape)

def __len__(self):

    if self.mode == "train":
        return (self.train.shape[0] - self.win_size) // self.step + 1
    elif (self.mode == 'val'):
        return (self.val.shape[0] - self.win_size) // self.step + 1
    elif (self.mode == 'test'):
        return (self.test.shape[0] - self.win_size) // self.step + 1
    else:
        return (self.test.shape[0] - self.win_size) // self.win_size + 1

def __getitem__(self, index):
    index = index * self.step
    if self.mode == "train":
        return np.float32(self.train[index:index + self.win_size]), np.float32(self.test_labels[0:self.win_size])
    elif (self.mode == 'val'):
        return np.float32(self.val[index:index + self.win_size]), np.float32(self.test_labels[0:self.win_size])
    elif (self.mode == 'test'):
        return np.float32(self.test[index:index + self.win_size]), np.float32(
            self.test_labels[index:index + self.win_size])
    else:
        return np.float32(self.test[
                          index // self.step * self.win_size:index // self.step * self.win_size + self.win_size]), np.float32(
            self.test_labels[index // self.step * self.win_size:index // self.step * self.win_size + self.win_size])

""" Add a new line about the SWaT dataset """
def getloadersegment(datapath, batchsize, winsize=100, step=100, mode='train', dataset='KDD'): if (dataset == 'SMD'): dataset = SMDSegLoader(datapath, winsize, step, mode) elif (dataset == 'MSL'): dataset = MSLSegLoader(datapath, winsize, 1, mode) elif (dataset == 'SMAP'): dataset = SMAPSegLoader(datapath, winsize, 1, mode) elif (dataset == 'PSM'): dataset = PSMSegLoader(datapath, winsize, 1, mode) elif (dataset == 'SWaT'): # added this dataset = SwatSegLoader(datapath, win_size, 1, mode)

shuffle = False
if mode == 'train':
    shuffle = True

data_loader = DataLoader(dataset=dataset,
                         batch_size=batch_size,
                         shuffle=shuffle,
                         num_workers=0)
return data_loader

```

SWaT.sh

bash ======================TEST MODE====================== Threshold : 0.0031170047065244427 pred: (449900,) gt: (449900,) pred: (449900,) gt: (449900,) Accuracy : 0.9775, Precision : 0.8841, Recall : 0.9371, F-score : 0.9099

```bash (Anomaly-Transformer) username@username-ubuntu:/media/username/3E6E20236E1FD28F/Dev/Anomaly-Transformer$ bash ./scripts/SWaT.sh ------------ Options ------------- anormlyratio: 0.1 batchsize: 32 datapath: dataset/SWaT dataset: SWaT inputc: 51 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 51 pretrainedmodel: 10 winsize: 100 -------------- End ---------------- test: (449919, 51) train: (496800, 51) test: (449919, 51) train: (496800, 51) test: (449919, 51) train: (496800, 51) test: (449919, 51) train: (496800, 51) ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/reduction.py:42: UserWarning: sizeaverage and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.0032192275498528246 pred: (449900,) gt: (449900,) pred: (449900,) gt: (449900,) Accuracy : 0.9771, Precision : 0.8965, Recall : 0.9172, F-score : 0.9067 (Anomaly-Transformer) username@username-ubuntu:/media/username/3E6E20236E1FD28F/Dev/Anomaly-Transformer$ bash ./scripts/SWaT.sh ------------ Options ------------- anormlyratio: 0.5 batchsize: 32 datapath: dataset/SWaT dataset: SWaT inputc: 51 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 51 pretrainedmodel: None winsize: 100 -------------- End ---------------- test: (449919, 51) train: (495000, 51) test: (449919, 51) train: (495000, 51) test: (449919, 51) train: (495000, 51) test: (449919, 51) train: (495000, 51) ======================TRAIN MODE====================== speed: 0.1428s/iter; left time: 6612.5713s speed: 0.1354s/iter; left time: 6253.0974s speed: 0.1352s/iter; left time: 6230.7968s speed: 0.1355s/iter; left time: 6233.4215s speed: 0.1390s/iter; left time: 6378.5659s speed: 0.1403s/iter; left time: 6425.6394s speed: 0.1348s/iter; left time: 6161.5534s speed: 0.1359s/iter; left time: 6194.7005s speed: 0.1356s/iter; left time: 6168.4861s speed: 0.1360s/iter; left time: 6174.7164s speed: 0.1395s/iter; left time: 6319.8666s speed: 0.1389s/iter; left time: 6276.0570s speed: 0.1371s/iter; left time: 6183.6705s speed: 0.1401s/iter; left time: 6302.9684s speed: 0.1351s/iter; left time: 6067.7535s speed: 0.1351s/iter; left time: 6050.1461s speed: 0.1365s/iter; left time: 6101.5390s speed: 0.1378s/iter; left time: 6144.6511s speed: 0.1372s/iter; left time: 6104.3526s speed: 0.1382s/iter; left time: 6137.7459s speed: 0.1381s/iter; left time: 6117.2903s speed: 0.1371s/iter; left time: 6061.2191s speed: 0.1365s/iter; left time: 6021.1155s speed: 0.1345s/iter; left time: 5915.7146s speed: 0.1349s/iter; left time: 5921.9364s speed: 0.1355s/iter; left time: 5933.2968s speed: 0.1354s/iter; left time: 5914.8026s speed: 0.1424s/iter; left time: 6210.5447s speed: 0.1359s/iter; left time: 5911.6384s speed: 0.1374s/iter; left time: 5964.3514s speed: 0.1348s/iter; left time: 5838.2409s speed: 0.1346s/iter; left time: 5816.1968s speed: 0.1346s/iter; left time: 5802.6837s speed: 0.1346s/iter; left time: 5788.3701s speed: 0.1353s/iter; left time: 5804.4665s speed: 0.1394s/iter; left time: 5966.4150s speed: 0.1423s/iter; left time: 6074.2209s speed: 0.1420s/iter; left time: 6049.9245s speed: 0.1345s/iter; left time: 5715.0185s speed: 0.1344s/iter; left time: 5698.0888s speed: 0.1411s/iter; left time: 5968.1633s speed: 0.1415s/iter; left time: 5970.2509s speed: 0.1408s/iter; left time: 5928.4565s speed: 0.1426s/iter; left time: 5990.2664s speed: 0.1357s/iter; left time: 5686.5547s speed: 0.1365s/iter; left time: 5704.3444s speed: 0.1358s/iter; left time: 5664.3863s speed: 0.1392s/iter; left time: 5791.4018s speed: 0.1372s/iter; left time: 5693.4794s speed: 0.1362s/iter; left time: 5638.4746s speed: 0.1376s/iter; left time: 5682.1785s speed: 0.1382s/iter; left time: 5695.7126s speed: 0.1381s/iter; left time: 5673.8861s speed: 0.1428s/iter; left time: 5854.1201s speed: 0.1353s/iter; left time: 5535.4870s speed: 0.1358s/iter; left time: 5538.6886s speed: 0.1359s/iter; left time: 5530.9134s speed: 0.1360s/iter; left time: 5520.0075s speed: 0.1369s/iter; left time: 5545.7690s speed: 0.1353s/iter; left time: 5466.8770s speed: 0.1356s/iter; left time: 5465.3225s speed: 0.1344s/iter; left time: 5403.5386s speed: 0.1346s/iter; left time: 5398.9813s speed: 0.1364s/iter; left time: 5455.0500s speed: 0.1344s/iter; left time: 5362.2032s speed: 0.1346s/iter; left time: 5357.8926s speed: 0.1349s/iter; left time: 5356.4118s speed: 0.1358s/iter; left time: 5375.5669s speed: 0.1393s/iter; left time: 5503.1650s speed: 0.1394s/iter; left time: 5492.4838s speed: 0.1436s/iter; left time: 5641.9017s speed: 0.1382s/iter; left time: 5417.8900s speed: 0.1361s/iter; left time: 5320.7390s speed: 0.1370s/iter; left time: 5342.9212s speed: 0.1406s/iter; left time: 5470.8222s speed: 0.1363s/iter; left time: 5289.7796s speed: 0.1354s/iter; left time: 5240.4470s speed: 0.1352s/iter; left time: 5218.9477s speed: 0.1415s/iter; left time: 5446.3757s speed: 0.1396s/iter; left time: 5360.8597s speed: 0.1403s/iter; left time: 5371.5712s speed: 0.1355s/iter; left time: 5174.8979s speed: 0.1347s/iter; left time: 5132.1327s speed: 0.1356s/iter; left time: 5153.9954s speed: 0.1357s/iter; left time: 5144.1700s speed: 0.1364s/iter; left time: 5156.1387s speed: 0.1370s/iter; left time: 5164.2171s speed: 0.1385s/iter; left time: 5208.6199s speed: 0.1450s/iter; left time: 5438.2515s speed: 0.1492s/iter; left time: 5580.7061s speed: 0.1395s/iter; left time: 5203.9755s speed: 0.1372s/iter; left time: 5103.0719s speed: 0.1402s/iter; left time: 5202.6695s speed: 0.1426s/iter; left time: 5274.6904s speed: 0.1393s/iter; left time: 5141.1819s speed: 0.1393s/iter; left time: 5126.7243s speed: 0.1356s/iter; left time: 4976.0627s speed: 0.1359s/iter; left time: 4975.5315s speed: 0.1387s/iter; left time: 5061.0901s speed: 0.1418s/iter; left time: 5162.7360s speed: 0.1377s/iter; left time: 4997.3233s speed: 0.1360s/iter; left time: 4922.3810s speed: 0.1459s/iter; left time: 5267.9064s speed: 0.1375s/iter; left time: 4949.4103s speed: 0.1393s/iter; left time: 5001.8771s speed: 0.1465s/iter; left time: 5245.6815s speed: 0.1383s/iter; left time: 4938.3803s speed: 0.1372s/iter; left time: 4884.7422s speed: 0.1372s/iter; left time: 4871.4987s speed: 0.1378s/iter; left time: 4877.6222s speed: 0.1345s/iter; left time: 4748.5213s speed: 0.1344s/iter; left time: 4730.7891s speed: 0.1346s/iter; left time: 4725.0120s speed: 0.1359s/iter; left time: 4756.4741s speed: 0.1351s/iter; left time: 4715.7454s speed: 0.1344s/iter; left time: 4676.1957s speed: 0.1359s/iter; left time: 4716.7439s speed: 0.1349s/iter; left time: 4668.0923s speed: 0.1355s/iter; left time: 4673.7719s speed: 0.1369s/iter; left time: 4708.6927s speed: 0.1346s/iter; left time: 4615.7443s speed: 0.1419s/iter; left time: 4851.2301s speed: 0.1420s/iter; left time: 4842.0394s speed: 0.1390s/iter; left time: 4726.7629s speed: 0.1400s/iter; left time: 4745.2550s speed: 0.1417s/iter; left time: 4789.7382s speed: 0.1388s/iter; left time: 4677.9071s speed: 0.1359s/iter; left time: 4566.2085s speed: 0.1362s/iter; left time: 4561.5282s speed: 0.1412s/iter; left time: 4717.2056s speed: 0.1399s/iter; left time: 4657.4927s speed: 0.1398s/iter; left time: 4639.9572s speed: 0.1386s/iter; left time: 4586.4466s speed: 0.1437s/iter; left time: 4742.9178s speed: 0.1414s/iter; left time: 4653.5077s speed: 0.1389s/iter; left time: 4555.4527s speed: 0.1407s/iter; left time: 4600.7729s speed: 0.1353s/iter; left time: 4410.3650s speed: 0.1359s/iter; left time: 4415.5378s speed: 0.1348s/iter; left time: 4368.9311s speed: 0.1359s/iter; left time: 4389.8059s speed: 0.1355s/iter; left time: 4362.4974s speed: 0.1355s/iter; left time: 4348.9004s speed: 0.1363s/iter; left time: 4360.6657s speed: 0.1350s/iter; left time: 4307.7063s speed: 0.1354s/iter; left time: 4305.2038s speed: 0.1359s/iter; left time: 4309.3123s speed: 0.1353s/iter; left time: 4276.3862s speed: 0.1357s/iter; left time: 4274.6798s speed: 0.1354s/iter; left time: 4249.9516s speed: 0.1356s/iter; left time: 4245.4502s speed: 0.1366s/iter; left time: 4261.3997s speed: 0.1374s/iter; left time: 4272.8442s speed: 0.1353s/iter; left time: 4194.5101s Epoch: 1 cost time: 2127.7168984413147 Epoch: 1, Steps: 15466 | Train Loss: -48.4473046 Vali Loss: -47.3290840 Validation loss decreased (inf --> -47.329084). Saving model ... Updating learning rate to 0.0001 speed: 6.2369s/iter; left time: 192301.2652s speed: 0.1412s/iter; left time: 4340.9789s speed: 0.1374s/iter; left time: 4207.7656s speed: 0.1395s/iter; left time: 4258.3836s speed: 0.1371s/iter; left time: 4172.0284s speed: 0.1385s/iter; left time: 4202.1003s speed: 0.1396s/iter; left time: 4219.4023s speed: 0.1378s/iter; left time: 4153.6058s speed: 0.1369s/iter; left time: 4110.4457s speed: 0.1371s/iter; left time: 4103.2520s speed: 0.1402s/iter; left time: 4181.8363s speed: 0.1373s/iter; left time: 4081.6900s speed: 0.1351s/iter; left time: 4003.4332s speed: 0.1390s/iter; left time: 4105.8047s speed: 0.1424s/iter; left time: 4190.9918s speed: 0.1417s/iter; left time: 4156.5764s speed: 0.1403s/iter; left time: 4102.5730s speed: 0.1420s/iter; left time: 4138.0644s speed: 0.1429s/iter; left time: 4147.9949s speed: 0.1402s/iter; left time: 4055.5878s speed: 0.1436s/iter; left time: 4141.6097s speed: 0.1432s/iter; left time: 4115.6612s speed: 0.1436s/iter; left time: 4112.7574s speed: 0.1425s/iter; left time: 4066.4701s speed: 0.1379s/iter; left time: 3921.2554s speed: 0.1488s/iter; left time: 4216.4488s speed: 0.1372s/iter; left time: 3873.5076s speed: 0.1406s/iter; left time: 3955.3746s speed: 0.1341s/iter; left time: 3759.9923s speed: 0.1341s/iter; left time: 3745.7212s speed: 0.1340s/iter; left time: 3730.7106s speed: 0.1341s/iter; left time: 3717.9604s speed: 0.1340s/iter; left time: 3703.4859s speed: 0.1340s/iter; left time: 3690.3612s speed: 0.1340s/iter; left time: 3677.0124s speed: 0.1340s/iter; left time: 3662.8140s speed: 0.1340s/iter; left time: 3650.3931s speed: 0.1340s/iter; left time: 3636.0835s speed: 0.1341s/iter; left time: 3624.2639s speed: 0.1341s/iter; left time: 3611.4985s speed: 0.1391s/iter; left time: 3732.2035s speed: 0.1365s/iter; left time: 3649.8741s speed: 0.1396s/iter; left time: 3719.0167s speed: 0.1363s/iter; left time: 3616.9123s speed: 0.1385s/iter; left time: 3659.8609s speed: 0.1389s/iter; left time: 3657.7603s speed: 0.1378s/iter; left time: 3614.8408s speed: 0.1432s/iter; left time: 3741.8558s speed: 0.1415s/iter; left time: 3683.3259s speed: 0.1433s/iter; left time: 3714.9921s speed: 0.1375s/iter; left time: 3552.5034s speed: 0.1403s/iter; left time: 3610.4658s speed: 0.1388s/iter; left time: 3559.0740s speed: 0.1391s/iter; left time: 3551.4850s speed: 0.1344s/iter; left time: 3419.0121s speed: 0.1383s/iter; left time: 3502.4784s speed: 0.1405s/iter; left time: 3544.0996s speed: 0.1440s/iter; left time: 3619.6918s speed: 0.1463s/iter; left time: 3663.1171s speed: 0.1437s/iter; left time: 3582.2442s speed: 0.1425s/iter; left time: 3538.2324s speed: 0.1430s/iter; left time: 3537.9648s speed: 0.1382s/iter; left time: 3405.0349s speed: 0.1349s/iter; left time: 3308.9421s speed: 0.1352s/iter; left time: 3304.2378s speed: 0.1352s/iter; left time: 3289.0986s speed: 0.1359s/iter; left time: 3292.5206s speed: 0.1351s/iter; left time: 3260.5215s speed: 0.1359s/iter; left time: 3266.8184s speed: 0.1381s/iter; left time: 3305.8850s speed: 0.1421s/iter; left time: 3387.8256s speed: 0.1362s/iter; left time: 3233.5133s speed: 0.1406s/iter; left time: 3323.1694s speed: 0.1390s/iter; left time: 3270.9122s speed: 0.1481s/iter; left time: 3470.9108s speed: 0.1480s/iter; left time: 3452.5954s speed: 0.1434s/iter; left time: 3332.1595s speed: 0.1444s/iter; left time: 3340.4601s speed: 0.1429s/iter; left time: 3290.3965s speed: 0.1418s/iter; left time: 3251.0263s speed: 0.1432s/iter; left time: 3269.5059s speed: 0.1446s/iter; left time: 3286.1990s speed: 0.1414s/iter; left time: 3200.7980s speed: 0.1359s/iter; left time: 3061.4477s speed: 0.1357s/iter; left time: 3043.1181s speed: 0.1374s/iter; left time: 3067.6164s speed: 0.1344s/iter; left time: 2988.2089s speed: 0.1344s/iter; left time: 2974.6237s speed: 0.1345s/iter; left time: 2962.6316s speed: 0.1391s/iter; left time: 3051.0582s speed: 0.1375s/iter; left time: 3002.8196s speed: 0.1360s/iter; left time: 2955.7436s speed: 0.1369s/iter; left time: 2962.3081s speed: 0.1407s/iter; left time: 3030.2290s speed: 0.1372s/iter; left time: 2940.8313s speed: 0.1365s/iter; left time: 2911.8519s speed: 0.1359s/iter; left time: 2885.3581s speed: 0.1359s/iter; left time: 2871.7553s speed: 0.1348s/iter; left time: 2835.1846s speed: 0.1355s/iter; left time: 2837.3081s speed: 0.1349s/iter; left time: 2810.6872s speed: 0.1380s/iter; left time: 2860.7424s speed: 0.1382s/iter; left time: 2850.8490s speed: 0.1380s/iter; left time: 2833.0666s speed: 0.1358s/iter; left time: 2775.4763s speed: 0.1364s/iter; left time: 2772.4910s speed: 0.1417s/iter; left time: 2866.6913s speed: 0.1408s/iter; left time: 2834.4955s speed: 0.1414s/iter; left time: 2833.6041s speed: 0.1386s/iter; left time: 2761.7363s speed: 0.1373s/iter; left time: 2722.3473s speed: 0.1450s/iter; left time: 2861.6037s speed: 0.1379s/iter; left time: 2707.0912s speed: 0.1358s/iter; left time: 2652.5041s speed: 0.1386s/iter; left time: 2694.2524s speed: 0.1394s/iter; left time: 2694.5749s speed: 0.1407s/iter; left time: 2706.0995s speed: 0.1428s/iter; left time: 2731.5554s speed: 0.1428s/iter; left time: 2718.7514s speed: 0.1411s/iter; left time: 2670.9368s speed: 0.1419s/iter; left time: 2673.1494s speed: 0.1371s/iter; left time: 2569.1856s speed: 0.1363s/iter; left time: 2540.2892s speed: 0.1393s/iter; left time: 2582.3330s speed: 0.1404s/iter; left time: 2588.2281s speed: 0.1390s/iter; left time: 2547.5025s speed: 0.1358s/iter; left time: 2475.4478s speed: 0.1347s/iter; left time: 2442.8927s speed: 0.1389s/iter; left time: 2504.9989s speed: 0.1393s/iter; left time: 2498.3219s speed: 0.1385s/iter; left time: 2469.3112s speed: 0.1439s/iter; left time: 2551.8338s speed: 0.1393s/iter; left time: 2455.4148s speed: 0.1361s/iter; left time: 2385.9529s speed: 0.1380s/iter; left time: 2405.8499s speed: 0.1386s/iter; left time: 2401.7565s speed: 0.1405s/iter; left time: 2421.6689s speed: 0.1346s/iter; left time: 2305.7041s speed: 0.1358s/iter; left time: 2312.4812s speed: 0.1366s/iter; left time: 2313.5609s speed: 0.1446s/iter; left time: 2433.9046s speed: 0.1391s/iter; left time: 2327.8682s speed: 0.1357s/iter; left time: 2257.7770s speed: 0.1372s/iter; left time: 2267.6875s speed: 0.1361s/iter; left time: 2236.3395s speed: 0.1362s/iter; left time: 2224.1642s speed: 0.1359s/iter; left time: 2205.6602s speed: 0.1355s/iter; left time: 2185.4923s speed: 0.1360s/iter; left time: 2180.5024s speed: 0.1358s/iter; left time: 2164.1690s speed: 0.1347s/iter; left time: 2132.9385s speed: 0.1351s/iter; left time: 2124.7418s speed: 0.1361s/iter; left time: 2127.1795s speed: 0.1354s/iter; left time: 2103.8047s Epoch: 2 cost time: 2142.760348558426 Epoch: 2, Steps: 15466 | Train Loss: -48.7614085 Vali Loss: -47.4089206 Validation loss decreased (-47.329084 --> -47.408921). Saving model ... Updating learning rate to 5e-05 speed: 6.1778s/iter; left time: 94934.8721s speed: 0.1406s/iter; left time: 2146.9605s speed: 0.1405s/iter; left time: 2131.2590s speed: 0.1406s/iter; left time: 2118.3086s speed: 0.1407s/iter; left time: 2105.8900s speed: 0.1407s/iter; left time: 2091.9465s speed: 0.1406s/iter; left time: 2076.6571s speed: 0.1406s/iter; left time: 2062.1350s speed: 0.1404s/iter; left time: 2044.6462s speed: 0.1415s/iter; left time: 2047.4681s speed: 0.1482s/iter; left time: 2129.4256s speed: 0.1492s/iter; left time: 2127.9783s speed: 0.1365s/iter; left time: 1934.1541s speed: 0.1399s/iter; left time: 1968.6168s speed: 0.1400s/iter; left time: 1955.1173s speed: 0.1475s/iter; left time: 2045.0873s speed: 0.1462s/iter; left time: 2012.1008s speed: 0.1486s/iter; left time: 2030.5059s speed: 0.1392s/iter; left time: 1889.0985s speed: 0.1359s/iter; left time: 1830.0555s speed: 0.1362s/iter; left time: 1821.0789s speed: 0.1380s/iter; left time: 1830.8140s speed: 0.1400s/iter; left time: 1842.7255s speed: 0.1408s/iter; left time: 1840.0198s speed: 0.1404s/iter; left time: 1821.1826s speed: 0.1390s/iter; left time: 1788.3899s speed: 0.1352s/iter; left time: 1725.9144s speed: 0.1353s/iter; left time: 1713.4649s speed: 0.1352s/iter; left time: 1699.5077s speed: 0.1351s/iter; left time: 1683.8779s speed: 0.1349s/iter; left time: 1668.2950s speed: 0.1350s/iter; left time: 1656.3314s speed: 0.1349s/iter; left time: 1641.9323s speed: 0.1358s/iter; left time: 1638.1283s speed: 0.1408s/iter; left time: 1684.5690s speed: 0.1397s/iter; left time: 1658.0442s speed: 0.1364s/iter; left time: 1605.3649s speed: 0.1355s/iter; left time: 1581.1990s speed: 0.1354s/iter; left time: 1565.9528s speed: 0.1353s/iter; left time: 1551.1118s speed: 0.1355s/iter; left time: 1539.6940s speed: 0.1426s/iter; left time: 1606.9790s speed: 0.1432s/iter; left time: 1599.1447s speed: 0.1396s/iter; left time: 1544.6739s speed: 0.1375s/iter; left time: 1508.0367s speed: 0.1357s/iter; left time: 1474.7991s speed: 0.1387s/iter; left time: 1493.3745s speed: 0.1369s/iter; left time: 1460.2498s speed: 0.1450s/iter; left time: 1532.1255s speed: 0.1354s/iter; left time: 1417.6454s speed: 0.1367s/iter; left time: 1417.3438s speed: 0.1383s/iter; left time: 1419.4605s speed: 0.1394s/iter; left time: 1417.0308s speed: 0.1381s/iter; left time: 1389.8998s speed: 0.1359s/iter; left time: 1354.2720s speed: 0.1379s/iter; left time: 1360.3001s speed: 0.1400s/iter; left time: 1366.8987s speed: 0.1357s/iter; left time: 1311.7372s speed: 0.1355s/iter; left time: 1296.3165s speed: 0.1400s/iter; left time: 1325.6356s speed: 0.1396s/iter; left time: 1307.1830s speed: 0.1438s/iter; left time: 1332.1646s speed: 0.1425s/iter; left time: 1306.2933s speed: 0.1405s/iter; left time: 1273.5351s speed: 0.1383s/iter; left time: 1240.3545s speed: 0.1399s/iter; left time: 1240.1732s speed: 0.1349s/iter; left time: 1182.6134s speed: 0.1357s/iter; left time: 1176.3826s speed: 0.1369s/iter; left time: 1172.9671s speed: 0.1396s/iter; left time: 1181.7322s speed: 0.1503s/iter; left time: 1257.1756s speed: 0.1516s/iter; left time: 1253.5654s speed: 0.1357s/iter; left time: 1108.4112s speed: 0.1349s/iter; left time: 1088.4797s speed: 0.1383s/iter; left time: 1102.0619s speed: 0.1379s/iter; left time: 1084.9288s speed: 0.1415s/iter; left time: 1098.7484s speed: 0.1359s/iter; left time: 1042.0906s speed: 0.1380s/iter; left time: 1044.4410s speed: 0.1410s/iter; left time: 1052.5578s speed: 0.1362s/iter; left time: 1003.2852s speed: 0.1407s/iter; left time: 1022.6825s speed: 0.1375s/iter; left time: 985.7831s speed: 0.1369s/iter; left time: 967.7417s speed: 0.1386s/iter; left time: 965.7697s speed: 0.1374s/iter; left time: 943.5992s speed: 0.1373s/iter; left time: 929.1964s speed: 0.1411s/iter; left time: 940.4838s speed: 0.1358s/iter; left time: 891.6402s speed: 0.1361s/iter; left time: 880.2868s speed: 0.1361s/iter; left time: 866.7270s speed: 0.1399s/iter; left time: 876.9626s speed: 0.1391s/iter; left time: 857.7881s speed: 0.1390s/iter; left time: 843.3105s speed: 0.1417s/iter; left time: 845.6014s speed: 0.1393s/iter; left time: 817.4279s speed: 0.1448s/iter; left time: 835.1416s speed: 0.1418s/iter; left time: 803.5788s speed: 0.1396s/iter; left time: 777.3048s speed: 0.1352s/iter; left time: 739.0622s speed: 0.1342s/iter; left time: 720.3296s speed: 0.1356s/iter; left time: 714.3432s speed: 0.1349s/iter; left time: 697.1035s speed: 0.1357s/iter; left time: 687.6707s speed: 0.1371s/iter; left time: 680.7852s speed: 0.1348s/iter; left time: 656.0860s speed: 0.1394s/iter; left time: 664.3335s speed: 0.1424s/iter; left time: 664.4477s speed: 0.1430s/iter; left time: 652.9596s speed: 0.1432s/iter; left time: 639.7950s speed: 0.1415s/iter; left time: 618.0999s speed: 0.1405s/iter; left time: 599.4433s speed: 0.1417s/iter; left time: 590.2657s speed: 0.1402s/iter; left time: 570.0331s speed: 0.1411s/iter; left time: 559.8401s speed: 0.1404s/iter; left time: 543.0194s speed: 0.1401s/iter; left time: 527.8371s speed: 0.1402s/iter; left time: 513.9767s speed: 0.1402s/iter; left time: 500.0044s speed: 0.1385s/iter; left time: 480.0465s speed: 0.1368s/iter; left time: 460.6511s speed: 0.1387s/iter; left time: 453.0378s speed: 0.1385s/iter; left time: 438.5996s speed: 0.1406s/iter; left time: 431.1120s speed: 0.1376s/iter; left time: 408.1366s speed: 0.1430s/iter; left time: 409.8844s speed: 0.1467s/iter; left time: 405.8109s speed: 0.1465s/iter; left time: 390.8402s speed: 0.1357s/iter; left time: 348.3139s speed: 0.1357s/iter; left time: 334.8425s speed: 0.1374s/iter; left time: 325.2861s speed: 0.1424s/iter; left time: 322.7762s speed: 0.1408s/iter; left time: 305.0053s speed: 0.1422s/iter; left time: 293.8833s speed: 0.1435s/iter; left time: 282.3321s speed: 0.1438s/iter; left time: 268.4056s speed: 0.1446s/iter; left time: 255.5926s speed: 0.1386s/iter; left time: 230.9797s speed: 0.1357s/iter; left time: 212.7075s speed: 0.1370s/iter; left time: 201.0489s speed: 0.1391s/iter; left time: 190.1014s speed: 0.1348s/iter; left time: 170.7725s speed: 0.1383s/iter; left time: 161.4011s speed: 0.1370s/iter; left time: 146.2138s speed: 0.1350s/iter; left time: 130.5128s speed: 0.1350s/iter; left time: 117.0379s speed: 0.1348s/iter; left time: 103.3958s speed: 0.1349s/iter; left time: 89.9780s speed: 0.1350s/iter; left time: 76.5349s speed: 0.1350s/iter; left time: 63.0486s speed: 0.1350s/iter; left time: 49.5276s speed: 0.1349s/iter; left time: 36.0312s speed: 0.1349s/iter; left time: 22.5222s speed: 0.1350s/iter; left time: 9.0454s Epoch: 3 cost time: 2149.257043838501 Epoch: 3, Steps: 15466 | Train Loss: -48.9121188 Vali Loss: -47.4772334 EarlyStopping counter: 1 out of 3 Updating learning rate to 2.5e-05 ------------ Options ------------- anormlyratio: 0.1 batchsize: 32 datapath: dataset/SWaT dataset: SWaT inputc: 51 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 51 pretrainedmodel: 10 winsize: 100 -------------- End ---------------- test: (449919, 51) train: (495000, 51) test: (449919, 51) train: (495000, 51) test: (449919, 51) train: (495000, 51) test: (449919, 51) train: (495000, 51) ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/reduction.py:42: UserWarning: sizeaverage and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.0031170047065244427 pred: (449900,) gt: (449900,) pred: (449900,) gt: (449900,) Accuracy : 0.9775, Precision : 0.8841, Recall : 0.9371, F-score : 0.9099

```

NeurIPS-TS

Bencmark:https://github.com/datamllab/tods

Python[]

pip install -e .

```bash Getting requirements to build wheel ... error error: subprocess-exited-with-error

Getting requirements to build wheel did not run successfully. exit code: 1

[154 lines of output] :15: DeprecationWarning: pkgresources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkgresources.html :51: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. :54: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. :51: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. performance hint: statsmodels/tsa/regimeswitching/hamiltonfilter.pyx:83:5: Exception check on 'shamiltonfilterlogiteration' will always require the GIL to be acquired. Possible solutions: 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. 2. Use an 'int' return type on the function to allow an error code to be returned.

```

****statsmodels==0.11.1setup.pystatsmodels==0.11.0rc1pip install -e .

bash ERROR: Could not find a version that satisfies the requirement tensorflow==2.4 (from tods) (from versions: 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0rc0, 2.6.0rc1, 2.6.0rc2, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.7.0rc0, 2.7.0rc1, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.8.0rc0, 2.8.0rc1, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.10.0, 2.10.1, 2.11.0rc0, 2.11.0rc1, 2.11.0rc2, 2.11.0, 2.11.1, 2.12.0rc0, 2.12.0rc1, 2.12.0, 2.12.1, 2.13.0rc0, 2.13.0rc1, 2.13.0rc2, 2.13.0, 2.13.1, 2.14.0rc0, 2.14.0rc1, 2.14.0, 2.14.1, 2.15.0rc0, 2.15.0rc1, 2.15.0, 2.15.0.post1) ERROR: No matching distribution found for tensorflow==2.4

****tensorflow 2.4setup.pytensorflow==2.5

bash ERROR: Could not find a version that satisfies the requirement keras-nightly~=2.5.0.dev (from tensorflow) (from versions: none) ERROR: No matching distribution found for keras-nightly~=2.5.0.dev

****keras-nightly~=2.5.0.devpypi.orghttps://pypi.org/project/keras-nightly/#historywhlhttps://pypi.org/project/keras-nightly/2.5.0.dev2021032900/pip install ./keras_nightly-2.5.0.dev2021032900-py2.py3-none-any.whl

bash ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. spyder 5.3.3 requires pyqt5<5.16, which is not installed. spyder 5.3.3 requires pyqtwebengine<5.16, which is not installed. daal4py 2021.6.0 requires daal==2021.4.0, which is not installed. anaconda-project 0.11.1 requires ruamel-yaml, which is not installed. pylint 2.14.5 requires typing-extensions>=3.10.0; python_version < "3.10", but you have typing-extensions 3.7.4.3 which is incompatible. imageio 2.19.3 requires pillow>=8.3.2, but you have pillow 7.1.2 which is incompatible. conda-repo-cli 1.0.20 requires clyent==1.2.1, but you have clyent 1.2.2 which is incompatible. conda-repo-cli 1.0.20 requires nbformat==5.4.0, but you have nbformat 5.5.0 which is incompatible. conda-repo-cli 1.0.20 requires PyYAML==6.0, but you have pyyaml 5.4.1 which is incompatible. conda-repo-cli 1.0.20 requires requests==2.28.1, but you have requests 2.26.0 which is incompatible. bokeh 2.4.3 requires typing-extensions>=3.10.0, but you have typing-extensions 3.7.4.3 which is incompatible. black 22.6.0 requires typing-extensions>=3.10.0.0; python_version < "3.10", but you have typing-extensions 3.7.4.3 which is incompatible. astroid 2.11.7 requires typing-extensions>=3.10; python_version < "3.10", but you have typing-extensions 3.7.4.3 which is incompatible. Successfully installed GitPython-3.1.24 absl-py-0.15.0 aiosignal-1.3.1 astunparse-1.6.3 cachetools-5.3.2 combo-0.1.3 custom-inherit-2.3.2 dateparser-1.1.8 flatbuffers-1.12 frozendict-1.2 frozenlist-1.4.0 gast-0.4.0 gitdb-4.0.11 google-auth-2.25.1 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 gputil-1.4.0 grpcio-1.34.1 grpcio-testing-1.32.0 grpcio-tools-1.34.1 h5py-3.1.0 jsonpath-ng-1.5.3 jsonschema-4.0.1 keras-2.4.0 keras-preprocessing-1.1.2 liac-arff-2.5.0 more-itertools-8.5.0 nimfa-1.4.0 numpy-1.19.5 oauthlib-3.2.2 openml-0.11.0 opt-einsum-3.3.0 pandas-1.3.4 pillow-7.1.2 protobuf-3.20.3 pyarrow-14.0.1 pyod-1.0.5 pytypes-1.0b10 pyyaml-5.4.1 ray-2.8.1 requests-2.26.0 requests-oauthlib-1.3.1 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rsa-4.9 scikit-learn-0.24.2 scipy-1.7.1 simplejson-3.12.0 six-1.15.0 smmap-5.0.1 statsmodels-0.11.0rc1 stumpy-1.4.0 tamu_axolotl-2021.2.11.1 tamu_d3m-2022.5.23 tensorboard-2.11.2 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorboardX-2.6.2.2 tensorflow-2.5.0 tensorflow-estimator-2.5.0 termcolor-1.1.0 tods-0.0.2 typing-extensions-3.7.4.3 typing-inspect-0.7.1 tzlocal-5.2 webcolors-1.11.1 wrapt-1.12.1 xgboost-2.0.2 xmltodict-0.13.0

Conda

Pycharmconda interpreter, python 3.8 ( Python 3.6 && pip 19+)

setup.py

todscondapythonpip install -e .

python interpreterdocker

test_example.py

```python import pandas as pd

from tods import schemas as schemasutils from tods import generatedataset, evaluate_pipeline

tablepath = 'datasets/anomaly/rawdata/yahoosub5.csv' targetindex = 6 # what column is the target metric = 'F1MACRO' # F1 on both label 0 and 1

Read data and generate dataset

df = pd.readcsv(tablepath) dataset = generatedataset(df, targetindex)

Load the default pipeline

pipeline = schemasutils.loaddefault_pipeline()

Run the pipeline

pipelineresult = evaluatepipeline(dataset, pipeline, metric) print(pipeline_result) ```

json {'method_called': 'evaluate', 'outputs': "[{'outputs.0': d3mIndex anomaly" '0 0 0' '1 1 0' '2 2 0' '3 3 0' '4 4 0' '... ... ...' '1395 1395 0' '1396 1396 0' '1397 1397 1' '1398 1398 1' '1399 1399 0' '' "[1400 rows x 2 columns]}, {'outputs.0': d3mIndex anomaly" '0 0 0' '1 1 0' '2 2 0' '3 3 0' '4 4 0' '... ... ...' '1395 1395 0' '1396 1396 0' '1397 1397 1' '1398 1398 1' '1399 1399 0' '' '[1400 rows x 2 columns]}]', 'pipeline': '<d3m.metadata.pipeline.Pipeline object at 0x7fcab1e73cd0>', 'scores': ' metric value normalized randomSeed fold' '0 F1_MACRO 0.708549 0.708549 0 0', 'status': 'COMPLETED'}

2.6

| Dataset \ Metrics | Accuracy | Precision | Recall | F1-score | | :---------------: | :------: | :-------: | :----: | :------: | | SMD / Ours | 99.26 | 89.27 | 93.29 | 91.24 | | SMD / Paper | \ | 89.40 | 95.45 | 92.33 | | MSL / Ours | 98.63 | 91.86 | 95.45 | 93.62 | | MSL / Paper | \ | 92.09 | 95.15 | 93.59 | | SMAP / Ours | 99.06 | 93.60 | 99.43 | 96.42 | | SMAP / Paper | \ | 94.13 | 99.40 | 96.69 | | SWaT / Ours | 97.75 | 88.41 | 93.71 | 90.99 | | SWaT / Paper | \ | 91.55 | 96.73 | 94.07 | | PSM / Ours | 98.82 | 96.97 | 98.83 | 97.89 | | PSM / Paper | \ | 96.91 | 98.90 | 97.89 |

4Table 1F1-ScoreTable 1

UCR dataset

https://compete.hexagon-ml.com/media/data/multi-dataset-time-series-anomaly-detection-39/data.zip

3.

tokenizertokenizer

3.1 Anomaly ratio $r$

$r$Anomaly Score $\delta$$r$. $r$

3.1.1

```bash Anomaly Ratio : 50.0 Threshold : 0.0 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.5032, Precision : 0.7028, Recall : 0.2204, F-score : 0.3356

Threshold : 0.0 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.5032, Precision : 0.7028, Recall : 0.2204, F-score : 0.3356 ```

3.2

3.2.1 NSL-KDD

normal01anomaly ratioNSLKDD

r=0.5%

``` bash (Anomaly-Transformer) username@username-ubuntu:/media/username/3E6E20236E1FD28F/Dev/Anomaly-Transformer$ bash ./scripts/NSLKDD.sh ------------ Options ------------- anormlyratio: 0.5 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None winsize: 100 -------------- End ---------------- ======================TRAIN MODE======================

------------ Options ------------- anormlyratio: 0.5 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD inputc: 122 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 122 pretrainedmodel: 10 winsize: 100 -------------- End ----------------

======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/reduction.py:42: UserWarning: sizeaverage and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.02469959240406773 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.4585, Precision : 0.9481, Recall : 0.0514, F-score : 0.0975 ```

r=1.0%

```bash (Anomaly-Transformer) username@username-ubuntu:/media/username/3E6E20236E1FD28F/Dev/Anomaly-Transformer$ bash ./scripts/NSLKDD.sh ------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None winsize: 100 -------------- End ---------------- test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) ======================TRAIN MODE====================== speed: 0.1369s/iter; left time: 1601.7550s speed: 0.1303s/iter; left time: 1512.0029s speed: 0.1306s/iter; left time: 1502.6880s speed: 0.1307s/iter; left time: 1490.6938s speed: 0.1308s/iter; left time: 1478.8884s speed: 0.1308s/iter; left time: 1465.4615s speed: 0.1308s/iter; left time: 1452.1905s speed: 0.1309s/iter; left time: 1440.6744s speed: 0.1313s/iter; left time: 1431.1096s speed: 0.1313s/iter; left time: 1418.0322s speed: 0.1312s/iter; left time: 1404.4908s speed: 0.1313s/iter; left time: 1392.5284s speed: 0.1313s/iter; left time: 1378.5787s speed: 0.1313s/iter; left time: 1365.6873s speed: 0.1312s/iter; left time: 1351.8481s speed: 0.1313s/iter; left time: 1339.4321s speed: 0.1312s/iter; left time: 1325.7825s speed: 0.1312s/iter; left time: 1312.7482s speed: 0.1312s/iter; left time: 1299.0108s speed: 0.1313s/iter; left time: 1286.7188s speed: 0.1312s/iter; left time: 1273.0868s speed: 0.1312s/iter; left time: 1260.0926s speed: 0.1316s/iter; left time: 1250.9483s speed: 0.1335s/iter; left time: 1254.8993s speed: 0.1328s/iter; left time: 1235.7682s speed: 0.1308s/iter; left time: 1204.1539s speed: 0.1309s/iter; left time: 1191.3958s speed: 0.1309s/iter; left time: 1178.2278s speed: 0.1322s/iter; left time: 1176.9927s speed: 0.1315s/iter; left time: 1157.5977s speed: 0.1315s/iter; left time: 1144.1452s speed: 0.1314s/iter; left time: 1130.5234s speed: 0.1314s/iter; left time: 1117.5005s speed: 0.1314s/iter; left time: 1104.2098s speed: 0.1314s/iter; left time: 1090.8631s speed: 0.1315s/iter; left time: 1078.5724s speed: 0.1314s/iter; left time: 1064.4714s speed: 0.1315s/iter; left time: 1052.1539s speed: 0.1353s/iter; left time: 1069.2737s Epoch: 1 cost time: 517.9919922351837 Epoch: 1, Steps: 3934 | Train Loss: -47.0414631 Vali Loss: -47.4802924 Validation loss decreased (inf --> -47.480292). Saving model ... Updating learning rate to 0.0001 speed: 0.4840s/iter; left time: 3759.9627s speed: 0.1332s/iter; left time: 1021.6347s speed: 0.1323s/iter; left time: 1001.5881s speed: 0.1319s/iter; left time: 985.3198s speed: 0.1345s/iter; left time: 990.9520s speed: 0.1343s/iter; left time: 976.3390s speed: 0.1390s/iter; left time: 996.3020s speed: 0.1346s/iter; left time: 951.4199s speed: 0.1360s/iter; left time: 947.8869s speed: 0.1341s/iter; left time: 921.2710s speed: 0.1379s/iter; left time: 933.7204s speed: 0.1326s/iter; left time: 883.9982s speed: 0.1394s/iter; left time: 915.8344s speed: 0.1355s/iter; left time: 876.6080s speed: 0.1428s/iter; left time: 909.5229s speed: 0.1436s/iter; left time: 900.0828s speed: 0.1437s/iter; left time: 886.3455s speed: 0.1433s/iter; left time: 869.4626s speed: 0.1445s/iter; left time: 862.6468s speed: 0.1449s/iter; left time: 850.2270s speed: 0.1418s/iter; left time: 818.0399s speed: 0.1367s/iter; left time: 774.7034s speed: 0.1367s/iter; left time: 761.4599s speed: 0.1367s/iter; left time: 747.4475s speed: 0.1364s/iter; left time: 732.2058s speed: 0.1368s/iter; left time: 721.0366s speed: 0.1367s/iter; left time: 706.8142s speed: 0.1367s/iter; left time: 693.1174s speed: 0.1367s/iter; left time: 679.1568s speed: 0.1367s/iter; left time: 665.7158s speed: 0.1368s/iter; left time: 652.5808s speed: 0.1367s/iter; left time: 638.4532s speed: 0.1362s/iter; left time: 622.4852s speed: 0.1364s/iter; left time: 609.4713s speed: 0.1364s/iter; left time: 595.9573s speed: 0.1364s/iter; left time: 582.4890s speed: 0.1364s/iter; left time: 568.5825s speed: 0.1364s/iter; left time: 554.9945s speed: 0.1365s/iter; left time: 541.9608s Epoch: 2 cost time: 539.6562712192535 Epoch: 2, Steps: 3934 | Train Loss: -48.5144279 Vali Loss: -48.1329151 EarlyStopping counter: 1 out of 3 Updating learning rate to 5e-05 speed: 0.4813s/iter; left time: 1845.6853s speed: 0.1364s/iter; left time: 509.2899s speed: 0.1362s/iter; left time: 495.1138s speed: 0.1364s/iter; left time: 482.0215s speed: 0.1365s/iter; left time: 468.8174s speed: 0.1364s/iter; left time: 455.0160s speed: 0.1364s/iter; left time: 441.3130s speed: 0.1364s/iter; left time: 427.6231s speed: 0.1364s/iter; left time: 414.0394s speed: 0.1364s/iter; left time: 400.4002s speed: 0.1363s/iter; left time: 386.2983s speed: 0.1363s/iter; left time: 372.6786s speed: 0.1364s/iter; left time: 359.5014s speed: 0.1364s/iter; left time: 345.8320s speed: 0.1364s/iter; left time: 332.1094s speed: 0.1363s/iter; left time: 318.1958s speed: 0.1364s/iter; left time: 304.7477s speed: 0.1362s/iter; left time: 290.8440s speed: 0.1366s/iter; left time: 277.9560s speed: 0.1363s/iter; left time: 263.7950s speed: 0.1364s/iter; left time: 250.2387s speed: 0.1364s/iter; left time: 236.6038s speed: 0.1363s/iter; left time: 222.8828s speed: 0.1363s/iter; left time: 209.2962s speed: 0.1364s/iter; left time: 195.7429s speed: 0.1363s/iter; left time: 181.9378s speed: 0.1364s/iter; left time: 168.4278s speed: 0.1362s/iter; left time: 154.6098s speed: 0.1360s/iter; left time: 140.7701s speed: 0.1364s/iter; left time: 127.5001s speed: 0.1359s/iter; left time: 113.5145s speed: 0.1362s/iter; left time: 100.1328s speed: 0.1363s/iter; left time: 86.5670s speed: 0.1362s/iter; left time: 72.8662s speed: 0.1361s/iter; left time: 59.1943s speed: 0.1363s/iter; left time: 45.6561s speed: 0.1361s/iter; left time: 31.9842s speed: 0.1363s/iter; left time: 18.3962s speed: 0.1364s/iter; left time: 4.7725s Epoch: 3 cost time: 536.1699142456055 Epoch: 3, Steps: 3934 | Train Loss: -48.7206043 Vali Loss: -48.3033336 EarlyStopping counter: 2 out of 3 Updating learning rate to 2.5e-05 ------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD inputc: 122 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 122 pretrainedmodel: 10 winsize: 100 -------------- End ---------------- test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/reduction.py:42: UserWarning: sizeaverage and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.007011290364898737 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.4537, Precision : 0.8757, Recall : 0.0468, F-score : 0.0888

/home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/reduction.py:42: UserWarning: sizeaverage and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.0031170047065244427 pred: (449900,) gt: (449900,) pred: (449900,) gt: (449900,) Accuracy : 0.9775, Precision : 0.8841, Recall : 0.9371, F-score : 0.9099

(Anomaly-Transformer) username@username-ubuntu:/media/username/3E6E20236E1FD28F/Dev/Anomaly-Transformer$ bash ./scripts/NSLKDD.sh ------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None winsize: 100 -------------- End ---------------- test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) ======================TRAIN MODE====================== speed: 0.1369s/iter; left time: 1601.7550s speed: 0.1303s/iter; left time: 1512.0029s speed: 0.1306s/iter; left time: 1502.6880s speed: 0.1307s/iter; left time: 1490.6938s speed: 0.1308s/iter; left time: 1478.8884s speed: 0.1308s/iter; left time: 1465.4615s speed: 0.1308s/iter; left time: 1452.1905s speed: 0.1309s/iter; left time: 1440.6744s speed: 0.1313s/iter; left time: 1431.1096s speed: 0.1313s/iter; left time: 1418.0322s speed: 0.1312s/iter; left time: 1404.4908s speed: 0.1313s/iter; left time: 1392.5284s speed: 0.1313s/iter; left time: 1378.5787s speed: 0.1313s/iter; left time: 1365.6873s speed: 0.1312s/iter; left time: 1351.8481s speed: 0.1313s/iter; left time: 1339.4321s speed: 0.1312s/iter; left time: 1325.7825s speed: 0.1312s/iter; left time: 1312.7482s speed: 0.1312s/iter; left time: 1299.0108s speed: 0.1313s/iter; left time: 1286.7188s speed: 0.1312s/iter; left time: 1273.0868s speed: 0.1312s/iter; left time: 1260.0926s speed: 0.1316s/iter; left time: 1250.9483s speed: 0.1335s/iter; left time: 1254.8993s speed: 0.1328s/iter; left time: 1235.7682s speed: 0.1308s/iter; left time: 1204.1539s speed: 0.1309s/iter; left time: 1191.3958s speed: 0.1309s/iter; left time: 1178.2278s speed: 0.1322s/iter; left time: 1176.9927s speed: 0.1315s/iter; left time: 1157.5977s speed: 0.1315s/iter; left time: 1144.1452s speed: 0.1314s/iter; left time: 1130.5234s speed: 0.1314s/iter; left time: 1117.5005s speed: 0.1314s/iter; left time: 1104.2098s speed: 0.1314s/iter; left time: 1090.8631s speed: 0.1315s/iter; left time: 1078.5724s speed: 0.1314s/iter; left time: 1064.4714s speed: 0.1315s/iter; left time: 1052.1539s speed: 0.1353s/iter; left time: 1069.2737s Epoch: 1 cost time: 517.9919922351837 Epoch: 1, Steps: 3934 | Train Loss: -47.0414631 Vali Loss: -47.4802924 Validation loss decreased (inf --> -47.480292). Saving model ... Updating learning rate to 0.0001 speed: 0.4840s/iter; left time: 3759.9627s speed: 0.1332s/iter; left time: 1021.6347s speed: 0.1323s/iter; left time: 1001.5881s speed: 0.1319s/iter; left time: 985.3198s speed: 0.1345s/iter; left time: 990.9520s speed: 0.1343s/iter; left time: 976.3390s speed: 0.1390s/iter; left time: 996.3020s speed: 0.1346s/iter; left time: 951.4199s speed: 0.1360s/iter; left time: 947.8869s speed: 0.1341s/iter; left time: 921.2710s speed: 0.1379s/iter; left time: 933.7204s speed: 0.1326s/iter; left time: 883.9982s speed: 0.1394s/iter; left time: 915.8344s speed: 0.1355s/iter; left time: 876.6080s speed: 0.1428s/iter; left time: 909.5229s speed: 0.1436s/iter; left time: 900.0828s speed: 0.1437s/iter; left time: 886.3455s speed: 0.1433s/iter; left time: 869.4626s speed: 0.1445s/iter; left time: 862.6468s speed: 0.1449s/iter; left time: 850.2270s speed: 0.1418s/iter; left time: 818.0399s speed: 0.1367s/iter; left time: 774.7034s speed: 0.1367s/iter; left time: 761.4599s speed: 0.1367s/iter; left time: 747.4475s speed: 0.1364s/iter; left time: 732.2058s speed: 0.1368s/iter; left time: 721.0366s speed: 0.1367s/iter; left time: 706.8142s speed: 0.1367s/iter; left time: 693.1174s speed: 0.1367s/iter; left time: 679.1568s speed: 0.1367s/iter; left time: 665.7158s speed: 0.1368s/iter; left time: 652.5808s speed: 0.1367s/iter; left time: 638.4532s speed: 0.1362s/iter; left time: 622.4852s speed: 0.1364s/iter; left time: 609.4713s speed: 0.1364s/iter; left time: 595.9573s speed: 0.1364s/iter; left time: 582.4890s speed: 0.1364s/iter; left time: 568.5825s speed: 0.1364s/iter; left time: 554.9945s speed: 0.1365s/iter; left time: 541.9608s Epoch: 2 cost time: 539.6562712192535 Epoch: 2, Steps: 3934 | Train Loss: -48.5144279 Vali Loss: -48.1329151 EarlyStopping counter: 1 out of 3 Updating learning rate to 5e-05 speed: 0.4813s/iter; left time: 1845.6853s speed: 0.1364s/iter; left time: 509.2899s speed: 0.1362s/iter; left time: 495.1138s speed: 0.1364s/iter; left time: 482.0215s speed: 0.1365s/iter; left time: 468.8174s speed: 0.1364s/iter; left time: 455.0160s speed: 0.1364s/iter; left time: 441.3130s speed: 0.1364s/iter; left time: 427.6231s speed: 0.1364s/iter; left time: 414.0394s speed: 0.1364s/iter; left time: 400.4002s speed: 0.1363s/iter; left time: 386.2983s speed: 0.1363s/iter; left time: 372.6786s speed: 0.1364s/iter; left time: 359.5014s speed: 0.1364s/iter; left time: 345.8320s speed: 0.1364s/iter; left time: 332.1094s speed: 0.1363s/iter; left time: 318.1958s speed: 0.1364s/iter; left time: 304.7477s speed: 0.1362s/iter; left time: 290.8440s speed: 0.1366s/iter; left time: 277.9560s speed: 0.1363s/iter; left time: 263.7950s speed: 0.1364s/iter; left time: 250.2387s speed: 0.1364s/iter; left time: 236.6038s speed: 0.1363s/iter; left time: 222.8828s speed: 0.1363s/iter; left time: 209.2962s speed: 0.1364s/iter; left time: 195.7429s speed: 0.1363s/iter; left time: 181.9378s speed: 0.1364s/iter; left time: 168.4278s speed: 0.1362s/iter; left time: 154.6098s speed: 0.1360s/iter; left time: 140.7701s speed: 0.1364s/iter; left time: 127.5001s speed: 0.1359s/iter; left time: 113.5145s speed: 0.1362s/iter; left time: 100.1328s speed: 0.1363s/iter; left time: 86.5670s speed: 0.1362s/iter; left time: 72.8662s speed: 0.1361s/iter; left time: 59.1943s speed: 0.1363s/iter; left time: 45.6561s speed: 0.1361s/iter; left time: 31.9842s speed: 0.1363s/iter; left time: 18.3962s speed: 0.1364s/iter; left time: 4.7725s Epoch: 3 cost time: 536.1699142456055 Epoch: 3, Steps: 3934 | Train Loss: -48.7206043 Vali Loss: -48.3033336 EarlyStopping counter: 2 out of 3 Updating learning rate to 2.5e-05 ------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD inputc: 122 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 122 pretrainedmodel: 10 winsize: 100 -------------- End ---------------- test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/reduction.py:42: UserWarning: sizeaverage and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.007011290364898737 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.4537, Precision : 0.8757, Recall : 0.0468, F-score : 0.0888

```

r=50.0%

bash ------------ Options ------------- anormly_ratio: 50.0 batch_size: 32 data_path: dataset/NSLKDD dataset: NSLKDD input_c: 122 k: 3 lr: 0.0001 mode: train model_save_path: checkpoints num_epochs: 10 output_c: 122 pretrained_model: None win_size: 100 -------------- End ---------------- test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) ======================TRAIN MODE====================== speed: 0.1355s/iter; left time: 5316.9313s speed: 0.1356s/iter; left time: 5307.0847s speed: 0.1361s/iter; left time: 5313.6697s speed: 0.1319s/iter; left time: 5136.0505s speed: 0.1325s/iter; left time: 5148.3551s speed: 0.1319s/iter; left time: 5109.8676s speed: 0.1323s/iter; left time: 5113.9623s speed: 0.1322s/iter; left time: 5096.4198s speed: 0.1315s/iter; left time: 5054.2963s speed: 0.1322s/iter; left time: 5068.8596s speed: 0.1324s/iter; left time: 5062.5358s speed: 0.1340s/iter; left time: 5111.2675s speed: 0.1310s/iter; left time: 4983.8302s speed: 0.1316s/iter; left time: 4993.7247s speed: 0.1310s/iter; left time: 4958.1371s speed: 0.1310s/iter; left time: 4944.1936s speed: 0.1310s/iter; left time: 4931.2664s speed: 0.1311s/iter; left time: 4920.0523s speed: 0.1310s/iter; left time: 4905.3923s speed: 0.1311s/iter; left time: 4894.8455s speed: 0.1310s/iter; left time: 4879.8760s speed: 0.1311s/iter; left time: 4869.0741s speed: 0.1310s/iter; left time: 4851.3908s speed: 0.1311s/iter; left time: 4841.5871s speed: 0.1311s/iter; left time: 4828.5362s speed: 0.1311s/iter; left time: 4816.3585s speed: 0.1310s/iter; left time: 4800.7019s speed: 0.1309s/iter; left time: 4784.6071s speed: 0.1310s/iter; left time: 4773.5566s speed: 0.1310s/iter; left time: 4761.9220s speed: 0.1310s/iter; left time: 4748.1666s speed: 0.1310s/iter; left time: 4734.0354s speed: 0.1310s/iter; left time: 4720.0640s speed: 0.1310s/iter; left time: 4708.9502s speed: 0.1310s/iter; left time: 4694.6541s speed: 0.1311s/iter; left time: 4685.8344s speed: 0.1309s/iter; left time: 4665.9214s speed: 0.1309s/iter; left time: 4654.0228s speed: 0.1310s/iter; left time: 4642.1672s Epoch: 1 cost time: 518.1424803733826 Epoch: 1, Steps: 3934 | Train Loss: -46.8131543 Vali Loss: -47.3336469 Validation loss decreased (inf --> -47.333647). Saving model ... Updating learning rate to 0.0001 speed: 0.4610s/iter; left time: 16276.7726s speed: 0.1310s/iter; left time: 4612.5694s speed: 0.1310s/iter; left time: 4599.6580s speed: 0.1310s/iter; left time: 4587.0812s speed: 0.1310s/iter; left time: 4572.4280s speed: 0.1312s/iter; left time: 4565.2154s speed: 0.1310s/iter; left time: 4546.2637s speed: 0.1310s/iter; left time: 4534.1559s speed: 0.1310s/iter; left time: 4519.9412s speed: 0.1310s/iter; left time: 4506.3366s speed: 0.1310s/iter; left time: 4493.2462s speed: 0.1309s/iter; left time: 4479.1329s speed: 0.1309s/iter; left time: 4465.1925s speed: 0.1309s/iter; left time: 4451.1384s speed: 0.1309s/iter; left time: 4439.9778s speed: 0.1309s/iter; left time: 4425.5404s speed: 0.1309s/iter; left time: 4412.6608s speed: 0.1309s/iter; left time: 4399.0118s speed: 0.1309s/iter; left time: 4386.5066s speed: 0.1310s/iter; left time: 4374.8416s speed: 0.1309s/iter; left time: 4360.6469s speed: 0.1310s/iter; left time: 4348.4842s speed: 0.1309s/iter; left time: 4333.9263s speed: 0.1309s/iter; left time: 4321.6431s speed: 0.1310s/iter; left time: 4309.9051s speed: 0.1309s/iter; left time: 4295.7049s speed: 0.1309s/iter; left time: 4282.2557s speed: 0.1309s/iter; left time: 4268.6591s speed: 0.1309s/iter; left time: 4256.5062s speed: 0.1309s/iter; left time: 4241.6757s speed: 0.1309s/iter; left time: 4228.4744s speed: 0.1309s/iter; left time: 4215.9839s speed: 0.1309s/iter; left time: 4203.7866s speed: 0.1310s/iter; left time: 4192.8665s speed: 0.1310s/iter; left time: 4179.2397s speed: 0.1310s/iter; left time: 4165.1929s speed: 0.1309s/iter; left time: 4151.0590s speed: 0.1310s/iter; left time: 4140.1505s speed: 0.1310s/iter; left time: 4126.8890s Epoch: 2 cost time: 515.0924828052521 Epoch: 2, Steps: 3934 | Train Loss: -48.4168298 Vali Loss: -47.9248486 EarlyStopping counter: 1 out of 3 Updating learning rate to 5e-05 speed: 0.4590s/iter; left time: 14398.8618s speed: 0.1309s/iter; left time: 4094.5120s speed: 0.1309s/iter; left time: 4079.1813s speed: 0.1309s/iter; left time: 4067.8194s speed: 0.1309s/iter; left time: 4054.6873s speed: 0.1309s/iter; left time: 4040.4983s speed: 0.1309s/iter; left time: 4027.7394s speed: 0.1309s/iter; left time: 4015.9503s speed: 0.1309s/iter; left time: 4002.8357s speed: 0.1319s/iter; left time: 4020.1789s speed: 0.1328s/iter; left time: 4032.3251s speed: 0.1311s/iter; left time: 3969.1873s speed: 0.1311s/iter; left time: 3956.2238s speed: 0.1311s/iter; left time: 3943.2807s speed: 0.1314s/iter; left time: 3937.5022s speed: 0.1317s/iter; left time: 3935.3920s speed: 0.1325s/iter; left time: 3944.7383s speed: 0.1329s/iter; left time: 3942.8348s speed: 0.1313s/iter; left time: 3883.1325s speed: 0.1340s/iter; left time: 3950.0506s speed: 0.1334s/iter; left time: 3919.4091s speed: 0.1309s/iter; left time: 3832.1620s speed: 0.1309s/iter; left time: 3818.9089s speed: 0.1309s/iter; left time: 3806.7406s speed: 0.1309s/iter; left time: 3791.2263s speed: 0.1309s/iter; left time: 3779.5752s speed: 0.1309s/iter; left time: 3765.7227s speed: 0.1309s/iter; left time: 3753.4292s speed: 0.1309s/iter; left time: 3739.4836s speed: 0.1309s/iter; left time: 3726.6624s speed: 0.1309s/iter; left time: 3714.4806s speed: 0.1310s/iter; left time: 3703.1740s speed: 0.1309s/iter; left time: 3688.7920s speed: 0.1309s/iter; left time: 3673.9377s speed: 0.1311s/iter; left time: 3667.6697s speed: 0.1309s/iter; left time: 3649.3432s speed: 0.1310s/iter; left time: 3637.1319s speed: 0.1309s/iter; left time: 3622.2054s speed: 0.1309s/iter; left time: 3608.6637s Epoch: 3 cost time: 516.3497984409332 Epoch: 3, Steps: 3934 | Train Loss: -48.6391877 Vali Loss: -48.1122985 EarlyStopping counter: 2 out of 3 Updating learning rate to 2.5e-05 speed: 0.4590s/iter; left time: 12595.1135s speed: 0.1309s/iter; left time: 3578.3130s speed: 0.1309s/iter; left time: 3566.5477s speed: 0.1309s/iter; left time: 3552.5281s speed: 0.1310s/iter; left time: 3541.1830s speed: 0.1309s/iter; left time: 3526.7139s speed: 0.1309s/iter; left time: 3514.5533s speed: 0.1309s/iter; left time: 3501.3495s speed: 0.1310s/iter; left time: 3490.4514s speed: 0.1309s/iter; left time: 3474.1579s speed: 0.1309s/iter; left time: 3461.5496s speed: 0.1309s/iter; left time: 3448.6966s speed: 0.1309s/iter; left time: 3434.1434s speed: 0.1309s/iter; left time: 3422.2355s speed: 0.1309s/iter; left time: 3407.6903s speed: 0.1310s/iter; left time: 3396.7607s speed: 0.1309s/iter; left time: 3381.6889s speed: 0.1309s/iter; left time: 3369.2955s speed: 0.1309s/iter; left time: 3355.4160s speed: 0.1309s/iter; left time: 3343.8095s speed: 0.1309s/iter; left time: 3329.5964s speed: 0.1309s/iter; left time: 3316.2136s speed: 0.1309s/iter; left time: 3303.9051s speed: 0.1309s/iter; left time: 3290.4389s speed: 0.1309s/iter; left time: 3277.7411s speed: 0.1309s/iter; left time: 3264.3852s speed: 0.1310s/iter; left time: 3254.0794s speed: 0.1310s/iter; left time: 3241.5225s speed: 0.1310s/iter; left time: 3226.8819s speed: 0.1310s/iter; left time: 3213.5240s speed: 0.1310s/iter; left time: 3201.2858s speed: 0.1309s/iter; left time: 3185.9262s speed: 0.1309s/iter; left time: 3172.8487s speed: 0.1309s/iter; left time: 3160.0894s speed: 0.1310s/iter; left time: 3148.5221s speed: 0.1309s/iter; left time: 3133.3825s speed: 0.1309s/iter; left time: 3121.3162s speed: 0.1309s/iter; left time: 3106.9576s speed: 0.1309s/iter; left time: 3095.5211s Epoch: 4 cost time: 514.96653175354 Epoch: 4, Steps: 3934 | Train Loss: -48.7457672 Vali Loss: -48.3127411 EarlyStopping counter: 3 out of 3 Early stopping ------------ Options ------------- anormly_ratio: 50.0 batch_size: 32 data_path: dataset/NSLKDD dataset: NSLKDD input_c: 122 k: 3 lr: 0.0001 mode: test model_save_path: checkpoints num_epochs: 10 output_c: 122 pretrained_model: 20 win_size: 100 -------------- End ---------------- test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.0 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.5032, Precision : 0.7028, Recall : 0.2204, F-score : 0.3356

r=60.0%

bash Threshold : 0.0 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.5284, Precision : 0.6548, Recall : 0.3625, F-score : 0.4666

```bash ------------ Options ------------- anormlyratio: 60.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 10 outputc: 122 pretrainedmodel: None winsize: 100 -------------- End ---------------- test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) ======================TRAIN MODE====================== speed: 0.1388s/iter; left time: 5446.8367s speed: 0.1314s/iter; left time: 5141.6661s speed: 0.1315s/iter; left time: 5133.9673s speed: 0.1315s/iter; left time: 5121.2421s speed: 0.1365s/iter; left time: 5300.2137s speed: 0.1374s/iter; left time: 5322.4496s speed: 0.1333s/iter; left time: 5149.1546s speed: 0.1323s/iter; left time: 5099.9489s speed: 0.1311s/iter; left time: 5038.8110s speed: 0.1310s/iter; left time: 5023.0063s speed: 0.1445s/iter; left time: 5524.1188s speed: 0.1505s/iter; left time: 5740.0504s speed: 0.1497s/iter; left time: 5694.0534s speed: 0.1498s/iter; left time: 5684.9724s speed: 0.1495s/iter; left time: 5657.5665s speed: 0.1501s/iter; left time: 5666.0902s speed: 0.1508s/iter; left time: 5676.4843s speed: 0.1447s/iter; left time: 5432.0509s speed: 0.1438s/iter; left time: 5383.0355s speed: 0.1459s/iter; left time: 5446.3010s speed: 0.1380s/iter; left time: 5138.5641s speed: 0.1528s/iter; left time: 5676.7783s speed: 0.1533s/iter; left time: 5678.8169s speed: 0.1487s/iter; left time: 5494.3238s speed: 0.1487s/iter; left time: 5478.8813s speed: 0.1354s/iter; left time: 4973.0808s speed: 0.1330s/iter; left time: 4874.2198s speed: 0.1327s/iter; left time: 4849.0647s speed: 0.1334s/iter; left time: 4863.0441s speed: 0.1332s/iter; left time: 4840.3917s speed: 0.1392s/iter; left time: 5045.4379s speed: 0.1484s/iter; left time: 5363.5204s speed: 0.1490s/iter; left time: 5370.1684s speed: 0.1318s/iter; left time: 4736.2374s speed: 0.1317s/iter; left time: 4719.7045s speed: 0.1317s/iter; left time: 4705.7837s speed: 0.1317s/iter; left time: 4693.7737s speed: 0.1380s/iter; left time: 4903.0555s speed: 0.1551s/iter; left time: 5496.2793s Epoch: 1 cost time: 553.4214758872986 Epoch: 1, Steps: 3934 | Train Loss: -47.2930476 Vali Loss: -47.4361793 Validation loss decreased (inf --> -47.436179). Saving model ... Updating learning rate to 0.0001 speed: 0.5466s/iter; left time: 19300.5711s speed: 0.1308s/iter; left time: 4605.8315s speed: 0.1311s/iter; left time: 4601.2093s speed: 0.1311s/iter; left time: 4589.1366s speed: 0.1417s/iter; left time: 4945.4468s speed: 0.1421s/iter; left time: 4945.5076s speed: 0.1327s/iter; left time: 4604.5278s speed: 0.1344s/iter; left time: 4650.3773s speed: 0.1396s/iter; left time: 4815.6763s speed: 0.1351s/iter; left time: 4649.1411s speed: 0.1332s/iter; left time: 4568.2611s speed: 0.1401s/iter; left time: 4793.6377s speed: 0.1325s/iter; left time: 4520.7700s speed: 0.1468s/iter; left time: 4991.2005s speed: 0.1412s/iter; left time: 4786.1676s speed: 0.1330s/iter; left time: 4496.7579s speed: 0.1337s/iter; left time: 4508.0182s speed: 0.1333s/iter; left time: 4479.6438s speed: 0.1326s/iter; left time: 4442.6618s speed: 0.1321s/iter; left time: 4413.8824s speed: 0.1310s/iter; left time: 4364.2314s speed: 0.1426s/iter; left time: 4734.3299s speed: 0.1338s/iter; left time: 4430.3676s speed: 0.1325s/iter; left time: 4372.0640s speed: 0.1327s/iter; left time: 4367.8091s speed: 0.1325s/iter; left time: 4345.4439s speed: 0.1327s/iter; left time: 4341.0565s speed: 0.1326s/iter; left time: 4324.5873s speed: 0.1356s/iter; left time: 4406.5620s speed: 0.1464s/iter; left time: 4743.1841s speed: 0.1395s/iter; left time: 4506.8946s speed: 0.1423s/iter; left time: 4583.7955s speed: 0.1461s/iter; left time: 4691.0209s speed: 0.1415s/iter; left time: 4530.2409s speed: 0.1423s/iter; left time: 4538.9798s speed: 0.1390s/iter; left time: 4421.9290s speed: 0.1395s/iter; left time: 4421.7746s speed: 0.1370s/iter; left time: 4330.0266s speed: 0.1368s/iter; left time: 4311.1386s Epoch: 2 cost time: 539.1221182346344 Epoch: 2, Steps: 3934 | Train Loss: -48.4677824 Vali Loss: -47.9757537 EarlyStopping counter: 1 out of 3 Updating learning rate to 5e-05 speed: 0.4911s/iter; left time: 15407.6395s speed: 0.1325s/iter; left time: 4144.1985s speed: 0.1348s/iter; left time: 4202.4603s speed: 0.1361s/iter; left time: 4230.0066s speed: 0.1315s/iter; left time: 4074.3328s speed: 0.1355s/iter; left time: 4182.4872s speed: 0.1483s/iter; left time: 4562.3436s speed: 0.1509s/iter; left time: 4627.7957s speed: 0.1495s/iter; left time: 4572.1388s speed: 0.1501s/iter; left time: 4574.2171s speed: 0.1498s/iter; left time: 4548.7989s speed: 0.1459s/iter; left time: 4416.7031s speed: 0.1436s/iter; left time: 4332.7850s speed: 0.1434s/iter; left time: 4311.9022s speed: 0.1465s/iter; left time: 4390.5848s speed: 0.1476s/iter; left time: 4409.8262s speed: 0.1477s/iter; left time: 4398.3314s speed: 0.1445s/iter; left time: 4286.7148s speed: 0.1463s/iter; left time: 4327.3261s speed: 0.1437s/iter; left time: 4235.5724s speed: 0.1437s/iter; left time: 4219.7474s speed: 0.1462s/iter; left time: 4280.5025s speed: 0.1448s/iter; left time: 4223.6321s speed: 0.1444s/iter; left time: 4198.9182s speed: 0.1446s/iter; left time: 4190.8026s speed: 0.1442s/iter; left time: 4164.7457s speed: 0.1446s/iter; left time: 4159.9640s speed: 0.1440s/iter; left time: 4129.8085s speed: 0.1447s/iter; left time: 4133.1227s speed: 0.1444s/iter; left time: 4111.4583s speed: 0.1449s/iter; left time: 4110.1645s speed: 0.1440s/iter; left time: 4072.0241s speed: 0.1438s/iter; left time: 4050.8166s speed: 0.1444s/iter; left time: 4055.0368s speed: 0.1441s/iter; left time: 4031.9231s speed: 0.1442s/iter; left time: 4020.3876s speed: 0.1445s/iter; left time: 4012.2267s speed: 0.1448s/iter; left time: 4007.0133s speed: 0.1445s/iter; left time: 3985.0134s Epoch: 3 cost time: 565.8869743347168 Epoch: 3, Steps: 3934 | Train Loss: -48.6567995 Vali Loss: -48.1764615 EarlyStopping counter: 2 out of 3 Updating learning rate to 2.5e-05 speed: 0.5192s/iter; left time: 14247.0282s speed: 0.1448s/iter; left time: 3958.9565s speed: 0.1448s/iter; left time: 3943.9087s speed: 0.1443s/iter; left time: 3916.2532s speed: 0.1443s/iter; left time: 3901.8156s speed: 0.1440s/iter; left time: 3880.5454s speed: 0.1444s/iter; left time: 3874.2425s speed: 0.1448s/iter; left time: 3871.7225s speed: 0.1443s/iter; left time: 3842.9239s speed: 0.1441s/iter; left time: 3824.7977s speed: 0.1441s/iter; left time: 3810.2506s speed: 0.1442s/iter; left time: 3797.6073s speed: 0.1442s/iter; left time: 3782.7356s speed: 0.1443s/iter; left time: 3771.7759s speed: 0.1444s/iter; left time: 3759.7137s speed: 0.1440s/iter; left time: 3736.1705s speed: 0.1444s/iter; left time: 3730.0108s speed: 0.1440s/iter; left time: 3707.2472s speed: 0.1445s/iter; left time: 3706.0638s speed: 0.1438s/iter; left time: 3671.9295s speed: 0.1442s/iter; left time: 3669.4479s speed: 0.1446s/iter; left time: 3664.2420s speed: 0.1446s/iter; left time: 3648.3342s speed: 0.1446s/iter; left time: 3634.8287s speed: 0.1474s/iter; left time: 3691.6503s speed: 0.1511s/iter; left time: 3767.4017s speed: 0.1494s/iter; left time: 3711.8324s speed: 0.1441s/iter; left time: 3564.2086s speed: 0.1439s/iter; left time: 3545.6107s speed: 0.1468s/iter; left time: 3603.3738s speed: 0.1452s/iter; left time: 3548.3801s speed: 0.1446s/iter; left time: 3520.4837s speed: 0.1441s/iter; left time: 3491.6876s speed: 0.1440s/iter; left time: 3476.8049s speed: 0.1448s/iter; left time: 3481.9156s speed: 0.1466s/iter; left time: 3508.7926s speed: 0.1447s/iter; left time: 3450.3449s speed: 0.1440s/iter; left time: 3418.7299s speed: 0.1439s/iter; left time: 3401.0288s Epoch: 4 cost time: 569.6859128475189 Epoch: 4, Steps: 3934 | Train Loss: -48.7182953 Vali Loss: -48.2986017 EarlyStopping counter: 3 out of 3 Early stopping ------------ Options ------------- anormlyratio: 60.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD inputc: 122 k: 3 lr: 0.0001 mode: test modelsavepath: checkpoints numepochs: 10 outputc: 122 pretrainedmodel: 20 winsize: 100 -------------- End ---------------- test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) test: (22544, 122) train: (125973, 122) ======================TEST MODE====================== /home/username/anaconda3/envs/Anomaly-Transformer/lib/python3.6/site-packages/torch/nn/reduction.py:42: UserWarning: sizeaverage and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) Threshold : 0.0 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.5284, Precision : 0.6548, Recall : 0.3625, F-score : 0.4666

```

```bash

train ar=0.5%

test ar=60%

Threshold : 8.954137840471525e-22 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.4903, Precision : 0.6855, Recall : 0.1930, F-score : 0.3012

train ar=60%

test ar=60%

Threshold : 1.6401431994555087e-32 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.4899, Precision : 0.6622, Recall : 0.2119, F-score : 0.3210 ```

KNN

OvR

10model checkpoint anomaly ratio

``` ------------ Options ------------- anormlyratio: 20.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD0 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.0 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.7100, Precision : 0.2341, Recall : 0.1776, F-score : 0.2020

------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD1 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.002423033353406936 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9736, Precision : 0.0107, Recall : 0.0094, F-score : 0.0100

------------ Options ------------- anormlyratio: 5.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD2 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 2.4234104793409644e-19 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9197, Precision : 0.0644, Recall : 0.0604, F-score : 0.0623

------------ Options ------------- anormlyratio: 5.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD3 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 2.2883160614427485e-21 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9076, Precision : 0.0818, Recall : 0.0675, F-score : 0.0740

------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD4 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.006830912414006861 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9606, Precision : 0.0408, Recall : 0.0151, F-score : 0.0221

------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD5 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.007120268438011376 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9576, Precision : 0.0259, Recall : 0.0082, F-score : 0.0124

------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD6 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.002397903576493261 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9557, Precision : 0.0319, Recall : 0.0123, F-score : 0.0178

------------ Options ------------- anormlyratio: 0.01 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD7 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.6700110692559966 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9985, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD8 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.006463531367480735 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9737, Precision : 0.0124, Recall : 0.0084, F-score : 0.0100

------------ Options ------------- anormlyratio: 5.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD9 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 9.709074460615489e-17 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9228, Precision : 0.0520, Recall : 0.0487, F-score : 0.0503

------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD10 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.0072950472310184325 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9819, Precision : 0.0127, Recall : 0.0169, F-score : 0.0145

------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD11 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.006282573062926521 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9598, Precision : 0.0256, Recall : 0.0088, F-score : 0.0131

------------ Options ------------- anormlyratio: 0.1 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD12 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.0748524039611232 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9941, Precision : 0.0108, Recall : 0.0244, F-score : 0.0149

------------ Options ------------- anormlyratio: 0.5 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD13 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.028188115973026333 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9874, Precision : 0.0129, Recall : 0.0150, F-score : 0.0139

------------ Options ------------- anormlyratio: 0.5 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD14 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.012299377284944485 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9887, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.01 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD15 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.7865708318292497 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9984, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD16 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.0023502711369655835 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9731, Precision : 0.0036, Recall : 0.0030, F-score : 0.0033

------------ Options ------------- anormlyratio: 0.5 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD17 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.02240190408192611 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9866, Precision : 0.0122, Recall : 0.0142, F-score : 0.0131

------------ Options ------------- anormlyratio: 1.0 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD18 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.007351175076328215 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9767, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.5 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD19 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.023260270589962658 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9855, Precision : 0.0115, Recall : 0.0127, F-score : 0.0121

------------ Options ------------- anormlyratio: 0.05 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD20 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9965, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.05 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD21 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.10992800116911451 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9961, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.05 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD22 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.13109133851528165 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9963, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.01 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD23 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.8262347285803151 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9996, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.05 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD24 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.09242837175354189 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9966, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.01 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD25 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.36379067861726466 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9995, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.05 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD26 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.1089880059286936 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9964, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.05 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD27 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.04048785941675266 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9976, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.01 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD28 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.6861327379285682 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9990, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.01 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD29 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.7651060473798674 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9992, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.01 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD30 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.6199230782323712 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9992, Precision : 0.0000, Recall : 0.0000, F-score : 0.0000

------------ Options ------------- anormlyratio: 0.01 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD31 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ---------------- Threshold : 0.5765992528795936 pred: (22500,) gt: (22500,) pred: (22500,) gt: (22500,) Accuracy : 0.9993, Precision : 0.0714, Recall : 0.2500, F-score : 0.1111

------------ Options ------------- anormlyratio: 0.01 batchsize: 32 datapath: dataset/NSLKDD dataset: NSLKDD32 inputc: 122 k: 3 lr: 0.0001 mode: train modelsavepath: checkpoints numepochs: 3 outputc: 122 pretrainedmodel: None win_size: 100 -------------- End ----------------

```

4. Optuna

bash pip install optuna pip install optuna-dashboard

NSLKDD88GPU bash python optuna_optimization.py --cuda=0 --dataset=NSLKDD --n_trials=30 --host & python optuna_optimization.py --cuda=1 --dataset=NSLKDD --n_trials=30 & python optuna_optimization.py --cuda=2 --dataset=NSLKDD --n_trials=30 & python optuna_optimization.py --cuda=3 --dataset=NSLKDD --n_trials=30 & python optuna_optimization.py --cuda=4 --dataset=NSLKDD --n_trials=30 & python optuna_optimization.py --cuda=5 --dataset=NSLKDD --n_trials=30 & python optuna_optimization.py --cuda=6 --dataset=NSLKDD --n_trials=30 & python optuna_optimization.py --cuda=7 --dataset=NSLKDD --n_trials=30

web bash optuna-dashboard sqlite:///db.sqlite3

db.sqlite3anomaly_transformer_swat_study.db

x. References

  • 1https://zhuanlan.zhihu.com/p/553509779
  • Detection Adjustment:https://blog.csdn.net/a571625338/article/details/127979281

  • https://zhuanlan.zhihu.com/p/149130456

  • KDD991https://cloud.tencent.com/developer/article/1621977

  • debug

    • CodeBERTbughttps://juejin.cn/post/7034105242841153550
    • TransformerDeepBugbughttps://arxiv.org/pdf/2105.09352.pdf
    • benchmarkBugs2fixhttps://github.com/microsoft/CodeXGLUE
    • benchmarkhttps://www.msra.cn/zh-cn/news/features/codexglue

Owner

  • Name: SZU-AdvTech-2023
  • Login: SZU-AdvTech-2023
  • Kind: organization

GitHub Events

Total
Last Year