job_logs/job-gpu-5bfd17d51afd943016b451b9.log - poetry (549e5c8) - (('Mo Repos',), {'htdigest_file': None, 'use_smarthttp': 0, 'require_browser_auth': 0, 'disable_push': 0, 'unauthenticated_push': 0, 'ctags

2018-11-27T10:09:39.680843461Z SYSTEM: Preparing env...
2018-11-27T10:09:39.97799912Z SYSTEM: Running...
2018-11-27T10:09:42.917073854Z Writing to /home/jovyan/work/results/output_poem/tensorboard_log/1543313382
2018-11-27T10:09:42.917115651Z 
2018-11-27T10:09:42.921576408Z ============================================================
2018-11-27T10:09:42.921590985Z All final and intermediate outputs will be stored in /home/jovyan/work/results/output_poem/
2018-11-27T10:09:42.92160796Z ============================================================
2018-11-27T10:09:42.921612797Z 
2018-11-27T10:09:42.921615725Z 06:09:42 INFO:args are:
2018-11-27T10:09:42.921620075Z Namespace(batch_size=16, best_model='', best_valid_ppl=inf, cell_type='lstm', data_path='/home/jovyan/work/data/poem/', debug=False, dropout=0.0, embedding_size=128, encoding='utf-8', hidden_size=128, init_dir='', init_model='', input_dropout=0.0, learning_rate=0.005, max_grad_norm=5.0, num_epochs=5, num_layers=2, num_unrollings=64, output_dir='/home/jovyan/work/results/output_poem', progress_freq=100, save_best_model='/home/jovyan/work/results/output_poem/best_model/model', save_model='/home/jovyan/work/results/output_poem/save_model/model', tb_log_dir='/home/jovyan/work/results/output_poem/tensorboard_log/1543313382', test=False, train_frac=0.9, valid_frac=0.05, verbose=0)
2018-11-27T10:09:42.921625405Z 06:09:42 INFO:Parameters are:
2018-11-27T10:09:42.92162832Z {
2018-11-27T10:09:42.921631027Z     "batch_size": 16,
2018-11-27T10:09:42.921634151Z     "cell_type": "lstm",
2018-11-27T10:09:42.92163743Z     "dropout": 0.0,
2018-11-27T10:09:42.921640321Z     "embedding_size": 128,
2018-11-27T10:09:42.921643287Z     "hidden_size": 128,
2018-11-27T10:09:42.92164597Z     "input_dropout": 0.0,
2018-11-27T10:09:42.921648802Z     "learning_rate": 0.005,
2018-11-27T10:09:42.921651533Z     "max_grad_norm": 5.0,
2018-11-27T10:09:42.921654202Z     "num_layers": 2,
2018-11-27T10:09:42.921656955Z     "num_unrollings": 64
2018-11-27T10:09:42.921659795Z }
2018-11-27T10:09:42.921662383Z 
2018-11-27T10:09:43.077707805Z tensor_file:/home/jovyan/work/data/poem/poem_ids.txt
2018-11-27T10:09:43.077982531Z Loading dataset from /home/jovyan/work/data/poem/poem_ids.txt
2018-11-27T10:09:43.388716364Z file maxSeqLen = 64
2018-11-27T10:09:43.390863888Z Loaded /home/jovyan/work/data/poem/:  training  samples:65235 ,validationSamples:3837,testingSamples:7676
2018-11-27T10:09:43.899007416Z tensor_file:/home/jovyan/work/data/poem/poem_ids.txt
2018-11-27T10:09:43.89904835Z Loading dataset from /home/jovyan/work/data/poem/poem_ids.txt
2018-11-27T10:09:44.240973562Z file maxSeqLen = 64
2018-11-27T10:09:44.241007275Z Loaded /home/jovyan/work/data/poem/:  training  samples:65235 ,validationSamples:3837,testingSamples:7676
2018-11-27T10:09:44.299236281Z tensor_file:/home/jovyan/work/data/poem/poem_ids.txt
2018-11-27T10:09:44.299265026Z Loading dataset from /home/jovyan/work/data/poem/poem_ids.txt
2018-11-27T10:09:44.770941043Z file maxSeqLen = 64
2018-11-27T10:09:44.770973167Z Loaded /home/jovyan/work/data/poem/:  training  samples:65235 ,validationSamples:3837,testingSamples:7676
2018-11-27T10:09:44.791900509Z 06:09:44 INFO:Creating graph
2018-11-27T10:09:55.183177953Z 06:09:55 INFO:Start training
2018-11-27T10:09:55.183218136Z 
2018-11-27T10:09:55.184259003Z 2018-11-27 18:09:55.183876: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-11-27T10:09:55.373253059Z 2018-11-27 18:09:55.373049: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-11-27T10:09:55.375447405Z 2018-11-27 18:09:55.375214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
2018-11-27T10:09:55.375461183Z name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
2018-11-27T10:09:55.37546483Z pciBusID: 0000:00:07.0
2018-11-27T10:09:55.375467846Z totalMemory: 15.90GiB freeMemory: 15.61GiB
2018-11-27T10:09:55.375470809Z 2018-11-27 18:09:55.375257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2018-11-27T10:09:56.360845496Z 2018-11-27 18:09:56.360051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-11-27T10:09:56.360894668Z 2018-11-27 18:09:56.360088: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2018-11-27T10:09:56.360903616Z 2018-11-27 18:09:56.360094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2018-11-27T10:09:56.360909021Z 2018-11-27 18:09:56.360448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15129 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:07.0, compute capability: 6.0)
2018-11-27T10:10:01.217951495Z 06:10:01 INFO:=================== Epoch 0 ===================
2018-11-27T10:10:01.217992526Z 
2018-11-27T10:10:01.217997421Z 06:10:01 INFO:Training on training set
2018-11-27T10:10:11.032344661Z 06:10:11 INFO:2.5%, step:99, perplexity: 579.997, speed: 10555 words
2018-11-27T10:10:15.540573754Z 06:10:15 INFO:4.9%, step:199, perplexity: 365.581, speed: 14412 words
2018-11-27T10:10:19.868017238Z 06:10:19 INFO:7.4%, step:299, perplexity: 280.805, speed: 16569 words
2018-11-27T10:10:24.319889409Z 06:10:24 INFO:9.8%, step:399, perplexity: 234.879, speed: 17815 words
2018-11-27T10:10:28.914214196Z 06:10:28 INFO:12.3%, step:499, perplexity: 205.698, speed: 18560 words
2018-11-27T10:10:33.221910427Z 06:10:33 INFO:14.7%, step:599, perplexity: 184.924, speed: 19264 words
2018-11-27T10:10:37.499999909Z 06:10:37 INFO:17.2%, step:699, perplexity: 169.067, speed: 19817 words
2018-11-27T10:10:41.782939554Z 06:10:41 INFO:19.6%, step:799, perplexity: 156.408, speed: 20250 words
2018-11-27T10:10:46.086558656Z 06:10:46 INFO:22.1%, step:899, perplexity: 146.158, speed: 20591 words
2018-11-27T10:10:50.391412226Z 06:10:50 INFO:24.5%, step:999, perplexity: 137.768, speed: 20871 words
2018-11-27T10:10:54.836464541Z 06:10:54 INFO:27.0%, step:1099, perplexity: 130.760, speed: 21052 words
2018-11-27T10:10:59.273939872Z 06:10:59 INFO:29.4%, step:1199, perplexity: 124.744, speed: 21206 words
2018-11-27T10:11:03.674085815Z 06:11:03 INFO:31.9%, step:1299, perplexity: 119.542, speed: 21352 words
2018-11-27T10:11:08.11125808Z 06:11:08 INFO:34.3%, step:1399, perplexity: 114.989, speed: 21466 words
2018-11-27T10:11:12.379560338Z 06:11:12 INFO:36.8%, step:1499, perplexity: 110.970, speed: 21618 words
2018-11-27T10:11:16.762358631Z 06:11:16 INFO:39.2%, step:1599, perplexity: 107.408, speed: 21720 words
2018-11-27T10:11:21.049873647Z 06:11:21 INFO:41.7%, step:1699, perplexity: 104.224, speed: 21836 words
2018-11-27T10:11:25.449693668Z 06:11:25 INFO:44.1%, step:1799, perplexity: 101.359, speed: 21911 words
2018-11-27T10:11:29.832817937Z 06:11:29 INFO:46.6%, step:1899, perplexity: 98.749, speed: 21983 words
2018-11-27T10:11:34.45785477Z 06:11:34 INFO:49.0%, step:1999, perplexity: 96.351, speed: 21991 words
2018-11-27T10:11:38.701768996Z 06:11:38 INFO:51.5%, step:2099, perplexity: 94.168, speed: 22084 words
2018-11-27T10:11:42.84874719Z 06:11:42 INFO:53.9%, step:2199, perplexity: 92.176, speed: 22190 words
2018-11-27T10:11:47.237563225Z 06:11:47 INFO:56.4%, step:2299, perplexity: 90.341, speed: 22238 words
2018-11-27T10:11:51.334678139Z 06:11:51 INFO:58.9%, step:2399, perplexity: 88.632, speed: 22340 words
2018-11-27T10:11:55.887346818Z 06:11:55 INFO:61.3%, step:2499, perplexity: 87.042, speed: 22347 words
2018-11-27T10:12:00.221511695Z 06:12:00 INFO:63.8%, step:2599, perplexity: 85.542, speed: 22393 words
2018-11-27T10:12:04.577282729Z 06:12:04 INFO:66.2%, step:2699, perplexity: 84.125, speed: 22433 words
2018-11-27T10:12:09.192723567Z 06:12:09 INFO:68.7%, step:2799, perplexity: 82.786, speed: 22424 words
2018-11-27T10:12:13.340539137Z 06:12:13 INFO:71.1%, step:2899, perplexity: 81.517, speed: 22495 words
2018-11-27T10:12:17.551625495Z 06:12:17 INFO:73.6%, step:2999, perplexity: 80.310, speed: 22551 words
2018-11-27T10:12:21.821009645Z 06:12:21 INFO:76.0%, step:3099, perplexity: 79.170, speed: 22595 words
2018-11-27T10:12:26.07197127Z 06:12:26 INFO:78.5%, step:3199, perplexity: 78.088, speed: 22638 words
2018-11-27T10:12:30.401157218Z 06:12:30 INFO:80.9%, step:3299, perplexity: 77.054, speed: 22668 words
2018-11-27T10:12:34.649919317Z 06:12:34 INFO:83.4%, step:3399, perplexity: 76.065, speed: 22708 words
2018-11-27T10:12:39.034718148Z 06:12:39 INFO:85.8%, step:3499, perplexity: 75.120, speed: 22726 words
2018-11-27T10:12:43.378686867Z 06:12:43 INFO:88.3%, step:3599, perplexity: 74.220, speed: 22749 words
2018-11-27T10:12:47.9802102Z 06:12:47 INFO:90.7%, step:3699, perplexity: 73.360, speed: 22735 words
2018-11-27T10:12:52.460013383Z 06:12:52 INFO:93.2%, step:3799, perplexity: 72.535, speed: 22738 words
2018-11-27T10:12:56.782595052Z 06:12:56 INFO:95.6%, step:3899, perplexity: 71.743, speed: 22761 words
2018-11-27T10:13:01.210315439Z 06:13:01 INFO:98.1%, step:3999, perplexity: 70.985, speed: 22771 words
2018-11-27T10:13:04.515673325Z 06:13:04 INFO:Perplexity: 41.065, speed: 22795 words per sec
2018-11-27T10:13:06.383265041Z 06:13:06 INFO:Latest model saved in /home/jovyan/work/results/output_poem/save_model/model-4078
2018-11-27T10:13:06.38335135Z 
2018-11-27T10:13:06.383361623Z 06:13:06 INFO:Evaluate on validation set
2018-11-27T10:13:10.150923351Z 06:13:10 INFO:41.7%, step:99, perplexity: 27.212, speed: 31566 words
2018-11-27T10:13:12.141522115Z 06:13:12 INFO:83.3%, step:199, perplexity: 27.008, speed: 39128 words
2018-11-27T10:13:12.852132512Z 06:13:12 INFO:Perplexity: 27.770, speed: 41338 words per sec
2018-11-27T10:13:14.134856897Z 06:13:14 INFO:Best model is saved in /home/jovyan/work/results/output_poem/best_model/model-4078
2018-11-27T10:13:14.134886734Z 06:13:14 INFO:Best validation ppl is 27.769833
2018-11-27T10:13:14.134891404Z 
2018-11-27T10:13:14.136491627Z 06:13:14 INFO:=================== Epoch 1 ===================
2018-11-27T10:13:14.136507328Z 
2018-11-27T10:13:14.136512125Z 06:13:14 INFO:Training on training set
2018-11-27T10:13:18.37093865Z 06:13:18 INFO:2.5%, step:99, perplexity: 31.367, speed: 24195 words
2018-11-27T10:13:22.67200999Z 06:13:22 INFO:4.9%, step:199, perplexity: 30.462, speed: 23999 words
2018-11-27T10:13:27.100723019Z 06:13:27 INFO:7.4%, step:299, perplexity: 30.253, speed: 23701 words
2018-11-27T10:13:31.506242791Z 06:13:31 INFO:9.8%, step:399, perplexity: 30.191, speed: 23585 words
2018-11-27T10:13:35.754251214Z 06:13:35 INFO:12.3%, step:499, perplexity: 30.156, speed: 23686 words
2018-11-27T10:13:40.123424997Z 06:13:40 INFO:14.7%, step:599, perplexity: 30.072, speed: 23644 words
2018-11-27T10:13:44.805115792Z 06:13:44 INFO:17.2%, step:699, perplexity: 29.913, speed: 23374 words
2018-11-27T10:13:49.109015968Z 06:13:49 INFO:19.6%, step:799, perplexity: 29.684, speed: 23425 words
2018-11-27T10:13:53.444719904Z 06:13:53 INFO:22.1%, step:899, perplexity: 29.479, speed: 23447 words
2018-11-27T10:13:57.70057894Z 06:13:57 INFO:24.5%, step:999, perplexity: 29.336, speed: 23507 words
2018-11-27T10:14:02.035195139Z 06:14:02 INFO:27.0%, step:1099, perplexity: 29.224, speed: 23517 words
2018-11-27T10:14:06.513941916Z 06:14:06 INFO:29.4%, step:1199, perplexity: 29.109, speed: 23461 words
2018-11-27T10:14:10.843937135Z 06:14:10 INFO:31.9%, step:1299, perplexity: 29.007, speed: 23476 words
2018-11-27T10:14:15.100460237Z 06:14:15 INFO:34.3%, step:1399, perplexity: 28.916, speed: 23517 words
2018-11-27T10:14:19.328567505Z 06:14:19 INFO:36.8%, step:1499, perplexity: 28.837, speed: 23562 words
2018-11-27T10:14:23.602516105Z 06:14:23 INFO:39.2%, step:1599, perplexity: 28.775, speed: 23587 words
2018-11-27T10:14:27.891736465Z 06:14:27 INFO:41.7%, step:1699, perplexity: 28.724, speed: 23603 words
2018-11-27T10:14:32.264798343Z 06:14:32 INFO:44.1%, step:1799, perplexity: 28.683, speed: 23593 words
2018-11-27T10:14:36.751389471Z 06:14:36 INFO:46.6%, step:1899, perplexity: 28.640, speed: 23551 words
2018-11-27T10:14:40.963914224Z 06:14:40 INFO:49.0%, step:1999, perplexity: 28.594, speed: 23588 words
2018-11-27T10:14:45.299287048Z 06:14:45 INFO:51.5%, step:2099, perplexity: 28.562, speed: 23589 words
2018-11-27T10:14:49.698047951Z 06:14:49 INFO:53.9%, step:2199, perplexity: 28.542, speed: 23575 words
2018-11-27T10:14:54.334874276Z 06:14:54 INFO:56.4%, step:2299, perplexity: 28.526, speed: 23507 words
2018-11-27T10:14:58.855924359Z 06:14:58 INFO:58.9%, step:2399, perplexity: 28.509, speed: 23469 words
2018-11-27T10:15:03.431963114Z 06:15:03 INFO:61.3%, step:2499, perplexity: 28.493, speed: 23423 words
2018-11-27T10:15:07.994242671Z 06:15:07 INFO:63.8%, step:2599, perplexity: 28.469, speed: 23384 words
2018-11-27T10:15:12.421086797Z 06:15:12 INFO:66.2%, step:2699, perplexity: 28.439, speed: 23375 words
2018-11-27T10:15:16.630874054Z 06:15:16 INFO:68.7%, step:2799, perplexity: 28.404, speed: 23407 words
2018-11-27T10:15:20.951036643Z 06:15:20 INFO:71.1%, step:2899, perplexity: 28.367, speed: 23417 words
2018-11-27T10:15:25.27899812Z 06:15:25 INFO:73.6%, step:2999, perplexity: 28.325, speed: 23425 words
2018-11-27T10:15:29.775674058Z 06:15:29 INFO:76.0%, step:3099, perplexity: 28.286, speed: 23404 words
2018-11-27T10:15:34.126985091Z 06:15:34 INFO:78.5%, step:3199, perplexity: 28.248, speed: 23408 words
2018-11-27T10:15:38.464845561Z 06:15:38 INFO:80.9%, step:3299, perplexity: 28.207, speed: 23415 words
2018-11-27T10:15:42.990394552Z 06:15:42 INFO:83.4%, step:3399, perplexity: 28.165, speed: 23390 words
2018-11-27T10:15:47.425927234Z 06:15:47 INFO:85.8%, step:3499, perplexity: 28.123, speed: 23381 words
2018-11-27T10:15:51.87598534Z 06:15:51 INFO:88.3%, step:3599, perplexity: 28.083, speed: 23371 words
2018-11-27T10:15:56.338438287Z 06:15:56 INFO:90.7%, step:3699, perplexity: 28.046, speed: 23359 words
2018-11-27T10:16:00.900162713Z 06:16:00 INFO:93.2%, step:3799, perplexity: 28.009, speed: 23334 words
2018-11-27T10:16:05.255897951Z 06:16:05 INFO:95.6%, step:3899, perplexity: 27.972, speed: 23339 words
2018-11-27T10:16:09.663934234Z 06:16:09 INFO:98.1%, step:3999, perplexity: 27.937, speed: 23336 words
2018-11-27T10:16:13.21823541Z 06:16:13 INFO:Perplexity: 26.581, speed: 23319 words per sec
2018-11-27T10:16:14.410917929Z 06:16:14 INFO:Latest model saved in /home/jovyan/work/results/output_poem/save_model/model-8156
2018-11-27T10:16:14.410963935Z 
2018-11-27T10:16:14.410971269Z 06:16:14 INFO:Evaluate on validation set
2018-11-27T10:16:17.491999462Z 06:16:17 INFO:41.7%, step:99, perplexity: 23.190, speed: 37610 words
2018-11-27T10:16:19.430069711Z 06:16:19 INFO:83.3%, step:199, perplexity: 23.023, speed: 43960 words
2018-11-27T10:16:20.172559347Z 06:16:20 INFO:Perplexity: 23.695, speed: 45476 words per sec
2018-11-27T10:16:21.32888922Z 06:16:21 INFO:Best model is saved in /home/jovyan/work/results/output_poem/best_model/model-8156
2018-11-27T10:16:21.328933688Z 06:16:21 INFO:Best validation ppl is 23.695059
2018-11-27T10:16:21.328941622Z 
2018-11-27T10:16:21.331146965Z 06:16:21 INFO:=================== Epoch 2 ===================
2018-11-27T10:16:21.331195602Z 
2018-11-27T10:16:21.33120213Z 06:16:21 INFO:Training on training set
2018-11-27T10:16:25.822634972Z 06:16:25 INFO:2.5%, step:99, perplexity: 26.108, speed: 22809 words
2018-11-27T10:16:30.154094196Z 06:16:30 INFO:4.9%, step:199, perplexity: 25.357, speed: 23218 words
2018-11-27T10:16:34.519838458Z 06:16:34 INFO:7.4%, step:299, perplexity: 25.189, speed: 23296 words
2018-11-27T10:16:38.863471994Z 06:16:38 INFO:9.8%, step:399, perplexity: 25.155, speed: 23365 words
2018-11-27T10:16:43.276922221Z 06:16:43 INFO:12.3%, step:499, perplexity: 25.147, speed: 23338 words
2018-11-27T10:16:47.667144852Z 06:16:47 INFO:14.7%, step:599, perplexity: 25.098, speed: 23331 words
2018-11-27T10:16:51.989385556Z 06:16:51 INFO:17.2%, step:699, perplexity: 24.986, speed: 23383 words
2018-11-27T10:16:56.648716698Z 06:16:56 INFO:19.6%, step:799, perplexity: 24.814, speed: 23197 words
2018-11-27T10:17:00.896572508Z 06:17:00 INFO:22.1%, step:899, perplexity: 24.662, speed: 23294 words
2018-11-27T10:17:05.339943441Z 06:17:05 INFO:24.5%, step:999, perplexity: 24.559, speed: 23270 words
2018-11-27T10:17:09.829883397Z 06:17:09 INFO:27.0%, step:1099, perplexity: 24.483, speed: 23227 words
2018-11-27T10:17:14.132195825Z 06:17:14 INFO:29.4%, step:1199, perplexity: 24.404, speed: 23274 words
2018-11-27T10:17:18.893229243Z 06:17:18 INFO:31.9%, step:1299, perplexity: 24.337, speed: 23127 words
2018-11-27T10:17:23.198893748Z 06:17:23 INFO:34.3%, step:1399, perplexity: 24.279, speed: 23173 words
2018-11-27T10:17:27.691213117Z 06:17:27 INFO:36.8%, step:1499, perplexity: 24.230, speed: 23147 words
2018-11-27T10:17:32.023904568Z 06:17:32 INFO:39.2%, step:1599, perplexity: 24.197, speed: 23177 words
2018-11-27T10:17:36.622163391Z 06:17:36 INFO:41.7%, step:1699, perplexity: 24.174, speed: 23122 words
2018-11-27T10:17:40.874052641Z 06:17:40 INFO:44.1%, step:1799, perplexity: 24.158, speed: 23173 words
2018-11-27T10:17:45.210869425Z 06:17:45 INFO:46.6%, step:1899, perplexity: 24.142, speed: 23196 words
2018-11-27T10:17:49.66595633Z 06:17:49 INFO:49.0%, step:1999, perplexity: 24.121, speed: 23185 words
2018-11-27T10:17:54.114666252Z 06:17:54 INFO:51.5%, step:2099, perplexity: 24.112, speed: 23177 words
2018-11-27T10:17:58.506224862Z 06:17:58 INFO:53.9%, step:2199, perplexity: 24.113, speed: 23183 words
2018-11-27T10:18:02.778089366Z 06:18:02 INFO:56.4%, step:2299, perplexity: 24.118, speed: 23217 words
2018-11-27T10:18:07.230983119Z 06:18:07 INFO:58.9%, step:2399, perplexity: 24.121, speed: 23209 words
2018-11-27T10:18:12.027530948Z 06:18:12 INFO:61.3%, step:2499, perplexity: 24.125, speed: 23127 words
2018-11-27T10:18:16.407152069Z 06:18:16 INFO:63.8%, step:2599, perplexity: 24.122, speed: 23137 words
2018-11-27T10:18:20.835552933Z 06:18:20 INFO:66.2%, step:2699, perplexity: 24.114, speed: 23136 words
2018-11-27T10:18:25.445355239Z 06:18:25 INFO:68.7%, step:2799, perplexity: 24.103, speed: 23102 words
2018-11-27T10:18:29.688021836Z 06:18:29 INFO:71.1%, step:2899, perplexity: 24.088, speed: 23136 words
2018-11-27T10:18:33.953928626Z 06:18:33 INFO:73.6%, step:2999, perplexity: 24.070, speed: 23164 words
2018-11-27T10:18:38.383252436Z 06:18:38 INFO:76.0%, step:3099, perplexity: 24.054, speed: 23163 words
2018-11-27T10:18:42.893777871Z 06:18:42 INFO:78.5%, step:3199, perplexity: 24.038, speed: 23149 words
2018-11-27T10:18:47.31285914Z 06:18:47 INFO:80.9%, step:3299, perplexity: 24.020, speed: 23149 words
2018-11-27T10:18:51.576226913Z 06:18:51 INFO:83.4%, step:3399, perplexity: 24.001, speed: 23173 words
2018-11-27T10:18:55.943989507Z 06:18:55 INFO:85.8%, step:3499, perplexity: 23.981, speed: 23181 words
2018-11-27T10:19:00.411905354Z 06:19:00 INFO:88.3%, step:3599, perplexity: 23.964, speed: 23174 words
2018-11-27T10:19:04.863901015Z 06:19:04 INFO:90.7%, step:3699, perplexity: 23.948, speed: 23169 words
2018-11-27T10:19:09.275138043Z 06:19:09 INFO:93.2%, step:3799, perplexity: 23.932, speed: 23170 words
2018-11-27T10:19:13.679228985Z 06:19:13 INFO:95.6%, step:3899, perplexity: 23.917, speed: 23172 words
2018-11-27T10:19:18.070953998Z 06:19:18 INFO:98.1%, step:3999, perplexity: 23.902, speed: 23176 words
2018-11-27T10:19:21.512945327Z 06:19:21 INFO:Perplexity: 23.384, speed: 23176 words per sec
2018-11-27T10:19:22.751208491Z 06:19:22 INFO:Latest model saved in /home/jovyan/work/results/output_poem/save_model/model-12234
2018-11-27T10:19:22.751259163Z 
2018-11-27T10:19:22.751265813Z 06:19:22 INFO:Evaluate on validation set
2018-11-27T10:19:26.027680038Z 06:19:26 INFO:41.7%, step:99, perplexity: 22.205, speed: 35042 words
2018-11-27T10:19:27.900094306Z 06:19:27 INFO:83.3%, step:199, perplexity: 22.039, speed: 42714 words
2018-11-27T10:19:28.716925585Z 06:19:28 INFO:Perplexity: 22.719, speed: 43791 words per sec
2018-11-27T10:19:30.073269798Z 06:19:30 INFO:Best model is saved in /home/jovyan/work/results/output_poem/best_model/model-12234
2018-11-27T10:19:30.073333517Z 06:19:30 INFO:Best validation ppl is 22.719276
2018-11-27T10:19:30.073345188Z 
2018-11-27T10:19:30.07483804Z 06:19:30 INFO:=================== Epoch 3 ===================
2018-11-27T10:19:30.074854818Z 
2018-11-27T10:19:30.075604575Z 06:19:30 INFO:Training on training set
2018-11-27T10:19:34.542630855Z 06:19:34 INFO:2.5%, step:99, perplexity: 23.989, speed: 22953 words
2018-11-27T10:19:38.991967833Z 06:19:38 INFO:4.9%, step:199, perplexity: 23.323, speed: 22973 words
2018-11-27T10:19:43.373945257Z 06:19:43 INFO:7.4%, step:299, perplexity: 23.167, speed: 23105 words
2018-11-27T10:19:47.800022625Z 06:19:47 INFO:9.8%, step:399, perplexity: 23.135, speed: 23111 words
2018-11-27T10:19:52.116562199Z 06:19:52 INFO:12.3%, step:499, perplexity: 23.130, speed: 23231 words
2018-11-27T10:19:56.633976252Z 06:19:56 INFO:14.7%, step:599, perplexity: 23.090, speed: 23136 words
2018-11-27T10:20:00.949289863Z 06:20:00 INFO:17.2%, step:699, perplexity: 22.991, speed: 23219 words
2018-11-27T10:20:05.290965539Z 06:20:05 INFO:19.6%, step:799, perplexity: 22.838, speed: 23264 words
2018-11-27T10:20:09.612679003Z 06:20:09 INFO:22.1%, step:899, perplexity: 22.705, speed: 23311 words
2018-11-27T10:20:13.943285394Z 06:20:13 INFO:24.5%, step:999, perplexity: 22.619, speed: 23344 words
2018-11-27T10:20:18.344987697Z 06:20:18 INFO:27.0%, step:1099, perplexity: 22.557, speed: 23338 words
2018-11-27T10:20:22.570522059Z 06:20:22 INFO:29.4%, step:1199, perplexity: 22.492, speed: 23409 words
2018-11-27T10:20:26.901527245Z 06:20:26 INFO:31.9%, step:1299, perplexity: 22.437, speed: 23427 words
2018-11-27T10:20:31.059737324Z 06:20:31 INFO:34.3%, step:1399, perplexity: 22.391, speed: 23509 words
2018-11-27T10:20:35.343045386Z 06:20:35 INFO:36.8%, step:1499, perplexity: 22.354, speed: 23539 words
2018-11-27T10:20:40.22345844Z 06:20:40 INFO:39.2%, step:1599, perplexity: 22.329, speed: 23357 words
2018-11-27T10:20:44.557509946Z 06:20:44 INFO:41.7%, step:1699, perplexity: 22.314, speed: 23373 words
2018-11-27T10:20:48.708852989Z 06:20:48 INFO:44.1%, step:1799, perplexity: 22.306, speed: 23441 words
2018-11-27T10:20:53.022344152Z 06:20:53 INFO:46.6%, step:1899, perplexity: 22.297, speed: 23456 words
2018-11-27T10:20:57.37546736Z 06:20:57 INFO:49.0%, step:1999, perplexity: 22.285, speed: 23460 words
2018-11-27T10:21:01.833109767Z 06:21:01 INFO:51.5%, step:2099, perplexity: 22.282, speed: 23436 words
2018-11-27T10:21:06.151597571Z 06:21:06 INFO:53.9%, step:2199, perplexity: 22.289, speed: 23449 words
2018-11-27T10:21:10.828731892Z 06:21:10 INFO:56.4%, step:2299, perplexity: 22.299, speed: 23376 words
2018-11-27T10:21:15.126494472Z 06:21:15 INFO:58.9%, step:2399, perplexity: 22.308, speed: 23395 words
2018-11-27T10:21:19.461160547Z 06:21:19 INFO:61.3%, step:2499, perplexity: 22.317, speed: 23404 words
2018-11-27T10:21:23.793901164Z 06:21:23 INFO:63.8%, step:2599, perplexity: 22.320, speed: 23413 words
2018-11-27T10:21:27.997158767Z 06:21:27 INFO:66.2%, step:2699, perplexity: 22.319, speed: 23447 words
2018-11-27T10:21:32.29960135Z 06:21:32 INFO:68.7%, step:2799, perplexity: 22.314, speed: 23459 words
2018-11-27T10:21:36.655561044Z 06:21:36 INFO:71.1%, step:2899, perplexity: 22.306, speed: 23461 words
2018-11-27T10:21:41.143006311Z 06:21:41 INFO:73.6%, step:2999, perplexity: 22.295, speed: 23439 words
2018-11-27T10:21:45.557130333Z 06:21:45 INFO:76.0%, step:3099, perplexity: 22.286, speed: 23431 words
2018-11-27T10:21:49.887959316Z 06:21:49 INFO:78.5%, step:3199, perplexity: 22.277, speed: 23437 words
2018-11-27T10:21:54.829782389Z 06:21:54 INFO:80.9%, step:3299, perplexity: 22.266, speed: 23345 words
2018-11-27T10:21:59.329112412Z 06:21:59 INFO:83.4%, step:3399, perplexity: 22.254, speed: 23327 words
2018-11-27T10:22:04.015500912Z 06:22:04 INFO:85.8%, step:3499, perplexity: 22.241, speed: 23282 words
2018-11-27T10:22:08.357942505Z 06:22:08 INFO:88.3%, step:3599, perplexity: 22.231, speed: 23290 words
2018-11-27T10:22:13.066890435Z 06:22:13 INFO:90.7%, step:3699, perplexity: 22.222, speed: 23246 words
2018-11-27T10:22:17.645017158Z 06:22:17 INFO:93.2%, step:3799, perplexity: 22.212, speed: 23222 words
2018-11-27T10:22:22.107816071Z 06:22:22 INFO:95.6%, step:3899, perplexity: 22.204, speed: 23215 words
2018-11-27T10:22:26.560310365Z 06:22:26 INFO:98.1%, step:3999, perplexity: 22.196, speed: 23209 words
2018-11-27T10:22:29.903176124Z 06:22:29 INFO:Perplexity: 21.945, speed: 23222 words per sec
2018-11-27T10:22:31.075931565Z 06:22:31 INFO:Latest model saved in /home/jovyan/work/results/output_poem/save_model/model-16312
2018-11-27T10:22:31.075967919Z 
2018-11-27T10:22:31.075972504Z 06:22:31 INFO:Evaluate on validation set
2018-11-27T10:22:34.245962411Z 06:22:34 INFO:41.7%, step:99, perplexity: 22.034, speed: 36666 words
2018-11-27T10:22:36.0676778Z 06:22:36 INFO:83.3%, step:199, perplexity: 21.854, speed: 44384 words
2018-11-27T10:22:36.77789724Z 06:22:36 INFO:Perplexity: 22.599, speed: 46162 words per sec
2018-11-27T10:22:37.860643715Z 06:22:37 INFO:Best model is saved in /home/jovyan/work/results/output_poem/best_model/model-16312
2018-11-27T10:22:37.860691125Z 06:22:37 INFO:Best validation ppl is 22.599243
2018-11-27T10:22:37.860699352Z 
2018-11-27T10:22:37.862282085Z 06:22:37 INFO:=================== Epoch 4 ===================
2018-11-27T10:22:37.862332248Z 
2018-11-27T10:22:37.862339865Z 06:22:37 INFO:Training on training set
2018-11-27T10:22:42.37426273Z 06:22:42 INFO:2.5%, step:99, perplexity: 23.107, speed: 22705 words
2018-11-27T10:22:46.680374852Z 06:22:46 INFO:4.9%, step:199, perplexity: 22.428, speed: 23232 words
2018-11-27T10:22:51.03296342Z 06:22:51 INFO:7.4%, step:299, perplexity: 22.259, speed: 23334 words
2018-11-27T10:22:55.426832231Z 06:22:55 INFO:9.8%, step:399, perplexity: 22.217, speed: 23327 words
2018-11-27T10:22:59.777773075Z 06:22:59 INFO:12.3%, step:499, perplexity: 22.202, speed: 23365 words
2018-11-27T10:23:04.145072125Z 06:23:04 INFO:14.7%, step:599, perplexity: 22.157, speed: 23378 words
2018-11-27T10:23:09.1136806Z 06:23:09 INFO:17.2%, step:699, perplexity: 22.057, speed: 22977 words
2018-11-27T10:23:13.648830834Z 06:23:13 INFO:19.6%, step:799, perplexity: 21.906, speed: 22893 words
2018-11-27T10:23:18.025400022Z 06:23:18 INFO:22.1%, step:899, perplexity: 21.775, speed: 22948 words
2018-11-27T10:23:22.348901926Z 06:23:22 INFO:24.5%, step:999, perplexity: 21.688, speed: 23019 words
2018-11-27T10:23:26.676677066Z 06:23:26 INFO:27.0%, step:1099, perplexity: 21.626, speed: 23076 words
2018-11-27T10:23:30.951963306Z 06:23:30 INFO:29.4%, step:1199, perplexity: 21.561, speed: 23146 words
2018-11-27T10:23:35.232935959Z 06:23:35 INFO:31.9%, step:1299, perplexity: 21.507, speed: 23204 words
2018-11-27T10:23:39.603813529Z 06:23:39 INFO:34.3%, step:1399, perplexity: 21.461, speed: 23221 words
2018-11-27T10:23:44.017493496Z 06:23:44 INFO:36.8%, step:1499, perplexity: 21.424, speed: 23219 words
2018-11-27T10:23:48.326467059Z 06:23:48 INFO:39.2%, step:1599, perplexity: 21.399, speed: 23252 words
2018-11-27T10:23:52.585672907Z 06:23:52 INFO:41.7%, step:1699, perplexity: 21.384, speed: 23298 words
2018-11-27T10:23:56.989893971Z 06:23:56 INFO:44.1%, step:1799, perplexity: 21.376, speed: 23295 words
2018-11-27T10:24:01.393023201Z 06:24:01 INFO:46.6%, step:1899, perplexity: 21.367, speed: 23293 words
2018-11-27T10:24:05.58477811Z 06:24:05 INFO:49.0%, step:1999, perplexity: 21.356, speed: 23347 words
2018-11-27T10:24:09.947020057Z 06:24:09 INFO:51.5%, step:2099, perplexity: 21.354, speed: 23353 words
2018-11-27T10:24:14.412994293Z 06:24:14 INFO:53.9%, step:2199, perplexity: 21.361, speed: 23333 words
2018-11-27T10:24:18.954771216Z 06:24:18 INFO:56.4%, step:2299, perplexity: 21.371, speed: 23298 words
2018-11-27T10:24:23.433249285Z 06:24:23 INFO:58.9%, step:2399, perplexity: 21.380, speed: 23280 words
2018-11-27T10:24:27.670924005Z 06:24:27 INFO:61.3%, step:2499, perplexity: 21.389, speed: 23314 words
2018-11-27T10:24:32.040370971Z 06:24:32 INFO:63.8%, step:2599, perplexity: 21.393, speed: 23318 words
2018-11-27T10:24:36.300870634Z 06:24:36 INFO:66.2%, step:2699, perplexity: 21.393, speed: 23344 words
2018-11-27T10:24:40.600139039Z 06:24:40 INFO:68.7%, step:2799, perplexity: 21.389, speed: 23361 words
2018-11-27T10:24:45.049495157Z 06:24:45 INFO:71.1%, step:2899, perplexity: 21.383, speed: 23349 words
2018-11-27T10:24:49.439408318Z 06:24:49 INFO:73.6%, step:2999, perplexity: 21.374, speed: 23348 words
2018-11-27T10:24:53.802195957Z 06:24:53 INFO:76.0%, step:3099, perplexity: 21.366, speed: 23353 words
2018-11-27T10:24:58.262550854Z 06:24:58 INFO:78.5%, step:3199, perplexity: 21.359, speed: 23339 words
2018-11-27T10:25:02.636673549Z 06:25:02 INFO:80.9%, step:3299, perplexity: 21.350, speed: 23342 words
2018-11-27T10:25:06.982697089Z 06:25:06 INFO:83.4%, step:3399, perplexity: 21.340, speed: 23348 words
2018-11-27T10:25:11.487683521Z 06:25:11 INFO:85.8%, step:3499, perplexity: 21.329, speed: 23330 words
2018-11-27T10:25:15.850716369Z 06:25:15 INFO:88.3%, step:3599, perplexity: 21.321, speed: 23334 words
2018-11-27T10:25:20.217657971Z 06:25:20 INFO:90.7%, step:3699, perplexity: 21.314, speed: 23337 words
2018-11-27T10:25:24.900127389Z 06:25:24 INFO:93.2%, step:3799, perplexity: 21.307, speed: 23296 words
2018-11-27T10:25:29.231327973Z 06:25:29 INFO:95.6%, step:3899, perplexity: 21.300, speed: 23304 words
2018-11-27T10:25:33.590834675Z 06:25:33 INFO:98.1%, step:3999, perplexity: 21.294, speed: 23309 words
2018-11-27T10:25:36.874541841Z 06:25:36 INFO:Perplexity: 21.121, speed: 23327 words per sec
2018-11-27T10:25:37.997983144Z 06:25:37 INFO:Latest model saved in /home/jovyan/work/results/output_poem/save_model/model-20390
2018-11-27T10:25:37.998028676Z 
2018-11-27T10:25:37.99803331Z 06:25:37 INFO:Evaluate on validation set
2018-11-27T10:25:41.034447802Z 06:25:41 INFO:41.7%, step:99, perplexity: 21.705, speed: 38149 words
2018-11-27T10:25:42.95159539Z 06:25:42 INFO:83.3%, step:199, perplexity: 21.544, speed: 44505 words
2018-11-27T10:25:43.688599759Z 06:25:43 INFO:Perplexity: 22.314, speed: 46031 words per sec
2018-11-27T10:25:44.848164248Z 06:25:44 INFO:Best model is saved in /home/jovyan/work/results/output_poem/best_model/model-20390
2018-11-27T10:25:44.848221175Z 06:25:44 INFO:Best validation ppl is 22.313692
2018-11-27T10:25:44.848228199Z 
2018-11-27T10:25:44.849321317Z 06:25:44 INFO:Latest model is saved in /home/jovyan/work/results/output_poem/save_model/model-20390
2018-11-27T10:25:44.849368157Z 06:25:44 INFO:Best model is saved in /home/jovyan/work/results/output_poem/best_model/model-20390
2018-11-27T10:25:44.849375611Z 06:25:44 INFO:Best validation ppl is 22.313692
2018-11-27T10:25:44.849378949Z 
2018-11-27T10:25:44.849382077Z 06:25:44 INFO:Evaluate the best model on test set
2018-11-27T10:25:48.459385356Z 06:25:48 INFO:20.8%, step:99, perplexity: 25.941, speed: 34469 words
2018-11-27T10:25:50.191089303Z 06:25:50 INFO:41.7%, step:199, perplexity: 27.051, speed: 43544 words
2018-11-27T10:25:52.06291597Z 06:25:52 INFO:62.5%, step:299, perplexity: 27.151, speed: 46722 words
2018-11-27T10:25:53.799448645Z 06:25:53 INFO:83.3%, step:399, perplexity: 27.173, speed: 49281 words
2018-11-27T10:25:55.447932056Z 06:25:55 INFO:Perplexity: 27.004, speed: 49355 words per sec
2018-11-27T10:25:56.815541581Z SYSTEM: Finishing...
2018-11-27T10:25:57.032365081Z SYSTEM: Done!
job-gpu-5bfd17d51afd943016b451b9.log @549e5c8 — raw · history · blame
