2018-11-30T04:39:18.771169882Z SYSTEM: Preparing env...
2018-11-30T04:39:19.230566549Z SYSTEM: Running...
2018-11-30T04:39:24.792998787Z Writing to /home/jovyan/work/results/tb_results/tensorboard_log/1543552764
2018-11-30T04:39:24.793042757Z 
2018-11-30T04:39:24.793049271Z ============================================================
2018-11-30T04:39:24.793054456Z All final and intermediate outputs will be stored in ./results/output_poem/
2018-11-30T04:39:24.793059361Z ============================================================
2018-11-30T04:39:24.793064116Z 
2018-11-30T04:39:24.793068359Z 12:39:24 INFO:args are:
2018-11-30T04:39:24.793078562Z Namespace(batch_size=16, best_model='', best_valid_ppl=inf, cell_type='lstm', data_path='./datasets/yangsaisai-poetrydatasets-0_0_1/', debug=False, dropout=0.0, embedding_size=128, encoding='utf-8', hidden_size=128, init_dir='', init_model='', input_dropout=0.0, learning_rate=0.005, max_grad_norm=5.0, num_epochs=8, num_layers=2, num_unrollings=64, output_dir='./results/output_poem', progress_freq=100, save_best_model='./results/output_poem/best_model/model', save_model='./results/output_poem/save_model/model', tb_log_dir='/home/jovyan/work/results/tb_results/tensorboard_log/1543552764', test=False, train_frac=0.9, valid_frac=0.05, verbose=0)
2018-11-30T04:39:24.793087349Z 12:39:24 INFO:Parameters are:
2018-11-30T04:39:24.793092061Z {
2018-11-30T04:39:24.793095972Z     "batch_size": 16,
2018-11-30T04:39:24.793100611Z     "cell_type": "lstm",
2018-11-30T04:39:24.793105184Z     "dropout": 0.0,
2018-11-30T04:39:24.793109809Z     "embedding_size": 128,
2018-11-30T04:39:24.793114104Z     "hidden_size": 128,
2018-11-30T04:39:24.79312085Z     "input_dropout": 0.0,
2018-11-30T04:39:24.793125397Z     "learning_rate": 0.005,
2018-11-30T04:39:24.793129944Z     "max_grad_norm": 5.0,
2018-11-30T04:39:24.79313419Z     "num_layers": 2,
2018-11-30T04:39:24.793138499Z     "num_unrollings": 64
2018-11-30T04:39:24.793143017Z }
2018-11-30T04:39:24.793147129Z 
2018-11-30T04:39:25.009827544Z tensor_file:./datasets/yangsaisai-poetrydatasets-0_0_1/poem_ids.txt
2018-11-30T04:39:25.010589006Z Loading dataset from ./datasets/yangsaisai-poetrydatasets-0_0_1/poem_ids.txt
2018-11-30T04:39:25.359910285Z file maxSeqLen = 64
2018-11-30T04:39:25.359961327Z Loaded ./datasets/yangsaisai-poetrydatasets-0_0_1/:  training  samples:65235 ,validationSamples:3837,testingSamples:7676
2018-11-30T04:39:26.276017793Z tensor_file:./datasets/yangsaisai-poetrydatasets-0_0_1/poem_ids.txt
2018-11-30T04:39:26.278968445Z Loading dataset from ./datasets/yangsaisai-poetrydatasets-0_0_1/poem_ids.txt
2018-11-30T04:39:26.821024692Z file maxSeqLen = 64
2018-11-30T04:39:26.821074339Z Loaded ./datasets/yangsaisai-poetrydatasets-0_0_1/:  training  samples:65235 ,validationSamples:3837,testingSamples:7676
2018-11-30T04:39:26.883903582Z tensor_file:./datasets/yangsaisai-poetrydatasets-0_0_1/poem_ids.txt
2018-11-30T04:39:26.883949908Z Loading dataset from ./datasets/yangsaisai-poetrydatasets-0_0_1/poem_ids.txt
2018-11-30T04:39:27.70684711Z file maxSeqLen = 64
2018-11-30T04:39:27.706886182Z Loaded ./datasets/yangsaisai-poetrydatasets-0_0_1/:  training  samples:65235 ,validationSamples:3837,testingSamples:7676
2018-11-30T04:39:27.732969217Z 12:39:27 INFO:Creating graph
2018-11-30T04:39:44.21809571Z 12:39:44 INFO:Start training
2018-11-30T04:39:44.218139835Z 
2018-11-30T04:39:44.222225853Z 2018-11-30 12:39:44.221828: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-11-30T04:39:44.814174911Z 2018-11-30 12:39:44.813206: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-11-30T04:39:44.817549103Z 2018-11-30 12:39:44.816320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
2018-11-30T04:39:44.81756827Z name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
2018-11-30T04:39:44.81757418Z pciBusID: 0000:00:07.0
2018-11-30T04:39:44.817578953Z totalMemory: 15.90GiB freeMemory: 15.61GiB
2018-11-30T04:39:44.817583447Z 2018-11-30 12:39:44.816366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2018-11-30T04:39:46.742354307Z 2018-11-30 12:39:46.736966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-11-30T04:39:46.742386365Z 2018-11-30 12:39:46.737031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2018-11-30T04:39:46.742393292Z 2018-11-30 12:39:46.737042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2018-11-30T04:39:46.742400066Z 2018-11-30 12:39:46.737548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15129 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:07.0, compute capability: 6.0)
2018-11-30T04:39:56.505560041Z 12:39:56 INFO:=================== Epoch 0 ===================
2018-11-30T04:39:56.505607493Z 
2018-11-30T04:39:56.50561525Z 12:39:56 INFO:Training on training set
2018-11-30T04:40:13.433002658Z 12:40:13 INFO:2.5%, step:99, perplexity: 603.783, speed: 6159 words
2018-11-30T04:40:24.835022537Z 12:40:24 INFO:4.9%, step:199, perplexity: 382.204, speed: 7306 words
2018-11-30T04:40:37.11490721Z 12:40:37 INFO:7.4%, step:299, perplexity: 294.070, speed: 7621 words
2018-11-30T04:40:46.951976568Z 12:40:46 INFO:9.8%, step:399, perplexity: 246.033, speed: 8168 words
2018-11-30T04:40:54.273857463Z 12:40:54 INFO:12.3%, step:499, perplexity: 215.291, speed: 8910 words
2018-11-30T04:41:01.898883206Z 12:41:01 INFO:14.7%, step:599, perplexity: 193.340, speed: 9439 words
2018-11-30T04:41:09.470843498Z 12:41:09 INFO:17.2%, step:699, perplexity: 176.586, speed: 9864 words
2018-11-30T04:41:16.48887215Z 12:41:16 INFO:19.6%, step:799, perplexity: 163.220, speed: 10281 words
2018-11-30T04:41:23.899031045Z 12:41:23 INFO:22.1%, step:899, perplexity: 152.391, speed: 10583 words
2018-11-30T04:41:30.936227426Z 12:41:30 INFO:24.5%, step:999, perplexity: 143.515, speed: 10878 words
2018-11-30T04:41:38.123902254Z 12:41:38 INFO:27.0%, step:1099, perplexity: 136.097, speed: 11117 words
2018-11-30T04:41:45.061928369Z 12:41:45 INFO:29.4%, step:1199, perplexity: 129.725, speed: 11351 words
2018-11-30T04:41:52.005683983Z 12:41:52 INFO:31.9%, step:1299, perplexity: 124.212, speed: 11556 words
2018-11-30T04:42:02.165205709Z 12:42:02 INFO:34.3%, step:1399, perplexity: 119.386, speed: 11439 words
2018-11-30T04:42:14.464941867Z 12:42:14 INFO:36.8%, step:1499, perplexity: 115.124, speed: 11158 words
2018-11-30T04:42:26.401843448Z 12:42:26 INFO:39.2%, step:1599, perplexity: 111.343, speed: 10952 words
2018-11-30T04:42:35.551908158Z 12:42:35 INFO:41.7%, step:1699, perplexity: 107.962, speed: 10966 words
2018-11-30T04:42:43.651462585Z 12:42:43 INFO:44.1%, step:1799, perplexity: 104.918, speed: 11047 words
2018-11-30T04:42:50.724941217Z 12:42:50 INFO:46.6%, step:1899, perplexity: 102.144, speed: 11187 words
2018-11-30T04:42:59.457983904Z 12:42:59 INFO:49.0%, step:1999, perplexity: 99.596, speed: 11212 words
2018-11-30T04:43:07.632098487Z 12:43:07 INFO:51.5%, step:2099, perplexity: 97.275, speed: 11269 words
2018-11-30T04:43:15.221914296Z 12:43:15 INFO:53.9%, step:2199, perplexity: 95.154, speed: 11354 words
2018-11-30T04:43:22.536957817Z 12:43:22 INFO:56.4%, step:2299, perplexity: 93.198, speed: 11448 words
2018-11-30T04:43:29.994808334Z 12:43:29 INFO:58.9%, step:2399, perplexity: 91.377, speed: 11528 words
2018-11-30T04:43:36.589198501Z 12:43:36 INFO:61.3%, step:2499, perplexity: 89.682, speed: 11648 words