Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

학습 스텝 완료 후 에러(Could not synthesize characters to Hangul.) #78

Open
froglike6 opened this issue May 18, 2020 · 9 comments

Comments

@froglike6
Copy link

Exiting due to exception: Could not synthesize characters to Hangul.
Traceback (most recent call last):
File "train.py", line 268, in train
log_dir, step, loss, "train")
File "train.py", line 104, in save_and_plot
parallel_run(fn, items, parallel=False)
File "/root/tts/multi-speaker-tacotron-tensorflow/utils/init.py", line 151, in parallel_run
out = fn(item)
File "train.py", line 95, in save_and_plot_fn
skip_eos_and_pad=True, combine_jamo=True))
File "/root/tts/multi-speaker-tacotron-tensorflow/text/init.py", line 77, in sequence_to_text
return jamo_to_korean(result)
File "/root/tts/multi-speaker-tacotron-tensorflow/text/korean.py", line 64, in jamo_to_korean
new_text += _get_text_from_candidates(candidates)
File "/root/tts/multi-speaker-tacotron-tensorflow/text/korean.py", line 53, in _get_text_from_candidates
return j2h(**dict(zip(["lead", "vowel", "tail"], candidates)))
File "/usr/local/lib/python3.6/site-packages/jamo/jamo.py", line 290, in j2h
return jamo_to_hangul(lead, vowel, tail)
File "/usr/local/lib/python3.6/site-packages/jamo/jamo.py", line 277, in jamo_to_hangul
'\x00')
jamo.jamo.InvalidJamoError: Could not synthesize characters to Hangul.
2020-05-18 10:29:45.504211: W tensorflow/core/kernels/queue_base.cc:295] _0_datafeeder/input_queue_1: Skipping cancelled enqueue attempt with queue not closed
2020-05-18 10:29:45.504287: W tensorflow/core/kernels/queue_base.cc:295] _1_datafeeder/input_queue: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
status, run_metadata)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
status, run_metadata)
File "/usr/lib64/python3.6/contextlib.py", line 88, in exit
next(self.gen)
File "/usr/lib64/python3.6/contextlib.py", line 88, in exit
next(self.gen)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
[[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/loss_coeff_0_3, _arg_datafeeder/mel_targets_0_4, _arg_datafeeder/linear_targets_0_2)]]
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
[[Node: datafeeder/input_queue_1_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue_1, _arg_datafeeder/inputs_1_0_1, _arg_datafeeder/input_lengths_1_0_0, _arg_datafeeder/loss_coeff_1_0_3, _arg_datafeeder/mel_targets_1_0_4, _arg_datafeeder/linear_targets_1_0_2)]]

During handling of the above exception, another exception occurred:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
Traceback (most recent call last):
File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 204, in run
self._enqueue_next_group()
File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 204, in run
self._enqueue_next_group()
File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 242, in _enqueue_next_group
self._session.run(self._enqueue_op, feed_dict=feed_dict)
File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 242, in _enqueue_next_group
self._session.run(self._enqueue_op, feed_dict=feed_dict)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
[[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/loss_coeff_0_3, _arg_datafeeder/mel_targets_0_4, _arg_datafeeder/linear_targets_0_2)]]

Caused by op 'datafeeder/input_queue_enqueue', defined at:
File "train.py", line 322, in
main()
File "train.py", line 318, in main
train(config.model_dir, config)
File "train.py", line 136, in train
data_type='train', batch_size=hparams.batch_size)
File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 160, in init
self._enqueue_op = queue.enqueue(self._placeholders)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/data_flow_ops.py", line 321, in enqueue
self._queue_ref, vals, name=scope)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 1587, in _queue_enqueue_v2
name=name)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled
[[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/loss_coeff_0_3, _arg_datafeeder/mel_targets_0_4, _arg_datafeeder/linear_targets_0_2)]]

tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
[[Node: datafeeder/input_queue_1_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue_1, _arg_datafeeder/inputs_1_0_1, _arg_datafeeder/input_lengths_1_0_0, _arg_datafeeder/loss_coeff_1_0_3, _arg_datafeeder/mel_targets_1_0_4, _arg_datafeeder/linear_targets_1_0_2)]]

Caused by op 'datafeeder/input_queue_1_enqueue', defined at:
File "train.py", line 322, in
main()
File "train.py", line 318, in main
train(config.model_dir, config)
File "train.py", line 139, in train
data_type='test', batch_size=config.num_test)
File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 160, in init
self._enqueue_op = queue.enqueue(self._placeholders)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/data_flow_ops.py", line 321, in enqueue
self._queue_ref, vals, name=scope)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 1587, in _queue_enqueue_v2
name=name)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled
[[Node: datafeeder/input_queue_1_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue_1, _arg_datafeeder/inputs_1_0_1, _arg_datafeeder/input_lengths_1_0_0, _arg_datafeeder/loss_coeff_1_0_3, _arg_datafeeder/mel_targets_1_0_4, _arg_datafeeder/linear_targets_1_0_2)]]

위와 같이 뜹니다.. 어떻게 해야 하나요?

@solalala-12
Copy link

synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데

@froglike6
Copy link
Author

froglike6 commented May 19, 2020 via email

@solalala-12
Copy link

학습 후 파일 저장 시 이 에러가 뜹니다 2020년 5월 18일 (월) 오후 5:51, LEE SORA [email protected]님이 작성:

synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHOQVVNV7KHGXPUWGFLRSDZKFANCNFSM4NDU5AKA .

#10 참고해보세요~

@froglike6
Copy link
Author

froglike6 commented May 19, 2020

학습 후 파일 저장 시 이 에러가 뜹니다 2020년 5월 18일 (월) 오후 5:51, LEE SORA [email protected]님이 작성:

synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHOQVVNV7KHGXPUWGFLRSDZKFANCNFSM4NDU5AKA .

#10 참고해보세요~

이 문제 해결했더니 list index out of range 에러뜨네요..

@solalala-12
Copy link

학습 후 파일 저장 시 이 에러가 뜹니다 2020년 5월 18일 (월) 오후 5:51, LEE SORA [email protected]님이 작성:

synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHOQVVNV7KHGXPUWGFLRSDZKFANCNFSM4NDU5AKA .

#10 참고해보세요~

이 문제 해결했더니 list index out of range 에러뜨네요..

train.py 를 실행했더니 Could not synthesize characters to Hangul. 에러가 뜨신건가요 ?
list index out of range 에러도 상세히 올려주시면 답변 드리겠습니다.

@froglike6
Copy link
Author

froglike6 commented May 20, 2020

학습 후 파일 저장 시 이 에러가 뜹니다 2020년 5월 18일 (월) 오후 5:51, LEE SORA [email protected]님이 작성:

synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHOQVVNV7KHGXPUWGFLRSDZKFANCNFSM4NDU5AKA .

#10 참고해보세요~

이 문제 해결했더니 list index out of range 에러뜨네요..

train.py 를 실행했더니 Could not synthesize characters to Hangul. 에러가 뜨신건가요 ?
list index out of range 에러도 상세히 올려주시면 답변 드리겠습니다.

[root@dhmowiki tts]# python3 train.py --data_path=datasets/ksw
[] MODEL dir: logs/ksw_2020-05-20_13-36-08
[
] PARAM path: logs/ksw_2020-05-20_13-36-08/params.json

[!] Detect non-krbook dataset. Set sampling rate from 24000 to 20000

[*] git recv-parse HEAD:
da58cf1

==================================================
[*] dit diff:
diff --git a/datasets/generate_data.py b/datasets/generate_data.py
index b351eb2..7afbb1d 100644
--- a/datasets/generate_data.py
+++ b/datasets/generate_data.py
@@ -37,7 +37,7 @@ def build_from_path(config):

 loss_coeff = defaultdict(one)
 if config.metadata_path.endswith("json"):
  •    with open(config.metadata_path) as f:
    
  •    with open(config.metadata_path, encoding='utf-8') as f:
           content = f.read()
       info = json.loads(content)
    
    elif config.metadata_path.endswith("csv"):
    diff --git a/train.py b/train.py
    index 3316c3a..be7ea5a 100644
    --- a/train.py
    +++ b/train.py
    @@ -135,7 +135,7 @@ def train(log_dir, config):
    coord, data_dirs, hparams, config, 32,
    data_type='train', batch_size=hparams.batch_size)
    test_feeder = DataFeeder(
  •            coord, data_dirs, hparams, config, 8,
    
  •            coord, data_dirs, hparams, config, 2,
               data_type='test', batch_size=config.num_test)
    

    Set up model:

==================================================
[] Checkpoint path: logs/ksw_2020-05-20_13-36-08/model.ckpt
[
] Loading training data from: ['datasets/ksw/data']
[*] Using model: logs/ksw_2020-05-20_13-36-08
Hyperparameters:
adam_beta1: 0.9
adam_beta2: 0.999
attention_size: 256
attention_state_size: 256
attention_type: bah_mon
batch_size: 16
cleaners: korean_cleaners
dec_layer_num: 2
dec_prenet_sizes: [256, 128]
dec_rnn_size: 256
decay_learning_rate_mode: 0
dropout_prob: 0.8
embedding_size: 256
enc_bank_channel_size: 128
enc_bank_size: 16
enc_highway_depth: 4
enc_maxpool_width: 2
enc_prenet_sizes: [256, 128]
enc_proj_sizes: [128, 128]
enc_proj_width: 3
enc_rnn_size: 128
frame_length_ms: 50
frame_shift_ms: 12.5
griffin_lim_iters: 60
ignore_recognition_level: 1
initial_data_greedy: True
initial_learning_rate: 0.002
initial_phase_step: 8000
main_data: ['']
main_data_greedy_factor: 0
max_iters: 200
min_iters: 30
min_level_db: -100
min_tokens: 50
model_type: single
num_freq: 1025
num_mels: 80
post_bank_channel_size: 256
post_bank_size: 8
post_highway_depth: 4
post_maxpool_width: 2
post_proj_sizes: [256, 80]
post_proj_width: 3
post_rnn_size: 256
power: 1.5
preemphasis: 0.97
prioritize_loss: False
recognition_loss_coeff: 0.2
reduction_factor: 4
ref_level_db: 20
sample_rate: 24000
skip_inadequate: False
speaker_embedding_size: 16
use_fixed_test_inputs: False
filter_by_min_max_frame_batch: 100%|███████████| 41/41 [00:00<00:00, 924.70it/s]
[datasets/ksw/data] Loaded metadata for 11 examples (0.01 hours)
[datasets/ksw/data] Max length: 737
[datasets/ksw/data] Min length: 229

{'datasets/ksw/data': 1.0}

filter_by_min_max_frame_batch: 100%|███████████| 41/41 [00:00<00:00, 800.97it/s]
[datasets/ksw/data] Loaded metadata for 11 examples (0.01 hours)
[datasets/ksw/data] Max length: 737
[datasets/ksw/data] Min length: 229

{'datasets/ksw/data': 1.0}

========================================
model_type: single

Initialized Tacotron model. Dimensions:
embedding: 256
speaker embedding: None
prenet out: 128
encoder out: 256
attention out: 256
concat attn & out: 512
decoder cell out: 256
decoder out (4 frames): 320
decoder out (1 frame): 80
postnet out: 512
linear out: 1025

model_type: single

Initialized Tacotron model. Dimensions:
embedding: 256
speaker embedding: None
prenet out: 128
encoder out: 256
attention out: 256
concat attn & out: 512
decoder cell out: 256
decoder out (4 frames): 320
decoder out (1 frame): 80
postnet out: 512
linear out: 1025
2020-05-20 13:36:15.699815: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2020-05-20 13:36:15.699851: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2020-05-20 13:36:15.699860: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2020-05-20 13:36:15.699867: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2020-05-20 13:36:15.699875: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Starting new training run at commit: None
Generated 2 batches of size 2 in 0.000 sec
Traceback (most recent call last):
File "/root/tts/tts/datasets/datafeeder.py", line 204, in run
self._enqueue_next_group()
File "/root/tts/tts/datasets/datafeeder.py", line 229, in _enqueue_next_group
for _ in range(int(n * self._batches_per_group // len(self.data_dirs)))]
File "/root/tts/tts/datasets/datafeeder.py", line 229, in
for _ in range(int(n * self._batches_per_group // len(self.data_dirs)))]
File "/root/tts/tts/datasets/datafeeder.py", line 257, in _get_next_example
data_path = data_paths[self._offset[data_dir]]
IndexError: list index out of range
원래 트레인 실행하면 hangul 에러 떴는데 hangul 에러 해결하고 다시 실행하니 out of range 에러가 뜹니다.

@solalala-12
Copy link

Loaded metadata for 11 examples을 보니 사용할 데이터는 11개 밖에 없는 것 같네요.
다음 batch를 가져오지 못해서 나오는 에러인 것 같습니다.
경로에 데이터가 잘 있는지 확인해보시거나 batch_size를 조절하거나 batches_per_group을 조절해보세요.

@froglike6
Copy link
Author

froglike6 commented May 20, 2020 via email

@solalala-12
Copy link

batch_size는 hparams.py에서 수정하시면 됩니다.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants