You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Did you generate one giant TFRecord for the Lakh MIDI dataset? Or did you process the data in shards? If the latter, how exactly does one go about sharding the data with the pipelines that you have in place? I'm finding that generating one giant TFRecord using convert_dir_to_note_sequences is too large to load into memory when doing scripts/generate_song_data_beam.py.
The text was updated successfully, but these errors were encountered:
I had a similar issue and I've managed to solve it by removing ReShuffle from the pipeline to avoid parallelism at the cost of reducing the speed of the process but at least it doesn't try to load entire data to the memory and runs smoothly (but very slowly)
Did you generate one giant TFRecord for the Lakh MIDI dataset? Or did you process the data in shards? If the latter, how exactly does one go about sharding the data with the pipelines that you have in place? I'm finding that generating one giant TFRecord using
convert_dir_to_note_sequences
is too large to load into memory when doingscripts/generate_song_data_beam.py
.The text was updated successfully, but these errors were encountered: