too many open files from dataloader

[jpata]

I'm seeing some issues with tfds, it does not seem to close open files properly after we switched to the split datasets in #350 that perhaps made the problem somewhat more apparent.

The reason is that random access to the concatenated datasets does not allow files to be closed.
With shuffling disabled here: https://github.com/jpata/particleflow/blob/main/mlpf/model/PFDataset.py#L259, the usage seems to be somewhat lower.

[jpata]

I'm not getting issues on my systems right now, as I've raised ulimits, but one needs to be careful about too many parallel jobs.

For a CLIC training, the number of open files currently looks like this: