[3pt] CatFIM duplicate records bug

[EmilyDeardorff]

In the most recent run of CatFIM, we have been finding duplicate lids in the HUC_messages.txt file which has translated to duplication of those LIDs in the CatFIM outputs.

Add a section of code to catch and mitigate duplicate LIDs in the HUC messages file if there are two and one of them starts with '---', because that is more of a warning than a message. This will help prevent LID duplication in the outputs.

Duplicate LIDs and HUCs that could be used for testing:
gaka2    19020101
gaka2    19020101
gcka2    19020101
gkna2    19020101
jcta2    19020101
mena2    19020101
mnta2    19020101
paxk    19020101
pxka2    19020101
scca2    19020101
slna2    19020101
ccga2    19020102
crta2    19020102
dada2    19020102
egha2    19020102
enka2    19020102
ghya2    19020102
glna2    19020102
gnna2    19020102
gpoa2    19020102
gpoa2    19020102
grsa2    19020102

Code snippet to adjust:

  # TODO: Aug 2024: This is now identical to the way flow handles messages            
  # but the system should probably be changed to somethign more elegant but good enough            
  # for now. At least is is MP safe.            
  # Write all_messages to huc-specific file.            
  # MP_LOG.lprint(f'Writing message file for {huc}')            
  huc_messages_txt_file = os.path.join(huc_messages_dir, str(huc) + '_messages.txt')            
with open(huc_messages_txt_file, 'w') as f:                
  for item in all_messages:                    
  item = item.strip()                    
  # f.write("%s\n" % item)```

[EmilyDeardorff]

This issue was fixed with the code added in Commit c589232 .

# Check for duplicate sites in the messages
lids  = [msg.split(':')[0] for msg in all_messages]
duplicate_lids = set([x for x in lids if lids.count(x) > 1])
# If there are duplicate sites and one of the lines has '---', drop that line
filtered_messages = [
     msg for msg in all_messages 
     if not (msg.split(':')[0] in duplicate_lids and ':---' in msg)
]