MappingAUVOps 2006-2009 missions are not loading into production database

[MBARIMike]

Here is a launch.json config that does load a 2006 Mission:

"args": ["-v", "2", "--bootstrap", "--skipuntil_regex", "--regex", "MappingAUVOps2006/20060828/ZTopo.grd$", "--limit", "1"],

The .fnv files are also loaded with:

"args": ["--fnv", "-v", "1", "--limit", "2", "--skipuntil", "MappingAUVOps2006/20060828"]

Look into why these aren't loading in production.

[MBARIMike]

Here are the ZTopo.grd files found in MappingAUVOps2006:

(base) ➜  MappingAUVOps2006 find . -name ZTopo.grd
./20060828/ZTopo.grd
./20060626_520m_RepeatMapping/ZTopo.grd
./20060904/ZTopo.grd
./20060420/ZTopo.grd
./20060902/ZTopo.grd
./20060322/ZTopo.grd
./20060607/ZTopo.grd
./BarkleyHydrates/swathdata/TN146/ZTopo.grd
./20060905/ZTopo.grd
./Davidson2008/Navadjust/compare/ZTopo.grd
./20060605/ZTopo.grd
./20060222/ZTopo.grd
./20060608/ZTopo.grd
./20060321/ZTopo.grd
./20060606/ZTopo.grd
./20061024/ZTopo.grd
./Axial/TN199/EM300/ZTopo.grd
./Axial/TN199/EM300/temp/ZTopo.grd
./20060609/ZTopo.grd
./20060221/ZTopo.grd
./20060628_to_20060629_100m_RepeatMapping/ZTopo.grd
./Old/20060904/ZTopo.grd
./Old/20060902/ZTopo.grd
./Old/20060905/ZTopo.grd
./Old/20060903/ZTopo.grd
./Old/20060901/ZTopo.grd
./20060816/ZTopo.grd
./20060222old/ZTopo.grd
./20060816_org/ZTopo.grd
./20060812_org/ZTopo.grd
./20060817/ZTopo.grd
./20061011/ZTopo.grd
./20060419/ZTopo.grd
./20060903/ZTopo.grd
./20060901/ZTopo.grd
./20061012/ZTopo.grd
./20060817_org/ZTopo.grd

[MBARIMike]

The reason these missions aren't being loaded is because the names don't match the default regular expression that's used:

"\/(?P<yr>\d\d\d\d)(?P<mo>\d\d)(?P<da>\d\d)(?P<miss_seq>\S\S)\/"

The directory containing the ZTopo.grd file must have the pattern yyyymmddXX, where "XX" is a two-character "mission sequence" designation like "m1", "d1", etc. @jbpaduan Should these directories be renamed to match the pattern?

[jbpaduan]

You're right, the XX-less naming scheme was when one mission a day seemed reasonable. But when missions failed or had to be modified, having the incremental names reduced confusion at sea and afterward. Now with more than one vehicle, it's necessary. If the directory names are changed, all our datalists calling them forever after will be broken. One option could be to rename them but establish soft-links to the old names. What do you think?

[MBARIMike]

It's easy to relax the pattern match requirement for these missions. Let me attempt that change.

[MBARIMike]

I'm testing the software change now on a development machine and plan to reload https://smdb.shore.mbari.org/ tonight if all looks good.

[jbpaduan]

A new "mission" appears to have been found with this fix: some data from 1985 in /Volumes/SeafloorMapping/mbsystem/Data/2012/20121101Mutter/RC2607
that forms a long track from the Pacific across to the Atlantic.
Data in the mbsystem directory can all be excluded. Those data sets would have been provided to Dave in his effort to support all the different sonar systems out there, or trouble-shoot code that maybe had previously worked for a file format. Don't know why this particular one was picked up by the change to your load script.

[jbpaduan]

And there's another new track showing in the Caribbean: coming in from swathdata/surveys/soi/fk007/processing/multibeam/em302/20130328.
While it would be cool to have a way to search on all the surveys in "swathdata/surveys" we decided we shouldn't go there yet, as they are ship-based surveys. But that suggests to me a new use-case for the SMDB, because once our new ship comes on line, we will be collecting multibeam data with it, too. We should talk about how we're going to manage that with Dave.

[jbpaduan]

It also appears that thumbnails to all of the surveys are broken now.

[MBARIMike]

I'll reload the database tonight with the changes from #103. I'll also check why so many thumbnails are missing.

[MBARIMike]

The changes in #103 were not sufficient. Working on a fix…

[MBARIMike]

It looks like that fix skipped over all the /mbsystem/ missions: https://smdb.shore.mbari.org/.

[MBARIMike]

If you see broken thumbnails you can now check the load log output for the reason why – search for the Mission name.

A link to this file is provided in a link on the upper right of the web page.

[MBARIMike]

https://smdb.shore.mbari.org/ has been reloaded with the updated exclude.list. The Caribbean swathdata missions are no longer being loaded.

[jbpaduan]

This issue can be closed.