-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathPEP_DataMgmtPlan.html
1230 lines (1204 loc) · 71.3 KB
/
PEP_DataMgmtPlan.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>
<meta charset="utf-8">
<meta name="generator" content="quarto-1.3.450">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
<meta name="author" content="Stacie Koslovsky">
<title>PEP Data Management Plan</title>
<style>
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list{list-style: none;}
ul.task-list li input[type="checkbox"] {
width: 0.8em;
margin: 0 0.8em 0.2em -1em; /* quarto-specific, see https://github.com/quarto-dev/quarto-cli/issues/4556 */
vertical-align: middle;
}
/* CSS for syntax highlighting */
pre > code.sourceCode { white-space: pre; position: relative; }
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
pre > code.sourceCode > span:empty { height: 1.2em; }
.sourceCode { overflow: visible; }
code.sourceCode > span { color: inherit; text-decoration: inherit; }
div.sourceCode { margin: 1em 0; }
pre.sourceCode { margin: 0; }
@media screen {
div.sourceCode { overflow: auto; }
}
@media print {
pre > code.sourceCode { white-space: pre-wrap; }
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
}
pre.numberSource code
{ counter-reset: source-line 0; }
pre.numberSource code > span
{ position: relative; left: -4em; counter-increment: source-line; }
pre.numberSource code > span > a:first-child::before
{ content: counter(source-line);
position: relative; left: -1em; text-align: right; vertical-align: baseline;
border: none; display: inline-block;
-webkit-touch-callout: none; -webkit-user-select: none;
-khtml-user-select: none; -moz-user-select: none;
-ms-user-select: none; user-select: none;
padding: 0 4px; width: 4em;
}
pre.numberSource { margin-left: 3em; padding-left: 4px; }
div.sourceCode
{ }
@media screen {
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
}
</style>
<script src="PEP_DataMgmtPlan_files/libs/clipboard/clipboard.min.js"></script>
<script src="PEP_DataMgmtPlan_files/libs/quarto-html/quarto.js"></script>
<script src="PEP_DataMgmtPlan_files/libs/quarto-html/popper.min.js"></script>
<script src="PEP_DataMgmtPlan_files/libs/quarto-html/tippy.umd.min.js"></script>
<script src="PEP_DataMgmtPlan_files/libs/quarto-html/anchor.min.js"></script>
<link href="PEP_DataMgmtPlan_files/libs/quarto-html/tippy.css" rel="stylesheet">
<link href="PEP_DataMgmtPlan_files/libs/quarto-html/quarto-syntax-highlighting-dark.css" rel="stylesheet" id="quarto-text-highlighting-styles">
<script src="PEP_DataMgmtPlan_files/libs/bootstrap/bootstrap.min.js"></script>
<link href="PEP_DataMgmtPlan_files/libs/bootstrap/bootstrap-icons.css" rel="stylesheet">
<link href="PEP_DataMgmtPlan_files/libs/bootstrap/bootstrap.min.css" rel="stylesheet" id="quarto-bootstrap" data-mode="dark">
</head>
<body>
<div id="quarto-content" class="page-columns page-rows-contents page-layout-article">
<div id="quarto-margin-sidebar" class="sidebar margin-sidebar">
<nav id="TOC" role="doc-toc" class="toc-active">
<h2 id="toc-title">Table of contents</h2>
<ul>
<li><a href="#objective" id="toc-objective" class="nav-link active" data-scroll-target="#objective">Objective</a></li>
<li><a href="#overview" id="toc-overview" class="nav-link" data-scroll-target="#overview">Overview</a>
<ul class="collapse">
<li><a href="#foundations-to-data-management" id="toc-foundations-to-data-management" class="nav-link" data-scroll-target="#foundations-to-data-management">Foundations to Data Management</a></li>
<li><a href="#general-guidelines" id="toc-general-guidelines" class="nav-link" data-scroll-target="#general-guidelines">General Guidelines</a></li>
<li><a href="#file-naming-conventions" id="toc-file-naming-conventions" class="nav-link" data-scroll-target="#file-naming-conventions">File Naming Conventions</a>
<ul class="collapse">
<li><a href="#three-principles-for-file-naming" id="toc-three-principles-for-file-naming" class="nav-link" data-scroll-target="#three-principles-for-file-naming">Three Principles for File Naming</a></li>
<li><a href="#file-naming-examples" id="toc-file-naming-examples" class="nav-link" data-scroll-target="#file-naming-examples">File Naming Examples</a></li>
</ul></li>
</ul></li>
<li><a href="#deiab-in-pep-data-management" id="toc-deiab-in-pep-data-management" class="nav-link" data-scroll-target="#deiab-in-pep-data-management">DEIAB in PEP Data Management</a></li>
<li><a href="#data-management-workflow" id="toc-data-management-workflow" class="nav-link" data-scroll-target="#data-management-workflow">Data Management Workflow</a>
<ul class="collapse">
<li><a href="#plan" id="toc-plan" class="nav-link" data-scroll-target="#plan">Plan</a></li>
<li><a href="#integrate" id="toc-integrate" class="nav-link" data-scroll-target="#integrate">Integrate</a></li>
<li><a href="#collect" id="toc-collect" class="nav-link" data-scroll-target="#collect">Collect</a>
<ul class="collapse">
<li><a href="#field-data-collection" id="toc-field-data-collection" class="nav-link" data-scroll-target="#field-data-collection">Field Data Collection</a></li>
</ul></li>
<li><a href="#process" id="toc-process" class="nav-link" data-scroll-target="#process">Process</a>
<ul class="collapse">
<li><a href="#raw-and-original-data" id="toc-raw-and-original-data" class="nav-link" data-scroll-target="#raw-and-original-data">Raw and Original Data</a></li>
<li><a href="#processed-and-final-data" id="toc-processed-and-final-data" class="nav-link" data-scroll-target="#processed-and-final-data">Processed and Final Data</a></li>
<li><a href="#backup-procedures" id="toc-backup-procedures" class="nav-link" data-scroll-target="#backup-procedures">Backup Procedures</a></li>
</ul></li>
<li><a href="#debrief" id="toc-debrief" class="nav-link" data-scroll-target="#debrief">Debrief</a></li>
<li><a href="#analyze" id="toc-analyze" class="nav-link" data-scroll-target="#analyze">Analyze</a></li>
<li><a href="#evaluate" id="toc-evaluate" class="nav-link" data-scroll-target="#evaluate">Evaluate</a></li>
<li><a href="#share" id="toc-share" class="nav-link" data-scroll-target="#share">Share</a>
<ul class="collapse">
<li><a href="#overall-workflow" id="toc-overall-workflow" class="nav-link" data-scroll-target="#overall-workflow">Overall Workflow</a></li>
<li><a href="#metadata" id="toc-metadata" class="nav-link" data-scroll-target="#metadata">Metadata</a></li>
<li><a href="#online-repositories" id="toc-online-repositories" class="nav-link" data-scroll-target="#online-repositories">Online Repositories</a></li>
</ul></li>
<li><a href="#deiab-in-data-management-workflow" id="toc-deiab-in-data-management-workflow" class="nav-link" data-scroll-target="#deiab-in-data-management-workflow">DEIAB in Data Management Workflow</a></li>
</ul></li>
<li><a href="#on-boarding" id="toc-on-boarding" class="nav-link" data-scroll-target="#on-boarding">On-boarding</a>
<ul class="collapse">
<li><a href="#software-requirements-and-recommendations" id="toc-software-requirements-and-recommendations" class="nav-link" data-scroll-target="#software-requirements-and-recommendations">Software Requirements and Recommendations</a>
<ul class="collapse">
<li><a href="#details-for-odbc-driver-installation" id="toc-details-for-odbc-driver-installation" class="nav-link" data-scroll-target="#details-for-odbc-driver-installation">Details for ODBC Driver Installation</a></li>
</ul></li>
<li><a href="#data-storage" id="toc-data-storage" class="nav-link" data-scroll-target="#data-storage">Data Storage</a>
<ul class="collapse">
<li><a href="#individual-storage-resources" id="toc-individual-storage-resources" class="nav-link" data-scroll-target="#individual-storage-resources">Individual Storage Resources</a></li>
<li><a href="#shared-storage-resources" id="toc-shared-storage-resources" class="nav-link" data-scroll-target="#shared-storage-resources">Shared Storage Resources</a></li>
<li><a href="#pep-postgresql-database" id="toc-pep-postgresql-database" class="nav-link" data-scroll-target="#pep-postgresql-database">PEP PostgreSQL Database</a></li>
</ul></li>
<li><a href="#frequently-used-program-wide-tools" id="toc-frequently-used-program-wide-tools" class="nav-link" data-scroll-target="#frequently-used-program-wide-tools">Frequently Used Program-wide Tools</a>
<ul class="collapse">
<li><a href="#pep-dashboard" id="toc-pep-dashboard" class="nav-link" data-scroll-target="#pep-dashboard">PEP Dashboard</a></li>
<li><a href="#inventory-database" id="toc-inventory-database" class="nav-link" data-scroll-target="#inventory-database">Inventory Database</a></li>
<li><a href="#extensis-portfolio" id="toc-extensis-portfolio" class="nav-link" data-scroll-target="#extensis-portfolio">Extensis Portfolio</a></li>
<li><a href="#zotero" id="toc-zotero" class="nav-link" data-scroll-target="#zotero">Zotero</a></li>
</ul></li>
</ul></li>
<li><a href="#off-boarding" id="toc-off-boarding" class="nav-link" data-scroll-target="#off-boarding">Off-boarding</a></li>
</ul>
</nav>
</div>
<main class="content" id="quarto-document-content">
<header id="title-block-header" class="quarto-title-block default">
<div class="quarto-title">
<h1 class="title">PEP Data Management Plan</h1>
</div>
<div class="quarto-title-meta">
<div>
<div class="quarto-title-meta-heading">Author</div>
<div class="quarto-title-meta-contents">
<p>Stacie Koslovsky </p>
</div>
</div>
</div>
</header>
<p>This document was last updated on 2025-01-08.</p>
<section id="objective" class="level1">
<h1>Objective</h1>
<p>Welcome to the Polar Ecosystems Program (PEP; pronounced both as P-E-P and pep) Data Management Plan. The objective of this document is to outline and implement consistent information management procedures for PEP data.</p>
</section>
<section id="overview" class="level1">
<h1>Overview</h1>
<p>Data are our legacy. The data we collect, manage and analyze today will continue to be available and used by staff (and others) in the future. It is, therefore, important to maintain, organize and share our data in a way that will be useful and meaningful to current and future users. This includes:</p>
<ul>
<li><p>Keeping files organized and up-to-date;</p></li>
<li><p>Storing data in machine-readable, human-readable, and tidy formats;</p></li>
<li><p>Documenting information about the files and the process(es) for creating the files;</p></li>
<li><p>Cleaning out files (and paperwork) that are not important for the future legacy of the data (e.g. intermediate files, outdated information); and</p></li>
<li><p>Sharing data with the public and interested partners.</p></li>
</ul>
<section id="foundations-to-data-management" class="level2">
<h2 class="anchored" data-anchor-id="foundations-to-data-management">Foundations to Data Management</h2>
<p>A recently-ish published paper in PLOS ONE, <a href="https://arxiv.org/pdf/1609.00037.pdf">Good Enough Practices in Scientific Computing</a>, provides a thorough overview of best practices and workflows for managing scientific data. While some of the focus is out of our scope, a few key principles in the data management section are worth focusing on:</p>
<ol type="1">
<li><p><strong>Create the data you wish to see in the world</strong>. The original, raw data collected in the field or offloaded from sensors is rarely the data type and quality of data we would like to share with the world and have our names associated with. Data formats, data organization, column names, value data types and formats (e.g. date-time) can all be transformed and improved into higher quality forms. The better formed the data are, the easier subsequent analyses will be, the more reproducible our science will be, and the more likely it is that others will find use in our data.</p></li>
<li><p><strong>Create analysis-friendly (e.g. tidy) data</strong>. In many cases, analysis-friendly data will be equivalent to the data you wish to see in the world. The key principle at play here is that of “tidy” data. Hadley Wickham’s <a href="https://www.jstatsoft.org/article/view/v059i10/v59i10.pdf">2014 manuscript</a> does an excellent job outlining the ideal structure of tidy data.</p>
<ol type="1">
<li><p>The key components of tidy data are:</p>
<ol type="a">
<li><p>Each variable forms a column.</p></li>
<li><p>Each observation forms a row.</p></li>
<li><p>Each type of observational unit forms a table.</p></li>
</ol></li>
<li><p>Not only are these principles important for analyzing data in programming languages (such as R or Python), they are also key components to a well-organized database. Organizing data in this way will allow easier ingestion into a database; once your data are in a central database, the more tools for exploring and analyzing your data will be available.</p></li>
</ol></li>
<li><p><strong>Record all the steps used to process data</strong>. Embracing scripts developed in a programming language (such as R or Python) is essential to providing a robust, reproducible workflow for your data. With large, complex datasets, manually processing data via mouse clicks and spreadsheet-centric workflows are often time-consuming and difficult to reproduce. As part of the planning process for the PEP data management workflow, we will evaluate existing data management processes and identify ways to simplify and automate existing or new steps.</p></li>
</ol>
</section>
<section id="general-guidelines" class="level2">
<h2 class="anchored" data-anchor-id="general-guidelines">General Guidelines</h2>
<p><strong>Create folders with a top-down hierarchy</strong>, i.e. folders should get more specific as you get deeper into folder organization. This will make long-term organization more efficient, will consolidate files to a layout that will be easy for others to interpret, and will improve information transfer among staff.</p>
<p><strong>Create and maintain metadata documentation for your files</strong>. These can be shorter documents describing the files within a specific folder or longer documents describing the work that went into creating each of the files. The complexity of each file/folder dictates the level of detail that should be documented in the metadata. This level of metadata can be most simply managed using a word document or a text file.</p>
<p><strong>Copy files…do not drag-and-drop</strong>. When you need to move files on the network or your computer, move them by either copying the files to the new location and deleting the files from the old location or cutting and pasting from the old location to the new location (either option instead of drag-and-drop). Use batch processing whenever dealing with large volumes of data (example link to SMK code for this in R).</p>
</section>
<section id="file-naming-conventions" class="level2">
<h2 class="anchored" data-anchor-id="file-naming-conventions">File Naming Conventions</h2>
<p>Name and organize your files in such a way that you know what they mean at 3 am (or in a crunch)!</p>
<section id="three-principles-for-file-naming" class="level3">
<h3 class="anchored" data-anchor-id="three-principles-for-file-naming">Three Principles for File Naming</h3>
<ul>
<li><p><strong>Human readable</strong></p>
<ul>
<li><p>File names are easy to understand by anyone, not just you. ☺</p></li>
<li><p>Using versioning (when appropriate) to track different versions of your files (dates are a great way to do this).</p></li>
<li><p>Keep file names as short as possible.</p></li>
</ul></li>
<li><p><strong>Machine readable</strong></p>
<ul>
<li><p>Avoid spaces, punctuation, accented characters.</p></li>
<li><p>Case-sensitivity!</p></li>
<li><p>Deliberately use of delimiters (e.g._ - .) instead of spaces.</p></li>
</ul></li>
<li><p><strong>Orders logically</strong></p>
<ul>
<li><p>Name files from most general to most specific, so similar items cluster together when sorted by name and are easy to read (e.g. HarborSeal_Locations and HarborSeal_Polygons).</p></li>
<li><p>YYYY-MM-DD sorts better than DD-MMM-YYYY.</p></li>
<li><p>Use leading zeroes! (e.g. 01, 02, 10 – instead of 1, 2, 10).</p></li>
</ul></li>
</ul>
</section>
<section id="file-naming-examples" class="level3">
<h3 class="anchored" data-anchor-id="file-naming-examples">File Naming Examples</h3>
<table class="table">
<thead>
<tr class="header">
<th>Not-so-good</th>
<th>Improved File Name</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Plan 1.docx</td>
<td>DataMgmtPlan_20170825.docx</td>
</tr>
<tr class="even">
<td>Why won’t this work.R</td>
<td>CHESS_DataImport.R</td>
</tr>
<tr class="odd">
<td>Asdfasdf.jpg</td>
<td>Figure01.jpg</td>
</tr>
<tr class="even">
<td>New.pdf</td>
<td>ActivityPlan_20170815.pdf</td>
</tr>
<tr class="odd">
<td>NewFinal.pdf</td>
<td>ActivityPlan_20170818.pdf</td>
</tr>
<tr class="even">
<td>NewFinalFinal.pdf</td>
<td>ActivityPlan_20170822.pdf</td>
</tr>
</tbody>
</table>
</section>
</section>
</section>
<section id="deiab-in-pep-data-management" class="level1">
<h1>DEIAB in PEP Data Management</h1>
<p>Data management <em>might seem</em> beyond the scope of DEIA, but there are important considerations that can be applied to data-related work. DEIAB stands for Diversity, Equity, Inclusivity, Accessibility, and Belonging.</p>
<ul>
<li><p><strong>Diversity</strong> recognizes and appreciates our different skills, technical abilities, ideas, backgrounds, and ways of learning (among many other things). It is my intent to approach data management application and training in our program with this in mind. Our jobs and how we interact with data are also diverse; different people will have different needs and different applications for the data we collect and maintain.</p></li>
<li><p><strong>Equity</strong>, <a href="https://www3.weforum.org/docs/WEF_Advancing_Data_Equity_2024.pdf">from the World Economic Forum</a>, “is a shared responsibility that requires collective action to create data practices and systems that promote fair and just outcomes for all. … By considering data equity throughout the data life cycle, data practices can be improved to promote fair, just and beneficial outcomes for all individuals” and projects.</p></li>
<li><p><strong>Inclusivity</strong> is “the practice of ensuring that data collection, storage, and analysis processes actively consider and represent diverse perspectives, experiences, and identities, preventing the exclusion of marginalized groups and promoting equitable representation across all datasets, ultimately leading to more accurate and comprehensive insights.” While we don’t have to consider inclusivity when it comes to seals, we do have to consider inclusivity with respect to the people who are collecting, managing and using the data. Some considerations include:</p>
<ul>
<li><p>Developing inclusive data collection protocols: design surveys and data gathering methods that are accessible.</p></li>
<li><p>Implementing data governance practices: establish clear guidelines for collecting, storing, and using data. While this document is a blueprint for program projects, it does not replace the need for this to be considered and applied on a project-by-project basis.</p></li>
</ul></li>
<li><p><strong>Accessibility</strong> - the data and the information about the data need to be accessible. This document (plus other efforts, like PEP Data Days) are intended to make information as accessible as possible for the diversity of needs and users within our program.</p></li>
<li><p><strong>Belonging</strong> is the feeling of being accepted and supported in each project and within our program. Ultimately, it is my goal that everyone feels like they have access to the information they need and are getting the support that is required. This is a two-way street between project and data management.</p></li>
</ul>
</section>
<section id="data-management-workflow" class="level1">
<h1>Data Management Workflow</h1>
<p>The figure below describes the general process that PEP uses for managing data, start to finish and cyclically for on-going projects. This workflow consists of eight steps, some of which occur concurrently or similarly timed yet independent of one another.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="docs/PEP_DataMgmtPlan_files/images/DataManagementWorkflow.png" class="img-fluid figure-img"></p>
</figure>
</div>
<section id="plan" class="level2">
<h2 class="anchored" data-anchor-id="plan">Plan</h2>
<details>
<summary>
Click to Expand
</summary>
<p>Planning is crucial for the success of any project for all components of the work, including future data management. Early data management efforts can pay in dividends later by establishing clear roles, responsibilities, expectations and timelines. Prior to any new data collection, work with S. Koslovsky to:</p>
<ul>
<li><p>Identify where data will be stored (both short- and long-term; and set up the workspace), what data products will be needed, and what data processing will be required.</p></li>
<li><p>Create storage locations on network for final data and on Google Drive/elsewhere for intermediate processing; ensure necessary staff have access to these locations.</p></li>
<li><p>Develop/evaluate/implement data collection strategies that facilitate and simplify data management steps that follow (e.g. datasheets, in-field data entry, in-field archive efforts).</p></li>
<li><p>Coordinate (or at least communicate) with the AFSC OFIS team regarding network storage space needs and any other concerns.</p></li>
<li><p>Decide how best to organize the metadata for the project. All metadata entries within InPort will be published publicly and, eventually, used to create standards compliant alongside any open data products. A one-to-one relationship between metadata records and eventual open data products often makes this process easier.</p></li>
</ul>
<p>The <a href="https://nosc.noaa.gov/EDMC/PD.DMP.php">NOAA Data Management Planning Procedural Directive</a> (PD) (<a href="https://nosc.noaa.gov/EDMC/documents/EDMC-PD-DMP-2.0.1.pdf">direct link</a> to PDF of current version 2.0.1) requires a data management plan to be developed for all environmental data collected by NOAA programs or systems. The PD provides a generic template and guideline for developing a data management plan and a <a href="https://drive.google.com/drive/u/0/folders/0Bwl6f-PNVtnndG0yOUJ5cGU0RGs">data management plan repository</a> has been established. A common tool for developing data management plans (<a href="https://dmptool.org" class="uri">https://dmptool.org</a>) has a template for NOAA. These represent a minimum requirement and, in fact, PEP data management plans will likely be more thorough. All data management plans should be developed in collaboration with the PEP data science lead (Stacie Koslovsky) and established prior to data collection. Work with S. Koslovsky to complete and submit a data management plan (as appropriate).</p>
</details></section>
<section id="integrate" class="level2">
<h2 class="anchored" data-anchor-id="integrate">Integrate</h2>
<details>
<summary>
Click to Expand
</summary>
<p>For on-going projects, we will incorporate feedback from the previous year’s/season’s effort to make improvements to the upcoming effort. For new projects, this might include considering lessons learned from other projects.</p>
</details></section>
<section id="collect" class="level2">
<h2 class="anchored" data-anchor-id="collect">Collect</h2>
<details>
<summary>
Click to Expand
</summary>
<p>It is critical to have clear instructions and expectations for how data are collected. It is also important to consider how to handle and/or document deviations from planned data collection – which are bound to happen!</p>
<section id="field-data-collection" class="level3">
<h3 class="anchored" data-anchor-id="field-data-collection">Field Data Collection</h3>
<p>During field data collection, begin preliminary data management, as able:</p>
<ul>
<li><p>Review datasheets for data recording errors.</p></li>
<li><p>Scan datasheets for backup.</p></li>
<li><p>Use temporary storage on external/portable drives, cloud providers, or individual government laptops to backup images, scanned datasheets and other critical files.</p></li>
<li><p>Enter field data into cloud-based data entry tools.</p></li>
</ul>
</section>
</details></section>
<section id="process" class="level2">
<h2 class="anchored" data-anchor-id="process">Process</h2>
<details>
<summary>
Click to Expand
</summary>
<p>After field work is completed, our aim will be to download, process and QA/QC data as soon as possible. All data should be archived to the network no later than <strong>one month</strong> after the completion of the project.</p>
<section id="raw-and-original-data" class="level3">
<h3 class="anchored" data-anchor-id="raw-and-original-data">Raw and Original Data</h3>
<p>The AFSC network (LAN) is the primary storage location for all data collected by PEP.</p>
<ul>
<li><p>In practice, the transfer of data from field collection storage to the network should be the first priority upon returning from the field. Temporary storage of files on external/portable drives, cloud providers, individual government laptops should not exceed 30 days after return from the field.</p></li>
<li><p>After data are transferred to the network, files and associated information should be reviewed for consistency.</p></li>
<li><p>In general, ‘raw’ or ‘original’ data collected in the field should not be altered.</p></li>
<li><p>In cases where the data are known to be incorrect or a more correct value is known, those files or entries should be edited.</p></li>
<li><p>A common occurrence is for files to not be named properly in the field. File names should be corrected at this time.</p></li>
<li><p>Additionally, data transcription errors can occur, and this is the appropriate time to fix those errors.</p></li>
</ul>
</section>
<section id="processed-and-final-data" class="level3">
<h3 class="anchored" data-anchor-id="processed-and-final-data">Processed and Final Data</h3>
<p>After raw/original data are archived to the network, data should be processed and reviewed for outliers. Data entry and QA/QC should be processed and finalized <strong>within three months</strong> of the field effort, so that other efforts (e.g. image review, counting) are able to proceed in a timely manner.</p>
<p>The processing and review steps will vary by project, but generally:</p>
<ul>
<li><p>We will use automated processes for streamlining and documenting processes whenever possible.</p></li>
<li><p>Custom data-entry forms and/or spatial processing templates will be created for data entry and processing when needed.</p></li>
<li><p>Queries and/or other data visualizations will be used to facilitate data QA/QC.</p></li>
<li><p>Spatial data (grid and other products) will be stored in their native projection in the database.</p>
<ul>
<li><p>For many projects, this will be WGS-84.</p></li>
<li><p>Spatial data from outside sources (e.g. environmental data) will be stored in the original projection. This may require data to be reprojected for specific analyses or other needs.</p></li>
</ul></li>
<li><p>Environmental data will be updated in the pep DB annually in August. Additional environmental data can be accessed from other data sources online. If you need help with this, contact E. Richmond or S. Koslovsky.</p></li>
</ul>
</section>
<section id="backup-procedures" class="level3">
<h3 class="anchored" data-anchor-id="backup-procedures">Backup Procedures</h3>
<p>All data copied to the network are backed up offsite in one of two ways: snap-mirrored to another NMFS facility, tape backup delivered off site (reserved for large files that change infrequently (e.g. imagery, acoustic files, video). In addition, any data that are snap-mirrored are also backed up differentially and that allows incremental restoration (daily for 7 days, weekly for 6 months).</p>
</section>
</details></section>
<section id="debrief" class="level2">
<h2 class="anchored" data-anchor-id="debrief">Debrief</h2>
<details>
<summary>
Click to Expand
</summary>
<p>After fieldwork is completed, part of the larger project debrief should include time for discussing in-field data collection and management. The general comments and any specific actionable changes uncovered during the debrief should be incorporated into the INTEGRATE step in the next cycle of the data management workflow for that project.</p>
</details></section>
<section id="analyze" class="level2">
<h2 class="anchored" data-anchor-id="analyze">Analyze</h2>
<details>
<summary>
Click to Expand
</summary>
<p>Part of the data workflow is to ensure the final products are easily usable and accessible for analyses. Accessing data for each analysis will be different; work with S. Koslovsky to identify the most efficient way to extract data from the DB and/or network for your needs.</p>
</details></section>
<section id="evaluate" class="level2">
<h2 class="anchored" data-anchor-id="evaluate">Evaluate</h2>
<details>
<summary>
Click to Expand
</summary>
<p>We want to emphasize the feedback loop from analytical processes to data management. Our goal with data management is to streamline processing and extraction of information for analyses. This may be updates/improvements to data management processes over time. Feedback and communication are important for ensuring data products meet current and future analytical needs, and this information should be incorporated into the INTEGRATE step in the next cycle of the data management workflow for any related field projects.</p>
</details></section>
<section id="share" class="level2">
<h2 class="anchored" data-anchor-id="share">Share</h2>
<details>
<summary>
Click to Expand
</summary>
<p>Data are considered final when a project is completed (e.g., ChESS) or when the annual data processing for a project (e.g., harbor seal surveys) is completed. After data are processed and considered final, S. Koslovsky will notify PI and data sharing staff to prepare final datasets for archive and distribution (when appropriate).</p>
<section id="overall-workflow" class="level3">
<h3 class="anchored" data-anchor-id="overall-workflow">Overall Workflow</h3>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="docs/PEP_DataMgmtPlan_files/images/FinalDataWorkflow.png" class="img-fluid figure-img"></p>
</figure>
</div>
</section>
<section id="metadata" class="level3">
<h3 class="anchored" data-anchor-id="metadata">Metadata</h3>
<p>NOAA Fisheries requires that all data collected be documented within the official NOAA Fisheries metadata repository, <a href="https://inport.nmfs.noaa.gov/inport">InPort</a>. InPort provides an extensive suite of tools for editing and managing metadata. For PEP projects, we will update InPort as follows:</p>
<ol type="1">
<li><p>S. Koslovsky and C. Christman will work with project leads to establish a metadata plan and to get started with InPort.</p></li>
<li><p>The project lead will enter the appropriate metadata information into a GoogleForm that will be used to create and maintain program metadata records.</p>
<ol type="1">
<li><p>The responsibility for creating, editing, and maintaining the GoogleForm entries is the responsibility of the project lead. Because these records will be available to the public and, in many cases, represent the authoritative documentation of the data set, project leads should devote an appropriate amount of time and thought to development of metadata records.</p></li>
<li><p>Once the necessary and complete information is entered into the GoogleForm, notify C. Christman.</p></li>
</ol></li>
<li><p>C. Christman will complete the necessary updates/entries.</p></li>
</ol>
<p>For more information on metadata workflow, <a href="https://docs.google.com/document/d/1kmEoKzXjS-O2r-6euBCL7A3We8mfAre3fpcEvytBD84/edit?tab=t.0">review this documentation</a> and/or contact C. Christman.</p>
</section>
<section id="online-repositories" class="level3">
<h3 class="anchored" data-anchor-id="online-repositories">Online Repositories</h3>
<p>Much of the data collected and processed as part of PEP research activities is intended for public release, either in compliance with NOAA policies (e.g. Public Access to Research Results [PARR]) or in support of best practices related to open science and reproducible research. Keeping track of the evolving policies and expanding tools/repositories available can be challenging. Here, we outline our plan for the use of available repositories.</p>
<p>If you are publishing a manuscript and the journal requires the data to be provided on an open data portal, work with S. Koslovsky to identify the most appropriate repository and to ensure metadata are created.</p>
</section>
</details></section>
<section id="deiab-in-data-management-workflow" class="level2">
<h2 class="anchored" data-anchor-id="deiab-in-data-management-workflow">DEIAB in Data Management Workflow</h2>
<p>Each step in this process is an opportunity to ensure we are achieving DEAIB in our data management practices within our projects. Misses at each step have the potential to introduce delays in processing, to cause staff to feel unwelcome, or to introduce errors.</p>
<ul>
<li><p>Planning</p>
<ul>
<li>Is the planning process inclusive (as appropriate)?</li>
</ul></li>
<li><p>Integrating</p>
<ul>
<li>Has feedback from a diversity of staff been incorporated into changes for the upcoming work?</li>
</ul></li>
<li><p>Collecting</p>
<ul>
<li><p>Are there clear instructions for how data will be collected?</p></li>
<li><p>Have staff been trained and are they confident in their ability to collect data?</p></li>
<li><p>Are staff comfortable sharing mis-steps or issues with data collection?</p></li>
</ul></li>
<li><p>Processing</p>
<ul>
<li><p>Have staff been given the appropriate training for processing data?</p></li>
<li><p>Does documentation of processing steps exist? Is it accessible?</p></li>
<li><p>Do we have processes in place to reduce process errors, and especially individual responsibility?</p></li>
</ul></li>
<li><p>Debriefing</p>
<ul>
<li>Are all perspectives being heard?</li>
</ul></li>
<li><p>Analyzing</p>
<ul>
<li><p>Is the information accessible and appropriate for analysis?</p></li>
<li><p>Does the process for retrieving and updating data clear and avoidant of errors?</p></li>
<li><p>Is the process for sharing data for an analysis inclusive of a diversity of perspectives? The people who collected the data might have different concerns about how data are analyzed based on their own observations/experiences than the project lead or data lead…</p></li>
</ul></li>
<li><p>Evaluating</p>
<ul>
<li>Do we have a process for evaluating a dataset and/or analysis that is inclusive of all staff?</li>
</ul></li>
<li><p>Sharing</p>
<ul>
<li><p>Are we ensuring that our data are accessible from an availability perspective?</p></li>
<li><p>Are the data available in a format that is accessible?</p></li>
<li><p>Are we ensuring that our data are accessible from an equity perspective? <a href="https://www.urban.org/sites/default/files/2021/06/08/do-no-harm-guide-recommendations.pdf">From the Urban Institute</a>, these are considerations for applying equity awareness in data visualization.</p></li>
</ul></li>
</ul>
</section>
</section>
<section id="on-boarding" class="level1">
<h1>On-boarding</h1>
<p>S. Koslovsky will orient new staff to the general information provided in the PEP Data Management Plan and the specifics detailed within this section. Staff responsible for other program-wide tools (e.g. PEP Dashboard, inventory DB, Zotero) should provide resources/training as needed.</p>
<section id="software-requirements-and-recommendations" class="level2">
<h2 class="anchored" data-anchor-id="software-requirements-and-recommendations">Software Requirements and Recommendations</h2>
<details>
<summary>
Click to Expand
</summary>
<p>Below is a list of software required for PEP computers.</p>
<table class="table">
<colgroup>
<col style="width: 21%">
<col style="width: 78%">
</colgroup>
<thead>
<tr class="header">
<th>Program</th>
<th>Justification</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>QGIS</td>
<td>For the most direct connection to PEP database for spatial data</td>
</tr>
<tr class="even">
<td>64-bit PostgreSQL ODBC driver</td>
<td>For connecting to PEP database via Microsoft Access</td>
</tr>
<tr class="odd">
<td>Microsoft Access 2016</td>
<td>For using PEP database front-ends (this is part of the Microsoft Suite, so might already be installed by default)</td>
</tr>
<tr class="even">
<td>EndNote and/or Zotero</td>
<td>For accessing the PEP library</td>
</tr>
<tr class="odd">
<td>ACDSee</td>
<td>For image management (field photos and data); we also have licenses for Lightroom</td>
</tr>
<tr class="even">
<td>Adobe Acrobat</td>
<td>For managing data and forms stored in pdf</td>
</tr>
</tbody>
</table>
<p>Below is a list of software recommended for PEP computers.</p>
<table class="table">
<colgroup>
<col style="width: 46%">
<col style="width: 53%">
</colgroup>
<thead>
<tr class="header">
<th>Program</th>
<th>Justification</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Anaconda(or miniconda) with Python>=3.6</td>
<td>Lots of our tools require Python</td>
</tr>
<tr class="even">
<td>R</td>
<td>Lots of our tools require R (and kept updated)</td>
</tr>
<tr class="odd">
<td>RStudio</td>
<td>A user-friendly GUI for R</td>
</tr>
<tr class="even">
<td>ArcGIS Pro</td>
<td>Spatial processing and analysis</td>
</tr>
<tr class="odd">
<td>VLC Media Player</td>
<td>For video management</td>
</tr>
</tbody>
</table>
<section id="details-for-odbc-driver-installation" class="level3">
<h3 class="anchored" data-anchor-id="details-for-odbc-driver-installation">Details for ODBC Driver Installation</h3>
<p>In order to connect to the DB from Access, you will need to ask IT to update your ODBC Data Sources to allow connection to PostgreSQL. Here’s a website for reference: <a href="#0" class="uri">https://help.interfaceware.com/v6/connect-to-postgresql-from-windows-with-odbc</a>. The most recent driver from this site: <a href="#0" class="uri">https://www.postgresql.org/ftp/odbc/versions/msi/</a> (x64.zip version).</p>
<p>The connection details for setting up the PostgreSQL ANSI ODBC connection configured after the driver is installed are in the screenshot below. Just use your own username and password. Once they add PostgreSQL to the System DSN, they can test your connection using an existing Access database by using the “Test” button in the Driver Setup screen.</p>
<p><img src="images/clipboard-4182424335.png" class="img-fluid"></p>
</section>
</details></section>
<section id="data-storage" class="level2">
<h2 class="anchored" data-anchor-id="data-storage">Data Storage</h2>
<section id="individual-storage-resources" class="level3">
<h3 class="anchored" data-anchor-id="individual-storage-resources">Individual Storage Resources</h3>
<details>
<summary>
Click to Expand
</summary>
<p>No data should be stored exclusively in any of these resources.</p>
<p>(this section to be completed after PEP Data Days 2025)</p>
<section id="laptop" class="level4">
<h4 class="anchored" data-anchor-id="laptop">Laptop</h4>
</section>
<section id="google-drive" class="level4">
<h4 class="anchored" data-anchor-id="google-drive">Google Drive</h4>
</section>
<section id="individual-users-folders-on-lan" class="level4">
<h4 class="anchored" data-anchor-id="individual-users-folders-on-lan">Individual Users folders on LAN</h4>
</section>
</details></section>
<section id="shared-storage-resources" class="level3">
<h3 class="anchored" data-anchor-id="shared-storage-resources">Shared Storage Resources</h3>
<details>
<summary>
Click to Expand
</summary>
<section id="pep_googledrive" class="level4">
<h4 class="anchored" data-anchor-id="pep_googledrive">PEP_GoogleDrive</h4>
<p>We have a centralized Google Drive (tied to a generic email account) that serves as our program <a href="https://drive.google.com/drive/folders/1XMadJkz4AqgHctLn0U2xnVChSjHxnyjb">Google Drive repository</a>. Google Drive is an excellent location for sharing files among staff for easy off-site access. Google drive, however, should not be treated as a final storage location for data. Files can be stored here in perpetuity and as appropriate, but data should not. As projects and/or collaborations are completed, data should either be deleted (if they are temporary) or archived to the appropriate location on the network.</p>
</section>
<section id="network-aka-lan" class="level4">
<h4 class="anchored" data-anchor-id="network-aka-lan">Network (aka LAN)</h4>
<p>All original data and supplemental information to data collection will be stored on the LAN in the appropriate location (detailed below). If you have questions about where to store data for a project, contact S. Koslovsky. </p>
<ul>
<li><p>For those projects where data are stored on other locations on the network (e.g. CHESS, harbor seal survey images), create shortcuts to these other network locations within each project folder. </p></li>
<li><p>Final files from intermediate locations will need to be moved to the appropriate location(s) on the network when processing is completed OR when the project becomes collaborative and multiple staff will need access. Project folders should contain final data, datasheets, databases, manuals, metadata, scripts, and intermediate files (as needed).</p></li>
<li><p>For folders under Polar\Analytics, Polar\Data, Polar\Projects on the network, this is a suggested folder structure for each project subfolder. If data easily fit into this schema, use it; if not, data should be organized and documented such that anyone in the program can identify what is where. Not all projects will be able to fit this mold, but this structure will help keep data organized across multiple projects.</p>
<ul>
<li><p><strong>\Data</strong> – contains data collected in numerous formats, including databases, spreadsheets, etc.</p></li>
<li><p><strong>\Docs</strong> – contains project proposals, reports, processing manuals, original field data sheets, etc.</p></li>
<li><p><strong>\GIS</strong> – spatial data and programs associated with the project.</p></li>
<li><p><strong>\Manuscripts</strong> – drafts of manuscript and any associated files.</p></li>
<li><p><strong>\Presentations</strong> – any presentations associated with the project.</p></li>
<li><p><strong>\Scripts</strong> – R, Python, etc. script files.</p></li>
<li><p><strong>READ_MEs</strong> – include read me files wherever appropriate to provide additional information on files within folders.</p></li>
</ul></li>
</ul>
<p>We store our files in a variety of locations on the AFSC internal network:</p>
<ul>
<li><p><a href="about:blank">\\nmfs\akc-nmml\NMML_CHESS_Imagery</a> - images and associated from 2016 CHESS survey</p></li>
<li><p><a href="about:blank">\\nmfs\akc-nmml\Polar</a></p>
<ul>
<li><p>\Analytics</p>
<ul>
<li><p>Will store files related to analytical processes.</p></li>
<li><p>Subfolders should be named YYYY_LastNamePI_ProjectTitle.</p></li>
<li><p>The PI is ultimately responsible for ensuring that all files get stored in this location.</p></li>
<li><p>You do not have to store files in this location if it is inconvenient while you are working; files just need to be archived from computers and Google Drive when processing is complete.</p></li>
</ul></li>
<li><p>\Data</p>
<ul>
<li><p>Will store files related to data collection and creating final datasets.</p></li>
<li><p>Each folder should be a project name, but we do not want too many files to clutter up this location such that it becomes hard to find project data. Folders are likely to rarely be added the root directory.</p></li>
<li><p>Archive datasheets, raw data, etc. should be stored within this directory. When data processing is complete (either for a field season or entirely), files from computers and Google Drive need to be archived to this location.</p></li>
<li><p>\CapturesAndHandling</p>
<ul>
<li>\Data_CruiseUnderway: subfolders should be named YYYY_Location_Species_CruiseName and should match planning folder name under Projects.</li>
</ul></li>
</ul></li>
<li><p>\Legacy</p>
<ul>
<li><p>Will store older project planning and data files that need to be kept for long-term storage, but were not broken out into the new structure.</p></li>
<li><p>Subfolders should be named YYYY-YYYY_ProjectName (as appropriate).</p></li>
</ul></li>
<li><p>\Projects</p>
<ul>
<li><p>Will store files, maps and other information related to field operations (e.g. a folder for coastal harbor seal surveys would include maps for the plane, instructions for the field season, etc.).</p></li>
<li><p>Subfolders should be named in the following format: YYYY_Region_Species_Platform (e.g. 2018_BeringSea_IceSeal_DysonCruise).</p></li>
<li><p>After field work is complete, associated files from computers and Google Drive need to be archived to this location.</p></li>
</ul></li>
<li><p>\ProgramMgmt – will store budget, library, permit, etc. information.</p></li>
<li><p>\ResearchProducts</p>
<ul>
<li><p>\Manuscripts: subfolders or pdfs should be named YYYY_Author_Subject.</p></li>
<li><p>\Presentations: final presentations should be named YYYY_Author_Subject.</p></li>
<li><p>\Reports: subfolders or pdfs should be named YYYY_ Subject.</p></li>
<li><p>\WorkshopsAndConferences: subfolders or pdfs should be named YYYY_Author_Subject.</p></li>
</ul></li>
<li><p>\Users</p>
<ul>
<li><p>Subfolder should be named with the staff member’s last name.</p></li>
<li><p>Subfolders should been cleaned out and what needs to be kept moved to _Legacy when staff are no longer employed/involved.</p></li>
</ul></li>
</ul></li>
<li><p><a href="about:blank">\\nmfs\akc-nmml\Polar_Imagery</a></p>
<ul>
<li><p>\Field_Photos – Portfolio monitors this folder for new images</p></li>
<li><p>\Field_Video – Portfolio monitors this folder for new videos</p></li>
<li><p>\Inbox</p></li>
<li><p>\PhotoConsentForms – contains consent forms</p></li>
<li><p>\Portfolio</p></li>
<li><p>\Portfolio_Previews</p></li>
<li><p>\Surveys_HS – contains original and processed images for coastal and glacial harbor seal surveys</p></li>
<li><p>\Surveys_IceSeals – contains original and processed images for ice seal surveys, except CHESS and some BOSS (which can be found on NMML_CHESS_Imagery and Polar_Imagery_2, respectively)</p></li>
<li><p>\Techniques_Test – contains images collected from flights testing equipment and image processing</p></li>
<li><p>\User_Photos – photos taken by staff that do not need to be managed in Portfolio; each staff member has a folder, and they are responsible for managing the photos in this location</p></li>
</ul></li>
<li><p><a href="about:blank">\\nmfs\akc-nmml\Polar_Imagery_2</a></p>
<ul>
<li><p>\Aleutian_HarborSeals_CaptureSiteAnalysis</p></li>
<li><p>\Surveys_IceSeasl_BOSS_2013</p></li>
<li><p>\Surveys_Iliamna_2013</p></li>
<li><p>\Techniques_test</p></li>
<li><p>\xTEMP_BOSS13_Skeyes - can this be deleted?</p></li>
</ul></li>
<li><p><a href="about:blank">\\nmfs\akc-nmml\Polar_Imagery_3</a></p>
<ul>
<li>\jobss_2021</li>
</ul></li>
</ul>
</section>
</details></section>
<section id="pep-postgresql-database" class="level3">
<h3 class="anchored" data-anchor-id="pep-postgresql-database">PEP PostgreSQL Database</h3>
<details>
<summary>
Click to Expand
</summary>
<section id="overview-1" class="level4">
<h4 class="anchored" data-anchor-id="overview-1">Overview</h4>
<p>The table below details the final data are stored in the pep PostgreSQL database.</p>
<ul>
<li><p>All staff will have read-only access to all data. </p></li>
<li><p>Staff will only have read-write access to the data they are required to edit.</p></li>
<li><p>There are a number of automated scripts that will replicate and/or process data into their final product. This database should be the go-to location data anytime data are needed. <strong>Intermediate copies and/or previous exports should always be replaced to ensure you are using the most up-to-date data.</strong></p></li>
<li><p>QGIS is the only software we will use for editing spatial data stored in the DB. It will also be used for importing shapefiles to PostGIS. ArcGIS or other GIS software should only be used for viewing spatial data stored in the DB or from the pepGeodatabase.</p></li>
</ul>
</section>
<section id="current-schemas" class="level4">
<h4 class="anchored" data-anchor-id="current-schemas">Current Schemas</h4>
<table class="table">
<colgroup>
<col style="width: 9%">
<col style="width: 5%">
<col style="width: 85%">
</colgroup>
<thead>
<tr class="header">
<th><strong>Project</strong></th>
<th><strong>Schema</strong></th>
<th><strong>Details</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>Acoustics</strong></td>
<td>acoustics</td>
<td>Contains data received from CAEP Acoustics program and pre-processed for import into the DB.</td>
</tr>
<tr class="even">
<td><strong>PEP Dashboard</strong></td>
<td>administrative</td>
<td>Contains data associated with the PEP Dashboard (activity planning, data tracking, etc.). An Access front-end is available for managing data.</td>
</tr>
<tr class="odd">
<td><strong>Annotations</strong></td>
<td>annotations</td>
<td>Contains data associated with application of ML models to imagery; the annotations and imagery data are stored within their project schemas; this schema tracks information related to annotation management. An Access front-end is available for managing data.</td>
</tr>
<tr class="even">
<td><strong>Base</strong></td>
<td>base</td>
<td>Contains “THE” grid and environmental covariates extracted to the grid.</td>
</tr>
<tr class="odd">
<td><strong>UAS Body Condition</strong></td>
<td>body_condition</td>
<td>Contains image, LRF and measurement data related to the UAS body condition data. An Access front-end is available for managing data.</td>
</tr>
<tr class="even">
<td><strong>Capture</strong></td>
<td>capture</td>
<td>Data have been imported from Excel files, which were exports from the Oracle DB. An Access front-end is available for managing data.</td>
</tr>
<tr class="odd">
<td><strong>Environmental</strong></td>
<td>environ</td>
<td>Includes NARR weather data from 2004-2016 and NSIDC CDR sea ice concentration data*.</td>
</tr>
<tr class="even">
<td><strong>Inventory</strong></td>
<td>inventory</td>
<td>Contains data for managing PEP gear inventory and field packlists. An Access front-end is available for managing data.</td>
</tr>
<tr class="odd">
<td><strong>Species Misclassification</strong></td>
<td>species_misclass</td>
<td>Contains ice seal species misclassification information from across all ice seal survey projects.</td>
</tr>
<tr class="even">
<td><strong>Stock</strong></td>
<td>stock</td>
<td>All data have been migrated from pepgeo to this database.</td>
</tr>
<tr class="odd">
<td><strong>BOSS</strong></td>
<td>surv_boss</td>
<td>All data from Oracle and from the original FMC logs have been ingested into DB. An Access front-end is available for viewing/querying data.</td>
</tr>
<tr class="even">
<td><strong>ChESS</strong></td>
<td>surv_chess</td>
<td>Data are uploaded to DB from CSV files through R script. All imported data have been QA/QC’ed. A final effort trackline is available. An Access front-end is available for viewing/querying data.</td>
</tr>
<tr class="odd">
<td><strong>Ice Seals: 2024</strong></td>
<td>surv_ice_seals_2024</td>
<td>Image metadata/inventory and annotations from 2024 Ice Seal Surveys (in Bering Sea).</td>
</tr>
<tr class="even">
<td><strong>Ice Seals: 2025</strong></td>
<td>surv_ice_seals_2025</td>
<td>To be created/populated after 2025 surveys.</td>
</tr>
<tr class="odd">
<td><strong>JoBSS</strong></td>
<td>surv_jobss</td>
<td>Image metadata/inventory and annotations from 2021 Beaufort Sea Surveys. An Access front-end is available for viewing/querying data.</td>
</tr>
<tr class="even">
<td><strong>Ice Seal: Polar Bear</strong></td>
<td>surv_polar_bear</td>
<td>Image metadata/inventory and annotations from 2019 surveys out of Kotzebue and Deadhorse targeting polar bears.</td>
</tr>
<tr class="odd">
<td><strong>Harbor Seal: Coastal Surveys</strong></td>
<td>surv_pv_cst</td>
<td>Image metadata/inventory and counts from 2004-2021 are currently QA/QC’ed and available in the DB. ADF&G, NOAA 1996-1997, NOAA 1998-2002 and NOAA 2003 data are available in the DB. Authoritative Iliamna and Pribilof data are also actively managed within this schema. An Access front-end is available for managing data.</td>
</tr>
<tr class="even">
<td><strong>Harbor Seal: Glacial Surveys</strong></td>
<td>surv_pv_gla</td>
<td>Fliight data and counts from surveys are available in the DB. An Access front-end is available for managing data.</td>
</tr>
<tr class="odd">
<td><strong>Ice Seal: Kotz</strong></td>
<td>surv_test_kotz</td>
<td>Image data and annotations from 2019 testing of in-flight system out of Kotzebue. Fight 01 and R camera views should not be used for any AI/ML.</td>
</tr>
<tr class="even">
<td><strong>Telemetry</strong></td>
<td>telem</td>
<td>All field-related telemetry information has been entered into the DB. Tag inventory and deployment information is managed through the Capture Access front-end.</td>
</tr>
</tbody>
</table>
<p>*The sea ice data stored on the pep database are the CDR and sea ice extent products. The original versions of these datasets are also on the network. In addition to these datasets, the bootstrap and nimbus are also stored on the network. The CDR and sea ice extent should be the go-to sea ice data products, which is why they are processed and available in the pep database.</p>
</section>
<section id="naming-conventions" class="level4">
<h4 class="anchored" data-anchor-id="naming-conventions">Naming Conventions</h4>
<p>General Guidelines</p>
<ul>
<li><p>Coordinate with S. Koslovsky before adding new data tables to the database or if you need a stored query added to the database.</p></li>
<li><p>Do not use spaces in database item names. Use underscores between words.</p></li>
<li><p>Data for analyses:</p>
<ul>
<li><p>If you are using data products that are finalized (e.g. CHESS survey) and you need data processed prior to use, the processed data should be stored in a view/query.</p></li>
<li><p>If you are using data products that will continue to be updated (e.g. telemetry data) and you need the processed data preserved, the processed data should either be stored in the database as a separate table or be exported and stored elsewhere (network or computer) with other associated files.</p></li>
</ul></li>
<li><p>For tables</p>
<ul>
<li><p>lku_ - use this prefix for look-up tables.</p></li>
<li><p>geo_ - use this prefix for spatially explicit datasets that are stored in the postgres DB. If you have a table that has spatial data, use the tbl_ prefix. If you have raster, basemap, etc., use the geo_ prefix.</p></li>
<li><p>tbl_ - use this prefix for data tables. These can be spatially enabled.</p></li>
<li><p>res_ - use this prefix for results tables. These are tables that are derived from data stored in tables (tbl_) that need to be stored but are not “raw” data. These should have clear indication of what they are, and if appropriate, a date</p></li>
</ul></li>
<li><p>For views</p>
<ul>
<li><p>qa_ - use this prefix for QA/QC queries.</p></li>
<li><p>res – use this prefix for views that are for analytical purposes.</p></li>
<li><p>Anything else – use logical consistency for any other views stored.</p></li>
</ul></li>
</ul>
</section>
<section id="accessing-data" class="level4">
<h4 class="anchored" data-anchor-id="accessing-data">Accessing Data</h4>
<section id="spatial-data" class="level5">
<h5 class="anchored" data-anchor-id="spatial-data"><strong>Spatial Data</strong></h5>
<p>For viewing spatial data, you can access data from one of these platforms:</p>
<ul>
<li><p>pep_geodatabase</p></li>
<li><p>ArcMap and ArcPro (to connect to the spatial data available on AGOL)</p>
<ul>
<li><p>ArcMap</p>
<ul>
<li><p>Click on the Add Data button, and select “Add Data From ArcGIS Online”</p></li>
<li><p>In the upper right corner of the pop-up window, sign into ArcGIS online using your CAC credentials.</p></li>
<li><p>Once you are logged in, you can add data shared with the PEP SpatialData group under “My Groups”.</p></li>
</ul></li>
<li><p>ArcPro</p>
<ul>
<li><p>You are automatically logged into ArcGIS online when you open ArcPro. </p></li>
<li><p>Once you have created a new project, you can add data from the Portal. Data shared to the PEP Spatial Data group will be accessible from Groups under Portal.</p></li>
</ul></li>
<li><p>ArcGIS Online</p></li>
</ul></li>
</ul>
<!-- -->
<ul>
<li><p>Once signed in to ArcGIS online, click on “Groups” at the top of the page</p></li>
<li><p>Click on the group, “PEP Spatial Data” and open the item page for the dataset you want to download</p></li>
<li><p>On the right side of the page, go to “Export Data” and choose the format you want to export</p></li>
<li><p>An export box will appear with a default title for the data export. You can rename this if you’d like. You must enter at least one tag for your dataset to allow for export. Exported items from ArcGIS online are stored in the root folder of “My Content”. If exporting to shapefile, a compressed file (.zip) is created and can be downloaded and saved your computer.</p></li>
</ul>
<!-- -->
<ul>
<li><p>QGIS (to connect to spatial data </p>
<ul>
<li><p>Right click PostGIS, and select “New Connection…”</p></li>
<li><p>In the Create a New PostGIS Connection window, specify the following options; then, click OK.</p>
<ul>
<li><p>Name: pep</p></li>
<li><p>Service: (leave blank)</p></li>
<li><p>Host: 161.55.120.122</p></li>
<li><p>Port: 5432</p></li>
<li><p>Database: pep</p></li>
<li><p>Username: your user name (check box to save)</p></li>
<li><p>Password: your password (check box to save)</p></li>
</ul></li>
<li><p>Once created, you will see a new connection to the database under PostGIS. The available data are listed under each schema, and only spatial data are visible.</p></li>
</ul></li>
</ul>
<p>For entering spatial data, process the data as instructed for each individual project. </p>
</section>
<section id="tabular-data" class="level5">
<h5 class="anchored" data-anchor-id="tabular-data"><strong>Tabular Data</strong></h5>
<p>For viewing tabular data: </p>
<ul>
<li><p>Project-specific Access frontends are available for the following projects:</p>
<ul>
<li><p>Capture (for entering/viewing data entered from datasheets)</p></li>
<li><p>Coastal Harbor Seal Surveys (for track level metadata)</p></li>
</ul></li>
<li><p>You can directly connect to the pep DB using R with the RPostgreSQL package. This connection string will get you started: con <- dbConnect(PostgreSQL(), dbname = “pep”, host = “161.55.120.122”, user = “UserName”, password = “Password”)</p></li>
<li><p>If you need access to or exports of other tabular data from the pep DB, contact S. Koslovsky.</p></li>
</ul>
</section>
<section id="pepdataconnect-r-package" class="level5">
<h5 class="anchored" data-anchor-id="pepdataconnect-r-package">pepDataConnect (R package)</h5>
<pre><code>install.packages("devtools") devtools::install_github("StacieKoslovsky-noaa/pepDataConnect")</code></pre>
<p>There are of tables (and linkages among them) in PEP database, and this can be overwhelming to learn and can lead to potential issues in how data are linked together or extracted from the DB. The pepDataConnect R package was created to easily connect PEP staff to the database and to ensure the quality of the data being retrieved. The R code snippet below will guide you through the process of installing the R package and an example of code to get a data table from the DB. There is <em>extensive</em> documentation in the</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Getting started with pepDataConnect remotes::install_github('staciekoslovsky-noaa/pepDataConnect') # To connect to the database, create a connection con <- pepDataConnect::pep_connect() # To load data into your R workspace, use one of the table functions data <- pepDataConnect::surv_jobss.tbl_detections_processed_ir(con)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
<p>Currently, the pepDataConnect R package has been set-up to interact with the database for the following schemas:</p>
<ul>
<li><p>Coastal harbor seal survey data (surv_pv_cst)</p></li>
<li><p>JoBSS ice seal survey (surv_jobss)</p></li>
</ul>
<p>Other schemas will slowly be added to the R package, as prioritized each year</p>
</section>
</section>
<section id="backup-procedure" class="level4">
<h4 class="anchored" data-anchor-id="backup-procedure">Backup Procedure</h4>
<p>The pepDB will be backed up according to the following schedule:</p>
<ul>
<li><p>Daily directory backup: The pep DB will be backed up daily to tape. The last 5 days will also be stored on the VM. </p></li>
<li><p>Monthly full VM backup: The virtual machine on which the pep DB resides will be fully backed up to tape monthly. The last 3 months will also be stored on the VM.</p></li>
</ul>
<p>The DB backups will be tested every 6 months (March and September) to ensure they are working properly, and data can be restored.</p>
</section>
</details></section>
</section>
<section id="frequently-used-program-wide-tools" class="level2">
<h2 class="anchored" data-anchor-id="frequently-used-program-wide-tools">Frequently Used Program-wide Tools</h2>
<details>
<summary>
Click to Expand
</summary>
<section id="pep-dashboard" class="level3">
<h3 class="anchored" data-anchor-id="pep-dashboard">PEP Dashboard</h3>
<p>The PEP dashboard is a schema (administrative) within the PEP PostgreSQL database that is used for tracking the roles of staff on projects and associated timelines and for entering responsibilities into the PAWS performance planning system. For more information:</p>
<ul>
<li><p>Documentation</p></li>
<li><p><a href="https://htmlpreview.github.io/?https://github.com/staciekoslovsky-noaa/PEP_Dashboard/blob/main/SchedulingReport/SchedulingReport.html">Weekly report</a></p></li>
<li><p>Contact M. Cameron</p></li>
</ul>
</section>
<section id="inventory-database" class="level3">
<h3 class="anchored" data-anchor-id="inventory-database">Inventory Database</h3>
<p>This database is a schema (inventory) within the PEP PostgreSQL database and tracks inventory of gear and supplies used by staff in PEP for projects. For more information:</p>
<ul>
<li><p>Documentation</p></li>
<li><p>Contact H. Ziel</p></li>
</ul>
</section>
<section id="extensis-portfolio" class="level3">
<h3 class="anchored" data-anchor-id="extensis-portfolio">Extensis Portfolio</h3>
<p>Extensis Portfolio is software used for tagging and querying images taken during PEP projects. For more information:</p>
<ul>
<li><p>Documentation</p></li>
<li><p>Contact S. Walcott</p></li>
</ul>
</section>
<section id="zotero" class="level3">
<h3 class="anchored" data-anchor-id="zotero">Zotero</h3>
<p>Zotero houses a central library of journal articles, reports and other references. For more information:</p>
<ul>
<li><p>Documentation</p></li>
<li><p>Contact S. Walcott</p></li>
</ul>
</section>
</details></section>
</section>
<section id="off-boarding" class="level1">
<h1>Off-boarding</h1>
<p>(this section to be completed after PEP Data Days 2025)</p>
</section>
</main>
<!-- /main column -->
<script id="quarto-html-after-body" type="application/javascript">
window.document.addEventListener("DOMContentLoaded", function (event) {
const toggleBodyColorMode = (bsSheetEl) => {
const mode = bsSheetEl.getAttribute("data-mode");
const bodyEl = window.document.querySelector("body");
if (mode === "dark") {
bodyEl.classList.add("quarto-dark");
bodyEl.classList.remove("quarto-light");