-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREAD.ME
722 lines (546 loc) · 32.8 KB
/
READ.ME
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
=======================================================================
@(#) SCCS module: READ.ME version: 1.1
Creation date: 10/13/97
=======================================================================
README file for moma.mpi
========================
This is an early release of moma.mpi, the mpi version of the moma and
moma.pvm codes. It has been tested succesfully on a pair of Silicon
Graphics RS10000 workstations running a 64 bit operating system and the
mpich version of the MPI message passing interface standard.
Except for the change to mpi, the code is the same as that used in moma.pvm, whose READ.ME file follows below.
Information on the MPI standard can be obtained from:
http://www.mcs.anl.gov/mpi/
http://www.mpi-forum.org/
Copies of mpich can be obtained from ...
http://www.mcs.anl.gov/mpi/mpich/
Introduction
============
moma.mpi is a mpi implementation of the message passing ocean model
moma.pvm. The latter used the pvm message passing system. The
initial release of the mpi codeis based on version 1.10 of moma.pvm.
This mpi release is known to work with the mpich implementation
of the MPI standard.
As users of moma.pvm will know, the organisation of the ocean model
is based on the idea of a master process which controls the model
run, and a series of slave processes, each of which is responsible
for one region of the ocean.
On most of the computer systems supported, mpich requires that
the compiled code is the same for all processes. However in a
unix workstation environment, mpich allows the compiled codes to be
different.
Default implementation
For this implementation, the default is to produce separate
master and slave programs. As discussed in the mpich users
guide the model can then be run using a command of the form
mpirun -p4pg pgfile master
on the machine where program master is to be run. The file pgfile
contains the information on both the computers to use and the
programs to run on each. Thus if master is to be run on the
computer with the name 'sgi1' and the one slave program is to
be run on each of the machines 'sgi2' to 'sgi6' and sun1, with
two copies of slave running on sun1, then the file will have the form
sgi1 0 /...path.../master
sgi2 1 /...path.../slave.sgi
sgi3 1 /...path.../slave.sgi
sgi4 1 /...path.../slave.sgi
sgi5 1 /...path.../slave.sgi
sgi6 1 /...path.../slave.sgi
sun1 2 /...path.../slave.sun
where "...path...' represents the unix path name leading to
the directory where the codes are stored.
Masterslave option
This produces a single program containing both master and
slave code. The first process, actually processor zero,
acts as the master and the remaining processes as slaves.
The option needs the subroutines to be compiled with the
flag -DMasterslave.
The program masterslave can then be run with
mpirun -np 3 masterslave
Note: this means spawn 3 copies of masterslave, one acting
as master and two as slaves.
Details on the changes
The conversion from moma.pvm to moma.mpi was fairly
straightforward. The changes include:
1) moma.mpi uses buffered messages. The message buffer 'bufmpi'
has been added to hold the outgoing messages.
2) To overcome some timing problems when reading the initial met
fields, the MSG_CONT entry in s_recv has been changed so that
the routine always returns after processing this message.
3) Timing is now done using the MPI_WTIME() routine.
4) The itida array is no longer used. The master is now always
processor 0. Slaves are numbered 1 to nproc.
4) Some bugs in the hdf routine which earlier compilers tolerated
have been corrected.
As this is a new version of the program, please let me know of any
further bugs or problems found when running the code on other systems.
David Webb
Southampton Oceanography Centre
13th october 1997
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Introduction
============
Moma.pvm is a PVM message passing multi-processor implementation of the
'moma' ocean model code. It can run on a network of Unix workstations. It
will also run on a Cray-T3D.
Moma itself is a version of the Bryan-Cox-Semtner code which was developed
for use with array processor computers. The basic finite difference scheme
used in the model is described in Bryan (1969), Semtner (1974) and Cox
(1984).
The main difference of the present code from the standard model is in the
solution of the barotropic equations where for simplicity the moma and
moma.pvm codes use an explicit (free surface) scheme.
In other respects the present code closely follows the format of the GFDL
Modular Ocean Model (Pacanowski et al, 1990), and makes use of many of the
subroutines and include files of that model.
More detailed information about the present code is given in Webb (1995).
Information on moma is given in Webb (1993).
The master and slave programs have been developed by members of the OCCAM
core team including:
David J. Webb [email protected]
Beverly A. de Cuevas [email protected]
Andrew C. Coward [email protected]
Catherine S. Gwilliam [email protected]
Advice on CRAY T3D specifics has also been received from:
Mike O'Neill and Bob Carruthers from Cray
The code may be freely adapted and used without charge for non-commercial
purposes. Publications that report on work that has made use of the code
should reference one or more of the relevant publications of the OCCAM core
team.
Copyright 1993, 1994, 1995, 1996, 1997:
D.J. Webb, B.A. de Cuevas, A.C. Coward,
A.C. Coward and C.S. Gwilliam,
Southampton Oceanography Centre,
Empress Dock, Southampton SO14 3ZH, U.K..
Differences from the GFDL MOM code
==================================
The main differences from the modular ocean model code are:
1. Collection of all 'array processor' loops (i.e. loops over the
horizontal indices ic and jc) in subroutine step.
2. Revised common block structures. All variables are stored in core.
3. The arrays u and v now contain the full velocity, not just the
baroclinic part of the velocity.
4. Removal of all 'slab' optimisation code in the program and the
introduction of code designed to optimise inner loop calculations
using the vertical index k.
5. To simplify the development and testing of the array processor code
the following features of the moma code were removed:
(a) Diagnostic calculations in routines clinic and tracer.
(b) Mom pre-processor code options.
It should now be straightforward (but possibly time consuming) to add
the features back into the present code if they are required.
6. The stream function code has been removed and replaced by a free-
surface scheme. This is based on Killworth, Stainforth, Webb and
Patterson (1989) but the Euler-backward timestepping scheme they used
has been replaced by a leapfrog and averaging scheme.
The averaging scheme, originally suggested by Semtner, overcomes an
aliasing instability that can develop with near the equator. The
instability leads to growth of those barotropic waves which travel
their own wavelength in a baroclinic timestep.
The other main changes from Killworth et al, introduced for improved
efficiency, are that the mean horizontal velocity is now used instead
of horizontal transport and the viscous terms are calculated in
routine clinic.
The Killworth et al Euler backward scheme, without averaging, is
available as the compiler pre-processor option 'free_eb'.
7. A compiler pre-processor option, 'de_checkbd', has been added. This
uses a method suggested by Deleersnijder to reduce the
checkerboarding found with the free surface model.
8. In the baroclinic momentum equation, a revised vertical and
horizontal advection scheme option is included. The old scheme can
still be used by specifying the compiler pre-processor flag 'oldadv'
9. An option to precalculate the baroclinic part of the pressure field
is included using compiler pre-processor flag 'presetp'.
10. The model can now produce both snapshot and archive files. The
snapshot files are designed to contain a few key model fields. These
can be used to keep track of the model behaviour.
The archive files contain all the model data from one timestep. They
are used to restart the model and for model analysis.
Note that because only one timestep is saved, restarts of the model
can only be made when this is an integer multiple of 'nmix'. This
ensures that the next baroclinic timestep is an Euler-backward one.
(This only requires data from the current timestep, unlike the
leapfrog scheme which requires data from two timesteps).
11. An option of using NCSA HDF archive and snapshot files has been
added. This needs the model to be compiled with the pre-processor
option 'ncsa_hdf'.
The standard code assumes that both real and integer variables are
32 bits long. Pre-processor option 'REAL8' should be used if the
real variables are 64 bits long.
12. For sites where HDF is not available, the program can also read and
write unformatted Fortran archive files which are compatible with
those of version 1.15 and above of moma. These files have file type
'ufm' (see below).
13. The flags and variables defining the restart, archive and snapshot
files have been changed from those found in the MOM code. The values
used for each run can be specified in the 'control' namelist of the
input file 'ocean.in'. The variables involved for archiving and
snapshots are now:
init = .true. when starting from rest,
.false. when restarting from an archived data set.
fnrest = name of the restart data set.
ftrest = file type of restart data set. Can be:
'ufm': unformatted data file compatible with the
restart files from moma_v1.15 and later.
'hdf': hdf file
snapd = number of days between snapshots. (0.0 = none).
ftsnap = file type of snapshot data set. Can be:
'asc': ascout format files
'hdf': hdf file
archd = number of days between archives. (0.0 = none).
ftarch = file type of archive data set. Options same as ftrest.
restrt = .true. if an archive data set is wanted at the end of
the run.
14. The full list of pre-processor options is:
hcomments - to turn listing of '.h' files comments on.
presetp - precalculate pressure field each baroclinic
timestep.
oldav - use the old numerical scheme for the vertical
advection of momentum.
free_eb - use the euler backward timestepping scheme without
averaging for the free surface equations.
NOTE: The default is now to use a leapfrog scheme
with time averaging.
de_checkbd - use the de-checkerboarding scheme for the free
surface height field.
ncsa_hdf - include the NCSA hdf subroutines.
REAL_8 - Used to inform PVM and HDF if the real variables
are 64 bits long (i.e. -r8 option has been used
with the f77 compiler).
pvm_buffer - use a buffer to reduce the number of calls to
the PVM pack and unpack routines.
cyclic_master - used when the model is cyclic in the east-west
direction.
cray_t3d - to turn cray-t3d specific code on
no_namelist - read input from file 'ocean.in2' using
formatted read statements. Used for compilers
that do not support namelist.
SYNC_2 - to synchronise slaves while in the barotropic
timestep loop.
SYNC_3 - to synchronise slaves while in the baroclinic
timestep loop.
The organisation of moma.pvm
============================
In moma.pvm, the ocean is split up into a number of sub-volumes, each the
responsibility of a separate process. Each process carries out all
calculations required for its volume of ocean. It also exchanges data on
bundary points with processes responsible for neighbouring areas of ocean.
The sub-volumes are each made up of vertical columns of grid boxes. This
is computationally efficient. It also means that the sub-volumns can be
specified by a two-dimensional horizontal array.
Moma.pvm uses one other process, 'master', which controls the model run,
reads the input files and handles archiving. The master also spawns the
'slave' processes which actually carry out the model calculations
Usually each process will be run on a separate processor, with the master
running on a front-end processor. However for testing purposes, standard
unix will also allow them to be run on the same processor.
The master program needs the standard MOM input file 'ocean.in'. It also
needs two extra files, 'ocean.kmt' and 'ocean.kpn'. 'ocean.kmt' contains
the array defining the ocean depth. (Strictly speaking it contains the
number of model levels at each horizontal grid point). 'ocean.kpn'
contains a processor map specifying which processor is responsible for each
horizontal grid point in the ocean. The region that each slave is
responsible for may be of arbitrary shape. If the model is cyclic in the
east-west direction then the master program must be compiled with the pre-
processor flag 'cyclic_master' set.
Two extra programs are provided for producing versions of these files
suitable for simple tests. 'make_kmt' generates a depth array for a
rectangular ocean, a channel or the simple MOM world ocean. 'make_kpn'
then takes this file and partitions it among processors in a simple way.
Both files are ASCII files and start with a single line containing a string
describing the type of file, 'kmt' or 'kpn'. This is followed by two
integers specifying the dimensions of the array. 'moma_kpn' also has an
integer giving the number of processors required. The rest of the file
consists of an integer array, ordered from west to east and south to north,
with each new east-west row of the array starting on a new line.
Starting
The program is started by running program master. This reads the basic
model data from file 'ocean.in' and then looks for the dataset defining the
ocean depths. If starting from rest this is read from file 'ocean.kmt',
otherwise it is read from the restart data set.
The master then reads in the processor map from file 'ocean.kpn'. If
starting from an archive data set, the number of processors and their areas
of responsibility can differ from those in use before the archive was made.
At this stage the master completes its initialisation and calls a PVM
routine to spawn the required number of slave processes. It sends to each
slave messages containing copies of the basic model data and information
about the slave's area of responsibility.
If restarting from a archive dataset, the master then reads the remainder
of the archive file and sends the data to the slaves. This completes the
initialisation.
Running
While running, each slave cycles through the timestepping loops in the
normal way. At the beginning of every timestep the slave sends messages to
its neighbours giving them permission to send boundary data for the new
timestep.
At the end of the timestep it waits until it has received similar messages
from its neighbours and then sends them the boundary data they require. At
the end of each barotropic timestep the data sent consists of the free
surface height and barotropic velocity. For baroclinic timesteps it
consists of the temperature, salinity and velocities at each level above
the ocean bottom.
The boundary region of each slave is conventionally called the slave's
halo. It consists of two parts, an inner and outer halo. The inner halo
contains grid points in the slave's area of responsibility for which data
needs to be sent to neighbouring processors. These grid points are also
the ones for which the numerical finite difference scheme requires data
from neighbouring processors. The outer halo consists of the corresponding
grid points belonging to neighbouring processors which the slave needs to
solve these equations.
Before it can carry out calculations on points in its inner halo, the slave
needs to receive data for the outer halo points belonging to its
neighbours. Because the messages may take some time to arrive, the slave
starts each timestep carrying out calculations in its core region. The
slave then checks that the outer halo data has arrived. If necessary it
waits before continuing the calculation for the grid points in its inner
halo.
Summary and archive data.
Both the master and slaves keep track of timesteps for which summary data,
snapshots or archive data are required. At the end of such a timestep the
slaves put the required data into a temporary buffer.
When it is ready, the master sends the slaves a request for the data. If
it is already waiting in the buffer, the slaves reply at once. Otherwise
they reply as soon as the data becomes available. For summary data this
requires a single request and transfer of data. For the large snapshot and
archive files, the master will normally not have enough memory to contain
the full dataset and so it sends out a number of requests, each for part of
the full dataset.
Finally when it has received all of the snapshot or archive data the master
sends out a clear flag. The slaves can then reuse the buffer for data from
a new timestep.
Compiling the model
===================
The main directory contains all the Fortran files (*.F) and include files
(*.h) needed to construct the master and slave programs and the make_kmt
and make_kpn utilities. It also contains copies of 'ocean.in' and
'ocean.in2' and in 'master.out', the output from a model run using this
copy of 'ocean.in'.
The main program and subroutines for the master program are written in
Fortran and are included in files:
master.F, checks.F, docmnt.F, hdf.F, init_kmt.F, init_kpn.F,
m_arch.F, m_grids.F, m_msg.F, m_snap.F, setkmt.F, topog.F,
and
sysmsg.F, tmngr.F, tmnset.F, indp.F.
The file hdf.F is only needed if the pre-compiler option 'ncsa_hdf' is
being used.
The main program and subroutines for the slave program are included in
files:
slave.F, addv.F, bcest.F, bound.F, clinic.F, diag.F,
frees.F, matrix.F, ocn1st.F, pdiag.F, s_arch.F, s_grids.F,
s_msg.F, s_snap.F, setvbc.F, setp.F, state.F, step.F,
theta0.F, tracer.F,
and
sysmsg.F, tmngr.F, tmnset.F, indp.F.
These files need to be passed through the c-compiler pre-processor cpp
before being compiled. On most Unix systems this is done automatically by
the Fortran compiler.
After compilation the programs also need subroutines from the PVM and HDF
libraries. In release 3.2.6 of PVM, the PVM libraries are 'libfpvm3' and
'libpvm3'. In release 3.3r3 of HDF, the libraries needed with the
'ncsa_hdf' pre-compiler option are 'libnetcdf.a' and 'libdf.a'.
Makefiles are included for compiling and linking master, slave and the two
utility programs. 'makefile.sun' has been developed for a Sun Unix system
but should work, with at most only minor modifications, on other Unix
systems.
The PVM and HDF software
========================
Moma.pvm has been developed using version 3.2.6 of the PVM library running
on Unix workstations. It should work equally well with later releases.
PVM is free software available from Al Giest and colleagues at the Oak
Ridge National Laboratory in the USA.
If you do not have PVM already in stalled on your system, copies can be
obtained on the internet by using the utility xarchie and looking for
'PVM3' or 'PVM3.3'. Once this has been found, search for and download a
file with a name like 'pvm3.3.6.tar.Z'. Use the Unix commands 'uncompress'
and 'tar' to create the PVM directories and source files. Then follow the
instructions given with the sources files to set-up PVM and its libraries
on your local system. You will also have to define the PVM root directory
(environment variable PVM_ROOT) and the machine architecture (PVM_ARCH).
For c-shell users this can be done by adding users to the '.cshrc' script
which is run when opening each shell.
The version of HDF used in developing the present code is version 3.3
release 3. This was developed by the US National Centre for
Supercomputing. Again the software is free and if it is not available on
your system it can be obtained on internet using xarchie and searching for
HDF3.3r3 or HDF3.3r4. Again follow the instructions to create the HDF
libraries.
References:
===========
Bryan, K., 1969: A numerical method for the circulation of the World
Ocean. Journal of Computational Physics, 4, 347- .
Semtner, A.J., 1974: A general circulation model for the World Ocean.
UCLA Department of Meteorology Technical Report No. 8, 99pp.
Cox, M.D., 1984: A primitive equation, 3-dimensional model of the ocean.
GFDL Ocean Technical Report No.1, Geophysical Fluid Dynamics
Laboratory/NOAA, Princeton University, Princeton N.J., U.S.A..
Deleershijder, 1995. (Unpublished manuscript).
Killworth, P.D., Stainforth, D., Webb, D.J. and Paterson, P.M., 1989: A
free surface Bryan-Cox-Semtner model. Report No. 270, Institute of
Oceanographic Sciences, Wormley, U.K..
Pacanowski, R.C., Dixon, K., Rosati, A., 1990: The GFDL Modular Ocean
Model 1.0. Geophysical Fluid Dynamics Laboratory/NOAA, Princeton
University, Princeton, N.J., U.S.A.. (Unpublished manuscript).
Webb, D.J., 1993: An ocean model code for array processor computers.
Internal Document No.324, Institute of Oceanographic Sciences, Wormley,
U.K..
Webb, D.J., 1995: In preparation.
Appendix I. Further details
============================
Code organisation
As was stated at the beginning, moma.pvm is a development of the standard
Bryan-Cox-Semtner ocean model. When developing a multi-processor message
passing version we realised that if we were not careful problems related to
domain decomposition and message passing could arise in every routine in
the model.
The decision was therefor made to only allow partition of the model in the
horizontal direction and to put all the loops over horizontal indices into
subroutine step. (This is the top level routine of standard versions of
the Bryan-Cox-Semtner model). It was also decided that all the top level
calls to the message passing routines and all checks that data had arrived
should be also made from subroutine step or the main program.
As a result, the key routines, clinic, tracer, etc., which solve the model
equations, are not affected either by domain decomposition or message
passing. Compared with previous implementations, they lose their loops
over the horizontal index, but they still retain their loops over the
vertical index. If the model physics is changed, then only these routines
should be affected.
Such changes should only need to take into account the fact that this is a
message passing implementation of the model if they introduce new fields.
In that case the message passing software will have to be modified to
transfer these files along with the standard temperature, salinity,
velocity and free surface height fields.
For simplicity the top level routines in master and slave do not call the
PVM message passing routines directly. Instead, in slave, the top level
routines use calls to entry points in files s_msg.F and sysmsg.F, usually
s_send and s_recv. For master the entry points are in m_msg and sysmsg.F.
In this case the main entry points are m_send and m_recv. If the model
needs to be changed to use another message passing system then only the
routines in these files need be modified.
In a similar way all the calls to HDF routines are made from subroutines in
the file hdf.F. If an alternative dataset format is used, for example net-
cdf, then it should be possible to do this by replacing the calls in hdf.F
by equivalent calls to the alternative software library.
Debugging
Debugging of message passing code can be particularly frustrating. To help
a number of special facilities are included in the moma.pvm code.
1. The master prints, at regular intervals, a table giving the position
reached by each slave when it last sent a position message. Slaves send
such messages each time they pass the top of the barotropic and baroclinic
timestepping loops and at other key points in the program.
The slave generates the messages by calling routine 'sendposn'. Additional
calls can be added if required. If the interval in seconds between the
reports printed by the master needs to be changed, this can be done by
changing the value of variable 'lapse' in routine 'prininfo'.
2. Variable 'idebug', read from the input file 'ocean.in', determines the
amount of information printed by the message passing routines. If idebug
is 3 or greater, then the master prints information on all messages it
sends and receives. If it is four or greater, then the slaves print
information on all messages sent to and received from the master. If it is
five or greater the slaves also print information on messages sent to and
received from other slaves.
3. PVM normally sends the standard output stream from all the slave
programs to a single file in directory /tmp. Unfortunately there is often
a limit on the length of this file so key information can be lost. To
overcome this problem, in moma.pvm, each slave tries to route its output to
a separate file.
It does this by opening an output file with the name 'slaveXXX', where XXX
is the number of the slave. If successful this is given Fortran unit
number 23 and variable 'ioslave' is set to this number. All output from
the slave is then written to Fortran stream ioslave.
If it is unsuccessful in creating the file, then 'ioslave' is set to
'stdout'. Output from the slave program will then be routed to the PVM
output file in /tmp.
4. The slaves can also send information directly to the master for
printing. To do this, use a Fortran write statement to place up to 80
ASCII characters in variable 'outstr' (see file 'mesdta.h'). Then use
'call s_send(MSG_OUTSTR)' to send the message to the master for printing.
Synchronous operation
The code is designed so that, as far as possible, the slaves operate
asynchronously. One advantage of this is that the slaves do not all try to
send messages at the same time and so overload the message passing system.
However when debugging new code, especially if new messages are involved,
there are advantages if the slaves act synchronously and wait until all
have completed the previous timestep before starting the next.
Two compiler pre-processor options are included which allow this. SYNC_2
synchronises the barotropic timestepping loop and SYNC_3 synchronises the
baroclinic loop. If additional synchronisation points are required these
can be achieved by adding slave calls to routine 'schkpnt(n)' and master
calls to 'mchkpnt(n)', where n is an integer. The call to the master
subroutine is required because this controls the checkpointing.
The master and slave calls should be in the same position relative to the
main baroclinic timestepping loop, and to other calls to schkpnt and
mchkpnt, or the program will hang. The master will continue printing
summary information while waiting at a checkpoint.
Cray-T3D
The 'cray-t3d' compiler pre-processor option is included to make the
changes needed before running the model on the Cray-T3D. The main change
arises because the T3D Fortran read subroutine does not support namelists.
A new input file is therefor included named 'ocean.in2'. This is read
using the Fortran free format read option.
=======================================================================
D.J.Webb, March 17, 1995.
=======================================================================
Version information
v1.6 Minor corrections suggested by Hugo Embrechets (Hugo.Embrechts@
cc.kuleuven.ac.be), affect i/o and compilation without presetp
but otherwise not affecting results.
v1.7 1. Additional minor corrections (Ivan Walton/David Webb).
2. Precompiler option 'inline' added. This replaces the call to
subroutine frees in subroutine step by inline code. Change
increases the efficiency of the code.
3. Precompiler option 'DOUBLE. On a workstation the code will
usually compile using 4 byte real variables or 8 bytes if
the -r8 option is used when calling the compiler. In the
latter case the PVM and HDF flags have to be set correctly.
This is done by setting the DOUBLE precompiler option. On
the Cray T3D, the cray-t3d precompiler option has the same
effect. (Changed to REAL_8 at v1.10 for compatability
with moma and PVM codes.)
4. Printing of machine status arrays. File 'ocean.in' contains
two new variables 'statms' and 'statis'. If 'statms' is
non-zero and positive then the master program prints out
the status information every statms seconds. If 'statis'
is non-zero and positive then the status arrays are printed
whenever the master has to wait 'statis' seconds without
receiving any messages from the slaves.
5. HDF constants file. The name of the hdf constants file has
been changed from 'constant.i' to 'hdf.inc'. (Both files
are produced when installing HDF but hdf.inc appears to be
the correct one to use for fortran programs).
6. Precompiler option 'pvm_buffer'. If set, buffering is now
used to send all large arrays between master and slave. Used
for initialising, sending snapshots and archiving.
7. Program timing. Precompiler option 'cray-t3d' now uses cray
routine rtc(). Elsewhere routines time(), etime() and dtime()
are used. These are vendor specific. They work with SUN and
SGI. IBM requires a trailing '_' to be added. (see above).
8. IBM RS6000 system. Calls to system routines dtime, etime,
time, sleep, flush and lnblnk have to be replaced with calls
to dtime_, etime_, time_, sleep_, flush_ and lnblnk_. The
statements defining time to be a integer must be changed
to refer to time_.
On the RS6000 system the specification of 'fnslave' may have
to be changed so that it points to an absolute directory.
On the SUNs and Siulicon graphics the default directory for
the slave files is the home directory. On the cray-t3d it
is the current directry.
An aix makefile developed by Hugo Embrechets is included.
v1.8 Additional corrections (Cai Wenju, Stephen Alderson,
Andrew Coward). These correct bugs affecting the saving
of timestep information, closing of unformatted restart
files, the timestep in ascii snapshot files, checking
kmt on restart and saving hdf unit numbers.
v1.9 The surface wind stress and flux data is now sent from the
master to the slaves each month (the period can be easily
changed). The slaves then use linear interpolation in time
and space (setvbc.F). The master routines for initialising
and 'reading' in the monthly data are in m_flx.F. For
compatability with earlier versions, the test problem uses
wind stress and flux data which is constant in time.
The error that arose when the slaves were cyclic is now
corrected.
v1.10 Corrects end of year, restart and tsi message bugs. Minor
change to make_kmt input. Option DOUBLE changed to REAL_8
to prevent name conflict. Bug in reading HDF restart data
sets corrected.