Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated ppml #356

Draft
wants to merge 92 commits into
base: main
Choose a base branch
from
Draft

Updated ppml #356

wants to merge 92 commits into from

Conversation

jobordner
Copy link
Contributor

This PR

  1. adds an adiabatic version of the PPML MHD solver,
  2. adds Ornstein-Uhlenbeck driven turbulence, and
  3. adds MHD turbulence.

New methods and initializers:

  EnzoMethodPpmlIG
  EnzoMethodTurbulenceOU
  EnzoInitialTurbulenceMhdIT
  EnzoMethodTurbulenceMhdIT

New parameters

    Method : ppml_dt_weight

    Method : ppml_ig : b0

    Initial : turbulence_mhd_it : density
    Initial : turbulence_mhd_it : density
    Initial : turbulence_mhd_it : bfieldx

    Method : turbulence_mhd_it : edot
    Method : turbulence_mhd_it : mach_number

    Method : turbulence_ou : apply_cooling
    Method : turbulence_ou : apply_forcing
    Method : turbulence_ou : apply_injection_rate
    Method : turbulence_ou : cooling_term
    Method : turbulence_ou : hc_alpha
    Method : turbulence_ou : hc_sigma
    Method : turbulence_ou : injection_rate
    Method : turbulence_ou : kfi
    Method : turbulence_ou : kfa
    Method : turbulence_ou : mach_number
    Method : turbulence_ou : olap
    Method : turbulence_ou : read_sol
    Method : turbulence_ou : sol_weight
    Method : turbulence_ou : totemp
    Method : turbulence_ou : update_solution

jobordner and others added 30 commits August 21, 2021 10:16
Method
  - Added enzo_EnzoMethodMHDTurbulenceIT.[ch]pp

Initial
  - Added enzo_EnzoInitialMHDTurbulenceIT.[ch]cpp

Fortran
  - Added turbforcing2D.F, turbforcingOU.F, and turbforcingOU.h
    (not called yet)

Input
  - Added ppml-turbo-test-it-128.in test
External

  - added odepack (public domain) source by Alan Hindmarsh: see
    https://computing.llnl.gov/projects/odepack

Build

  - removed -pedantic compiler flag---breaks compiling odepack
  - added crand.c interface required by turbforcingOU.F to build
  - added odepack files opkd*f to build
  - added turbforcing2D.F and turbforcingOU.F to build

Cello

  - added cello::field (block,id_field) (removes need for explicit (cello_float *) cast)

Enzo

  - changed EnzoInitialTurbulenceOU to EnzoMethodTurbulenceOU since called every time step
  - renamed EnzoMethodMHDTurbulenceIT as EnzoMethodTurbulenceMhdIT
  - added EnzoMethodTurbulenceOU class
  - factored out monitor_output_() in EnzoMethodTurbulenceMhdIT
  - added calls to turbforcing2D.F initialization and update from EnzoMethodTurbulenceOU
    (incomplete)

Parameters

  - Added method_turbulence_* parameters or EnzoMethodTurbulenceOU
  - moved method_grackle_chemistry initialization from constructor body to initializer list
  - bug fix: use "mach" default for Method:turbulence:mach_number, which is
    read in multiple times depending on turbulence model
  - renamed TurbulenceMhdIT method from "mhd_turbulence_it" to "turbulence_mhd_it"
Enzo

  - Added r_method_turbulence_ou_* entry methods for reductions
  - Removed unused arguments to FORTRAN(cello_init_turbulence_ou)
    and FORTRAN(turbForceOU)
  - Fixed dt argument to FORTRAN(turbForceOU)
  - Extracted MPI_AllReduce calls from turbForceOU
  - Restructured turbForceOU for reductions, split into turbForceShift()
    and turbForceUpdate()
  - Added gv and av reductions as r_gv128 and r_av128
  - Added debugging printf's and dummy values to  verify reductions
  - Auto-indented turbforcingOU.F

Input

  - Added temporary test_turbulence_ou.in
Enzo

  - Replaced w(1:5,i,j,k) array in turbforcingOU.F with cello_[density|momentum_[xyz]|energy]
  - Scale/unscale Enzo-E fields by density for turboforcingOU subroutines since
    expecting conservative form
  - Similarly replaced res(1:5,i,j,k) array with resid_[density|momentum_[xyz]|energy]
  - Copy in/out work fields "work_[123]" in to work array for turbforcingOU.F

Input

  - added acceleration and work fields to input/test_turbulence_ou.in

Cello

  - added cello::field(block,field_name)
Data
  - cleaning: renamed (nx,ny,nz) as (mx,my,mz) in FieldData to
    keep with current naming conventions

EnzoMethodTurbulenceOU

  - Added debugging for field stats
  - Support non-zero ghost zones by only passing in non-ghost arrays
    and setting olap to 0
  - Changed accelerations only to all fields for refresh (overkill)
  - Updated parameters for Fortran interface to turbforcingOU.F
  - Added work_[1-7] fields for fortran work array (work 1-3) and
    turbAcc (work 4-7)
  - Fixed array offset bug
  - Fixed grid() array of cell centers for block offset in domain
  - Added static iupdate_phases_ attribute so only first block on node
    can update phases in Fortran if scheduled to (will need node lock
    for SMP mode)
  - Fixed memory leak delete [] turbAcc

turbforcingOU

  - Changed updatePhases() decision to include cello_update_phases
    to support multiple blocks per static variable
  - Use time <= time_{phase_update} < time+dt test to determine
    updatePhase() and calcPhases() instead of static totTime
  - Avoid accessing field_momentum_z if Ndims == 2 (not verified)
Method [EnzoMethodMHDTurbulenceIG]

  - Renamed EnzoMethodMHDTurbulenceIG as MethodTurbulenceMhdIG
  - Moved global reduction indices to file-local in
    MethodTurbulenceMhdIG
  - Added gamma and mach_number to MethodTurbulenceMhdIG constructor
  - Updated MethodTurbulenceMhdIG for "new" refresh
  - Commented out OUpumpInit (kritsuk: "There should be some seeds for
    OU forcing in the distribution I sent to you."
  - Renamed nd[xyz] as m[xyz] to follow current naming convention
  - Fixed "rank >= 1" code-copy bug

Method [EnzoMethodPpmlIg]

   PPML_IG_MAIN.F

  - Added INCLUDE_GRAVITY and USE_PN1_PN2 to comment out
    unfinished/broken code in
  - Added missing comma after pn in "Call Conserv"

   EnzoMethodPpmlIG

  - added gamma parameter
  - updated refresh
  - Removed field_descr parameters

Method [EnzoMethodTurbulenceOU]

  - Replace refresh "add_all_fields()" with specific fields
  - Add attributes to pup
  - Fix reduction array limits
  - Monitor print div-B
  - Fixed scheduling for phase updates

Enzo

  - Removed obsolete ENZO enzo_fluxes struct
  - Removed MAX_NUMBER_OF_BARYON_FIELDS
  - Added EnzoBlock::SolveMHDEquationsIG()
  - Added CHECK_FIELD to SolveMHDEquations.cpp

Added files

  - added PPML_IG_*
  - added enzo_EnzoMethodTurbulenceMhdIG.?pp
  - added enzo_EnzoMethodPpmlIG.?pp
  - added EnzoSolveMHDEquationsIG.cpp

Input

  - added method-ppml-turbou.in to test MethodTurbulenceOU with PPML
PPML_IG

  - add _ig suffix to PPML_IG subroutines/functions to distinguish
    from PPML_IT
  - remove duplicate declarations (eg gamma in quzx_ig, etc.)
  - added INCLUDE_GRAVITY around incomplete/inconsistent code including
    gravity in ppml_ig
  - bug fix: added missing BS to VECTRLZ_ig and HLLDz_ig calls in potokz_ig
  - bug fix: added missing QU[5-7] declarations in VECTLRz_ig,
    VECTEGy_ig
  - bug fix: fixed CA->VA declaration in VECTEGz_ig
  - bug fix: added missing M declaration to FPMR_ig,FPML_ig
  - bug fix: added missing L declaration to AMPLTD_ig

Build
  - added remaining PPML_IG*F files to SConscript

Enzo
  - added debugging code to EnzoMethodPpmlIG(), SolveMHDEquationsIG()
  - removed config parameter to EnzoMethodPpmlIG constructor
  - added "ppml_ig" method to EnzoProblem
  - added copying of velocities between "velocity_?" and "velo?" fields
  - added copying of pressure between "pressure" and "press_r?" fields
Build
 - updated fortran compiler options for debugging uninitialized values

Output
  - updated OutputImage to avoid division by 0.0 when min == max
Enzo-E
  - enable floating point exceptions in enzo-p.cpp

PPML
  - commented-out PPML_IG_calc_dt.F: use PPML_IG_Timestep
  - added missing parameter BS to calls to MATR_A[XY]_ig()
  - fixed index in initialization of QL(6,8) (was QL(7,8))
  - add b0 parameter to SolveMHDEquationsIG, EnzoMethodPpmlIG
Parameters
  - added EnzoConfig::read_method_ppml_()
  - added method:ppml_ig:b0 parameter
  - support "ppml_it" as alias for "ppml"
  - cleaned SolveMHDEquationsIG, removed unused commented-out code
Method
  - fixed PPML to call compute_done() for non-leaf nodes
    (required when other methods require negative levels)
  - renamed turbulence *.f files as *.F for consistency
  - added missing [in] to Ppml and PpmlIG methods and
    SolveMHDEquationsIG for EnzoBlock static attribute
    access

Build
  - added "-std=legacy" for newer gfortran versions

Charm

  - removed check on Charm++ accumulation array size since
    failing for mhd-turbulence with PPML test
Enzo

  - Added double and int arrays to EnzoSimulation for saving Fortran
    phase and random number generator state from
    EnzoMethodTurbulenceOU

  - Added corresponding EnzoSimulation [get|put]_turbou_state() methods

  - Added IoEnzoSimulation to save EnzoSimulation attributes above to
    checkpoint files

  - added setcrandseed() to crand.c to restore saved random generator
    state

  - removed CALL_FORTRAN ifdef from EnzoMethodTurbulenceOU

  - added cello_[get|put]_turbou_state in turbforcingOU.F to
    save/restore state variables, plus cello_turbou_state_size
    to return array sizes
Check/restart

  - Added turbou state data to EnzoSimulation

Io

  - Added IoEnzoSimulation for turbou state data
Utilities

- Added "extract_field" utility to extract a field from files in a
  (e.g.) checkpoint dump into a single file (root-level only)

Output

  - Added "is_leaf" parameter to IoBlock
  - Added "root_size" and "blocking" to IoHierarchy
Input
  - Fixed include path in test_turbulence_ou.in

Refresh

  - Reordering constructor initializations to address compiler warnings

Enzo-E

  - Turned off EnzoMethodPpmlIg debugging define
  - Add cello::define_field() to EnzoMethodTurbulenceOU
  - Renamed work_[456] as acceleration_[xyz]
  - Turned off SolveMHDEquations debugging define
extract_field

  - check is_leaf to avoid copying non-leaf data to output

Mesh

  - Update Block is_leaf initialization to be true iff level >=0 not
    true always, since otherwise is_leaf=true for level<0 blocks

PPML_IG

  - Removed suspect scaling of pressure by (gamma-1.0) in
    PPML_IG_Conservative
  - Update array limits 7 -> 8 (INCLUDE_GRAVITY only)
  - Initialize s?(i,8)=0
  - Fix POTK?_ig parameters (INCLUDE_GRAVITY only)
  - Removed unused PPML_IG_calc and PPML_ODD6
@jobordner jobordner added the enhancement New feature or request label Dec 6, 2023
@mabruzzo
Copy link
Contributor

mabruzzo commented Feb 19, 2024

Ideally, I would like to get PRs #293 and #338 merged in before this. It does some reorganization of the ppm/ppml methods (it makes them more consistent with other methods)

@jobordner
Copy link
Contributor Author

Ideally, I would like to get PRs #293 and #338 merged in before this. It does some reorganization of the ppm/ppml methods (it makes them more consistent with other methods)

That's fine; note this is a "work-in-progress" PR.

@jwise77 jwise77 requested review from mabruzzo and jwise77 February 23, 2024 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants