Skip to content

cybergis/cybergis-compute-core

Repository files navigation

CyberGIS Compute Core

Scalable middleware framework for enabling high-performance and data-intensive geospatial research and education on CyberGISX.

Supported Git Projects

Name URL
hello_world https://github.com/cybergis/cybergis-compute-hello-world.git
summa3 https://github.com/cybergis/cybergis-compute-v2-summa.git
wrfhydro-5.x https://github.com/cybergis/cybergis-compute-v2-wrfhydro.git
uncertainty_in_spatial_accessibility https://github.com/JinwooParkGeographer/Uncertainty-in-Spatial-Accessibility.git
mpi_hello_world https://github.com/cybergis/cybergis-compute-mpi-helloworld.git
data_fusion https://github.com/CarnivalBug/data_fusion.git

If you'd like to add your Git project to our supported list, please contact [email protected]

Supported HPC & Computing Resources

Name Code Description
Keeling Computing Cluster keeling_community University of Illinois HPC for enabling geospatial research and education at the CyberGIS Center
Bridges-2 bridges_community Pittsburgh Supercomputing Center HPC
XSEDE Expanse expanse_community San Diego Supercomputer Center

Kernels supported through CVMFS

  • CVMFS option can be activated by using plugging one of these kernels into the container field in your manifest.json and setting connector field to SingCVMFSConnector. Here's an example of a manifest.json using CVMFS

    • cybergisx/python3-0.8.0
    • cybergisx/python3-0.9.0
    • cybergisx/geoai-0.8.0
    • cybergisx/hydro-0.8.0
    • cybergisx/rpy-1.0.0
    • cybergisx/sos-0.8.0
    • cybergisx/ssh-0.8.0
    • cjw/landlab-2021-09
    • cjw/pysumma-2021-06
    • cjw/pysumma-2021-12
    • cjw/python3-2021-09
    • cjw/python3-2021-12
    • cjw/python3-2022-06
    • cjw/python3-2021-06
    • cjw/rhessys-2022-03
    • cjw/taudem-2021-09
    • cjw/taudem-2022-12
    • cjw/wrfhydro-2021-06
    • cjw/wrfhydro-2022-03

Server Setup

  1. Requirements

    • Docker & Docker Compose
  2. Initialize

    git clone https://github.com/cybergis/cybergis-compute-core.git
    cd cybergis-compute-core
    # run init script
    ./script/init.sh
  3. Configure the following options

    • config.json
      • local_key
        • private_key_path
        • passphrase (if required)
    • configs/hpc.json
      • external_key
        • private_key_path
        • passphrase (if required)
  4. Run server

    # for development
    # - run in foreground with log output
    ./script/develop-start.sh
    # - run in background, add -b background failing
    ./script/develop-start.sh -b
    # - some HPC requires university network
    # - to connect to a University VPN, add the AnyConnect options
    ./script/develop-start.sh -l vpn.cites.illinois.edu -u NetID -p "password" -g 5_SplitTunnel_NoPrivate
    
    # for production server only
    ./script/production-start.sh
  5. Stop all running containers

    ./script/stop.sh

Configurations

  1. HPC configurations are located at configs/hpc.json

    • community account example
    {
        "keeling_community": {
            "ip": "keeling.earth.illinois.edu",
            "port": 22,
            "is_community_account": true,
            "community_login": {
                "user": "cigi-gisolve",
                "use_local_key": false,
                "external_key": {
                    "private_key_path": "cigi-gisolve.key",
                    "passphrase": null
                }
              },
              "root_path": "/data/keeling/a/cigi-gisolve/scratch/dev"
        }
    }
    • private account example
    {
        "hadoop": {
            "ip": "hadoop01.cigi.illinois.edu",
            "port": 50022,
            "is_community_account": false,
            "connector": "SlurmConnector"
        }
    }
  2. Maintainer configurations are located at configs/maintainer.json

    • example maintainer with user upload file
    {
        "SUMMA": {
            "hpc": ["keeling_community"],
            "job_pool_capacity": 5,
            "executable_folder": {
                "from_user": true,
                "file_config": {
                    "ignore": [],
                    "must_have": [
                        "summa_options.json",
                        "installTestCases_local.sh",
                        "data",
                        "output",
                        "settings"
                    ],
                    "ignore_everything_except_must_have": true
                }
            },
            "maintainer": "SUMMAMaintainer"
        }
    }
    • example maintainer that disables user upload
    {
        "hello_world_singularity": {
            "hpc": ["keeling_community"],
            "job_pool_capacity": 5,
            "executable_folder": {
                "from_user": false
            },
            "maintainer": "HelloWorldSingularityMaintainer"
        }
    }

Related Documentations