Lot3 work

[sambles]

**IMPORTANT: Please attach or create an issue after submitting a Pull Request.

Lot3 work

• Added SQL endpoints for exposure data queries (Currently disabled)
• Replaced Remote file storage manager with OasisDataManager package
• Added the option to read model_data from a remote object store (S3, Azure BlobStorage)

There are two sets of 'remote storage' configuration options.

• General Storage, for files like results and exposure in an ObjectStore - Env var prefix = OASIS_{option}
• Model Data Storage, for reading 'model_data' during an analysis execution - Env var prefix = OASIS_WORKER_MODEL_DATA_{option}

For Valid options see src/common/filestore/filestore.py

OasisPlatform/src/common/filestore/filestore.py

Lines 21 to 59 in cc6907a

	root_dir=settings.get(section, 'ROOT_DIR', fallback=""),
	bucket_name=settings.get(section, 'AWS_BUCKET_NAME'),
	access_key=settings.get(section, 'AWS_ACCESS_KEY_ID', fallback=None),
	secret_key=settings.get(section, 'AWS_SECRET_ACCESS_KEY', fallback=None),
	endpoint_url=settings.get(section, 'AWS_S3_ENDPOINT_URL', fallback=None),
	file_overwrite=settings.getboolean(section, 'AWS_S3_FILE_OVERWRITE', fallback=True),
	object_parameters=settings.get(section, 'AWS_S3_OBJECT_PARAMETERS', fallback={}),
	auto_create_bucket=settings.getboolean(section, 'AWS_AUTO_CREATE_BUCKET', fallback=False),
	default_acl=settings.get(section, 'AWS_DEFAULT_ACL', fallback=None),
	bucket_acl=settings.get(
	section,
	'AWS_BUCKET_ACL',
	fallback=settings.get(section, 'AWS_DEFAULT_ACL', fallback=None),
	),
	querystring_auth=settings.getboolean(section, 'AWS_QUERYSTRING_AUTH', fallback=False),
	querystring_expire=settings.get(section, 'AWS_QUERYSTRING_EXPIRE', fallback=604800),
	reduced_redundancy=settings.getboolean(section, 'AWS_REDUCED_REDUNDANCY', fallback=False),
	location=settings.get(section, 'AWS_LOCATION', fallback=''),
	encryption=settings.getboolean(section, 'AWS_S3_ENCRYPTION', fallback=False),
	security_token=settings.get(section, 'AWS_SECURITY_TOKEN', fallback=None),
	secure_urls=settings.getboolean(section, 'AWS_S3_SECURE_URLS', fallback=True),
	file_name_charset=settings.get(section, 'AWS_S3_FILE_NAME_CHARSET', fallback='utf-8'),
	gzip=settings.getboolean(section, 'AWS_IS_GZIPPED', fallback=False),
	preload_metadata=settings.getboolean(section, 'AWS_PRELOAD_METADATA', fallback=False),
	url_protocol=settings.get(section, 'AWS_S3_URL_PROTOCOL', fallback='http:'),
	region_name=settings.get(section, 'AWS_S3_REGION_NAME', fallback=None),
	use_ssl=settings.getboolean(section, 'AWS_S3_USE_SSL', fallback=True),
	verify=settings.get(section, 'AWS_S3_VERIFY', fallback=None),
	max_memory_size=settings.get(section, 'AWS_S3_MAX_MEMORY_SIZE', fallback=0),
	shared_bucket=settings.getboolean(section, 'AWS_SHARED_BUCKET', fallback=False),
	aws_log_level=settings.get(section, 'AWS_LOG_LEVEL', fallback=''),
	gzip_content_types=settings.get(section, 'GZIP_CONTENT_TYPES', fallback=(
	'text/css',
	'text/javascript',
	'application/javascript',
	'application/x-javascript',
	'image/svg+xml',
	)),
	cache_dir=settings.get(section, 'CACHE_DIR', fallback='/tmp/data-cache'),

OasisPlatform/src/common/filestore/filestore.py

Lines 63 to 83 in cc6907a

	root_dir=settings.get(section, 'ROOT_DIR', fallback=""),
	account_name=settings.get(section, 'AZURE_ACCOUNT_NAME'),
	account_key=settings.get(section, 'AZURE_ACCOUNT_KEY'),
	azure_container=settings.get(section, 'AZURE_CONTAINER'),
	location=settings.get(section, 'AZURE_LOCATION', fallback=''),
	connection_string=settings.get(section, 'AZURE_CONNECTION_STRING', fallback=None),
	shared_container=settings.get(section, 'AZURE_SHARED_CONTAINER', fallback=True),
	azure_ssl=settings.get(section, 'AZURE_SSL', fallback=True),
	upload_max_conn=settings.get(section, 'AZURE_UPLOAD_MAX_CONN', fallback=2),
	timeout=settings.get(section, 'AZURE_CONNECTION_TIMEOUT_SECS', fallback=20),
	max_memory_size=settings.get(section, 'AZURE_BLOB_MAX_MEMORY_SIZE', fallback=2 * 1024 * 1024),
	expiration_secs=settings.get(section, 'AZURE_URL_EXPIRATION_SECS', fallback=None),
	overwrite_files=settings.get(section, 'AZURE_OVERWRITE_FILES', fallback=True),
	default_content_type=settings.get(section, 'AZURE_DEFAULT_CONTENT', fallback='application/octet-stream'),
	cache_control=settings.get(section, 'AZURE_CACHE_CONTROL', fallback=None),
	sas_token=settings.get(section, 'AZURE_SAS_TOKEN', fallback=None),
	custom_domain=settings.get(section, 'AZURE_CUSTOM_DOMAIN', fallback=None),
	token_credential=settings.get(section, 'AZURE_TOKEN_CREDENTIAL', fallback=None),
	azure_log_level=settings.get(section, 'AWS_LOG_LEVEL', fallback=logging.ERROR),
	cache_dir=settings.get(section, 'CACHE_DIR', fallback='/tmp/data-cache'),
	endpoint_url=settings.get(section, 'ENDPOINT_URL', fallback=None),

Example

OASIS_WORKER_MODEL_DATA_STORAGE_TYPE: S3
OASIS_WORKER_MODEL_DATA_AWS_BUCKET_NAME: oasislmf-model-library-oasis-piwind
OASIS_WORKER_MODEL_DATA_AWS_ACCESS_KEY_ID:  {_ID_}
OASIS_WORKER_MODEL_DATA_AWS_SECRET_ACCESS_KEY:  {_KEY_}
OASIS_WORKER_MODEL_DATA_ROOT_DIR: model_data/

When set on a worker container, these are used to create a model_storage.json file which is then passed into the execution command so model data is pulled directly from OASIS_WORKER_MODEL_DATA_AWS_BUCKET_NAME

[sambles]

The error retry: Retry in 6s: OasisException("Could not find events data file: ['events_p.bin', 'events_p.bin']") is caused by invalid bucket credentials. This needs to be checked and caught before preparing the model's run dir:

E.g.

(Pdb) model_storage.listdir()
*** PermissionError: The request signature we calculated does not match the signature you provided. Check your key and signing method.

[sambles]

[2024-05-01 09:17:57,658: INFO/ForkPoolWorker-1] RUNNING: oasislmf.execution.bin.prepare_run_directory
[2024-05-01 09:17:57,659: WARNING/ForkPoolWorker-1] 
  0%|          | 0/1 [00:00<?, ?it/s]
[2024-05-01 09:17:57,659: INFO/ForkPoolWorker-1] run_analysis[1accf3a6-fd79-49d8-94a3-7920ed6e7366]: Exception: <class 'oasis_data_manager.errors.OasisException'>: Error preparing the 'run' directory: '/tmp/tmpa4zrb8bx/input/oasis' and '/tmp/tmpa4zrb8bx/input/oasis' are the same file
[2024-05-01 09:17:57,711: INFO/ForkPoolWorker-1] Store file: /var/log/oasis/tasks/analysis_1_1accf3a6-fd79-49d8-94a3-7920ed6e7366.log -> d9c7e404160744949b43a2db39f376d8.log

This is caused by setting OASIS_AWS_LOCATION on the storage manager