Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better error message when database name doesn't have valid regex match. #432

Open
whiphi92 opened this issue Feb 21, 2024 · 2 comments
Open
Labels
enhancement New feature or request

Comments

@whiphi92
Copy link

Describe the bug
I received the following error message during YML validation:

yamale.yamale_error.YamaleError: Error validating data '/home/pwhite/projects/heat-pump-deployment/resstock/project_national/national_upgrades_HPs.yml' with schema '/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/lib/python3.11/site-packages/buildstockbatch/schemas/v0.3.yaml' postprocessing.aws.athena.database_name: 'heat-pump-deployment' is not a regex match.

I commented out the database block in the YML and it validated successfully.

For reference, my s3 bucket is heat-pump-deployment, which at the beginning of this ResStock run was completely empty and no database had been created yet.

To Reproduce
Steps to reproduce the behavior:

  1. Use buildstock_kestrel on a YML that has an empty s3 bucket and no database yet
  2. This error should proc

Expected behavior
I expect bsb to automatically make the database during the upload and pass this error. Maybe it works differently? I have never encountered this before.

Logs

(buildstock-2023.11.0) [pwhite@kl1 resstock]$ buildstock_kestrel project_national/national_upgrades_HPs.yml

     _ __         _     __,              _ __
    ( /  )    o  //   /(    _/_       / ( /  )     _/_    /
     /--< , ,,  // __/  `.  /  __ _, /<  /--< __,  /  _, /
    /___/(_/_(_(/_(_/_(___)(__(_)(__/ |_/___/(_/(_(__(__/ /_
      Executing BuildStock projects with grace since 2018


Traceback (most recent call last):
  File "/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/bin/buildstock_kestrel", line 8, in <module>
    sys.exit(kestrel_cli())
             ^^^^^^^^^^^^^
  File "/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/lib/python3.11/site-packages/buildstockbatch/hpc.py", line 838, in kestrel_cli
    user_cli(KestrelBatch, argv)
  File "/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/lib/python3.11/site-packages/buildstockbatch/hpc.py", line 893, in user_cli
    Batch.validate_project(project_filename)
  File "/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/lib/python3.11/site-packages/buildstockbatch/hpc.py", line 794, in validate_project
    super().validate_project(project_file)
  File "/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/lib/python3.11/site-packages/buildstockbatch/hpc.py", line 82, in validate_project
    super().validate_project(project_file)
  File "/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/lib/python3.11/site-packages/buildstockbatch/base.py", line 271, in validate_project
    assert cls.validate_project_schema(project_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/lib/python3.11/site-packages/buildstockbatch/base.py", line 381, in validate_project_schema
    return yamale.validate(schema, data, strict=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/lib/python3.11/site-packages/yamale/yamale.py", line 43, in validate
    raise YamaleError(results)
yamale.yamale_error.YamaleError: Error validating data '/home/pwhite/projects/heat-pump-deployment/resstock/project_national/national_upgrades_HPs.yml' with schema '/kfs2/shared-projects/buildstock/envs/buildstock-2023.11.0/lib/python3.11/site-packages/buildstockbatch/schemas/v0.3.yaml'
        postprocessing.aws.athena.database_name: 'heat-pump-deployment' is not a regex match.
(buildstock-2023.11.0) [pwhite@kl1 resstock]$ buildstock_kestrel project_national/national_upgrades_HPs.yml

     _ __         _     __,              _ __
    ( /  )    o  //   /(    _/_       / ( /  )     _/_    /
     /--< , ,,  // __/  `.  /  __ _, /<  /--< __,  /  _, /
    /___/(_/_(_(/_(_/_(___)(__(_)(__/ |_/___/(_/(_(__(__/ /_
      Executing BuildStock projects with grace since 2018


INFO:2024-02-21 09:41:37:buildstockbatch.base:Base Validation Successful
INFO:2024-02-21 09:41:37:buildstockbatch.hpc:HPC Validation Successful
INFO:2024-02-21 09:41:37:buildstockbatch.hpc:Kestrel Validation Successful
INFO:2024-02-21 09:41:37:buildstockbatch.hpc:Creating output directory /projects/heatpump/test_runs/n_500_20240221_v1
INFO:2024-02-21 09:41:37:buildstockbatch.hpc:Submitting sampling job to task scheduler
Submitted batch job 2668859
INFO:2024-02-21 09:41:38:buildstockbatch.hpc:Run squeue -u $USER to monitor the progress of your jobs
(buildstock-2023.11.0) [pwhite@kl1 resstock]$ watch squeue -u pwhite
(buildstock-2023.11.0) [pwhite@kl1 resstock]$ 

The successful validation is after I removed the database block

Platform (please complete the following information):

  • Simulation platform: Kestrel
  • BuildStockBatch version, branch, or sha: buildstock-2023.11.0
  • resstock or comstock repo version, branch, or sha: max-airflow-args2 branch of resstock
  • Local Desktop OS: Windows

Additional context
None

@whiphi92 whiphi92 added the bug Something isn't working label Feb 21, 2024
@rajeee
Copy link
Contributor

rajeee commented Feb 21, 2024

Database name must be lowercase, start from letters and contain only letters, numbers and underscore character.

https://buildstockbatch.readthedocs.io/en/stable/project_defn.html#postprocessing-configuration-options

The database name cannot contain dash.

@rajeee rajeee closed this as completed Feb 21, 2024
@afontani afontani reopened this Feb 21, 2024
@afontani
Copy link
Collaborator

We want to re-open this issue to get a more intellagent error message in the validator.

@rajeee rajeee added the enhancement New feature or request label Oct 28, 2024
@rajeee rajeee changed the title Receiving error that database name for a new s3 bucket is not a regex match Better Error message when database name doesn't have valid regex match. Oct 28, 2024
@rajeee rajeee changed the title Better Error message when database name doesn't have valid regex match. Better error message when database name doesn't have valid regex match. Oct 28, 2024
@rajeee rajeee removed the bug Something isn't working label Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants