diff --git a/website/docs/adding_meters_buildings.md b/website/docs/adding_meters_buildings.md index 5f19c07..547e0d6 100644 --- a/website/docs/adding_meters_buildings.md +++ b/website/docs/adding_meters_buildings.md @@ -12,7 +12,7 @@ description: Checklist of tasks needed for adding new meters / buildings ::: -## Overview +## MySQL Workbench The following tables in the SQL database should be updated each time a new meter and/or building is added: @@ -27,6 +27,20 @@ The following tables in the SQL database should be updated each time a new meter - Match the corresponding meter group and meter ID together in. One meter group can be mapped to several meters, but not the other way around +- See [Database](database) document for help on updating / inserting buildings and meters + +## automated-jobs + - Also need to update [validIDs file in automated-jobs](https://github.com/OSU-Sustainability-Office/automated-jobs/blob/main/check-acq/validIDs.json) to reflect the added buildings, meters, meter groups - Needed to check if the newly added API endpoints are up at any given time. See [Cloudwatch](cloudwatch) for more info - Might integrate this better in the database at some point + +## AWS S3 + +- Go to AWS Web Console > S3 > Buckets > osu-energy-images +- Upload a picture of the newly added building (if applicable). Google images ought to be enough to find a good result +- Upload 1 picture to root directory of osu-energy-images +- Resize image for thumbnail with https://imageresizer.com/ + - Set width as **400px**, leave the height option blank to let the height scale with original proportions + - Upload resized images to `thumbnails` directory of osu-energy-images +- Go back and make sure the file name of the image you uploaded matches the image filename as listed in `buildings` table in the SQL database as seen in [MySQL Workbench](adding_meters_buildings#MySQL-Workbench) diff --git a/website/docs/cloudwatch.md b/website/docs/cloudwatch.md index 780fdb9..f759553 100644 --- a/website/docs/cloudwatch.md +++ b/website/docs/cloudwatch.md @@ -33,50 +33,11 @@ description: Overview of Cloudwatch and SNS Email Alerts - If you get a missed meter upload notification (TimeoutError) email or otherwise notice some missing or incorrect data for the Solar Panel buildings ([SEC Solar](https://dashboard.sustainability.oregonstate.edu/#/building/30/2) and [OSU Operations](https://dashboard.sustainability.oregonstate.edu/#/building/42/2)), then insert the missing data via MySQL workbench - - Check the `.env` file in the automated-jobs repo to reference where to log in for solar panel data. Clicking on one of the building names on the Plants page after you log in will bring up a table with daily and monthly data, including historical data + - Check the [`.env`](https://drive.google.com/file/d/12dCdA5E5e6qPgkSYehqOcX_zVy9YztFF/view?usp=sharing) file (must be OSU employee to see) for automated-jobs to reference where to log in for solar panel data. Clicking on one of the building names on the Plants page after you log in will bring up a table with daily and monthly data, including historical data - Most of the fields should be pretty self explanatory to insert into the Solar_Meters table in MySQL workbench, but for the time_seconds value, reference the playcode below for how to get the Unix timestamp -- INSERT (for missing data) and UPDATE (for fixing incorrect data etc) will be the most useful here as far as SQL commands -- Use basic precautions, make sure you have highlighted only the lines of SQL you want to run before running it (clicking the yellow lightning symbol in MySQL workbench) -- By default, MySQL workbench will forbid you from inserting, updating, or deleting multiple data entries without specifying an index range, so this should help prevent careless errors, but still, be careful! - - We don't have a dev database, so any changes in MySQL workbench hit production right away, so to speak. It can also be a good idea to back up data (e.g. as an Excel table, or at least taking some screenshots of what the database looked like) before performing any operation that could affect a lot of data entries -- It can be useful to sort by time_seconds (just click the column after running `SELECT * from Solar_Meters`) to keep track of the data entries in order, especially if you had to at some point retroactively insert missing data into the database +- Review the [database](database) document for general instructions / tips for how to insert or update etc. data to the Solar_Meters table -### Unix Timestamps - -- Useful reference / converter: https://www.unixtimestamp.com/index.php - - We are using millisecond precision for the webscrapers, to keep in mind for the Unix timestamps, your time_seconds values should have 10 digits -- Useful sandbox - [https://playcode.io/1457582](https://playcode.io/1457582) - -```js -const date = new Date('May 27, 2023 23:59:59 GMT+0'); - -// Calculate the Unix time in seconds -const unixTimeSeconds = Math.round(date.getTime() / 1000); - -console.log(unixTimeSeconds); -``` - -### SQL Command Examples - -**These are example commands below, please substitute the correct values as needed!** - -Again, refer to the [Unix Timestamps section above](webscraper_tutorial#unix-timestamps) for `time_seconds` value. - -Rest should be pretty self-explanatory. Remember that the `energy_change` value of `OSU_Operations_Total` = "OSU Operations" + "OSU Operations Lube Shop" in the portal website linked in the `.env` file - -For inserting missing data: - -`select * from Solar_Meters;` - -`` INSERT INTO Solar_Meters (`time`, `time_seconds`, `energy_change`, `tableid`) VALUES ('2023-7-02T23:59:59', 1688342399, 233.74, 'SEC_Solar'); `` - -`` INSERT INTO Solar_Meters (`time`, `time_seconds`, `energy_change`, `tableid`) VALUES ('2023-7-02T23:59:59', 1688342399, 2424.89, 'OSU_Operations_Total'); `` - -If you just need to update a value (example): - -``` -UPDATE Solar_Meters -SET time = '2023-7-2T23:59:59' -WHERE id IN (737, 738); -``` +- It can be useful to sort by `time_seconds` (just click the `time_seconds` column after running `SELECT * from Solar_Meters`) to keep track of the data entries in order, especially if you had to at some point retroactively insert missing data into the database +- Refer to [Database](database) document for more detailed instrutions on how to insert or update any missing data +- Rest should be pretty self-explanatory. Remember that the `energy_change` value of `OSU_Operations_Total` = `energy_change` of `OSU Operations` + `energy_change` of `OSU Operations Lube Shop` in the portal website linked in the [`.env`](https://drive.google.com/file/d/12dCdA5E5e6qPgkSYehqOcX_zVy9YztFF/view?usp=sharing) file diff --git a/website/docs/database.md b/website/docs/database.md new file mode 100644 index 0000000..e0d3ed5 --- /dev/null +++ b/website/docs/database.md @@ -0,0 +1,101 @@ +--- +title: Database +description: Deep Dive on MySQL Database +--- + +- **DO NOT** run any commands here blindly in the MySQL Workbench. Edit to fit your needs, checking you have set correct parameters + +## General Tips + +- For MySQL Workbench, I like using the commands: + - `SELECT` for viewing database tables + - `INSERT INTO` for adding elements + - `UPDATE` for editing tables + - `DELETE` for deleting table elements +- Remember for general syntax: + - Database or column etc names should use backticks (`/`) + - Actual values (the payload) should use single quote (`` ` ``) (if applicable, like if you have a string value. Numbers don't need the single quote for value) + - Example: `` INSERT INTO Solar_Meters (`time`, `time_seconds`, `energy_change`, `tableid`) VALUES ('2023-3-6T23:59:59', 1678175999999, 766.72, 'SEC_OSU_Op'); `` +- **WARNING**: Messing with the data in MySQL workbench **directly mutates the data** and I'm not sure about data backups. Use with care, although by default MySQL workbench should use "safe" settings + - Make sure you have highlighted only the lines of SQL you want to run before running it (clicking the yellow lightning symbol in MySQL workbench) + - Remember to specify the primary key value (e.g. add `WHERE ID = 5` to the query) for: + - `UPDATE` + - `DELETE` + - You don't have to specify the primary key (usually called `id`) for `INSERT INTO`, but it is good practice + - You can use a range for the primary key like `WHERE ID < 5` if you have to mass delete / insert / update something, but **do so with caution** +- We don't have a dev database, so any changes in MySQL workbench hit production right away, so to speak. It can also be a good idea to back up data (e.g. as an Excel table, or at least taking some screenshots of what the database looked like) before performing any operation that could affect a lot of data entries + +## Relationships between Tables + +Entity Relationship Diagram: +![Alt text](../static/img/erd.png) + +- Note that `buildings` table primary key `building_id` corresponds to `building_id_2` in `meter_groups` +- Each `meter_groups` value corresponds to several `meters` values via meter_group_relation as an interface table +- Each `buildings` value uses a `meter_groups` ID (primary key) value as a foreign key +- Each `campaigns` value corresponds to several `campaign_groups` values +- Each `campaign_groups` value corresponds to a `meter_groups` ID (primary key) value + +## More Specific SQL Instructions + +- Values surrounded by `<>` should be read as placeholder values in the SQL commands listed below + +### Inserting (Creating) + +1. `SELECT * from ` +2. `` INSERT INTO (
<`every column name you want in your table`>) VALUES () `` + - For this command, remember that the column names should be surrounded by backticks (`` ` ``) while **string** values should be surrounded by single quotes (`'`) + - Technically you can leave out the primary key for the column name / value. It will increment the primary key automatically in that case + +EXAMPLE (edit for your own needs): + +- `` INSERT INTO buildings (`id`,`map_id`,`image`,`group`,`name`,`hidden`) VALUES (43,'',NULL,'Solar','OSU Operations',0); `` + +### Selecting (Reading) + +` SELECT * from
` + +- See [Relationships between Tables](#relationships-between-tables) section above + +### Updating + +1. `SELECT * from
` +2. `UPDATE
SET = where = ;` + - For this command, **string** values should be surrounded by single quotes (`'`). The column names do not need backticks here. + +EXAMPLE (edit for your own needs): + +- `UPDATE campaign_groups SET group_id = 179 WHERE id = 53;` + +### Deleting + +1. `SELECT * from
` +2. `DELETE from
where = ;` + +EXAMPLE (edit for your own needs and DO NOT run this command unedited in MySQL: + +- `delete from meter_group_relation where id = 526;`) + +## Example SQL Commands File + +The below SQL Commands file is meant to serve as an example for correct syntax in case anything above is still not clear. + +- [Click Here for File](https://drive.google.com/file/d/1otAp6gcCr3qWLkSHZZJ4xBGOd4RxmjJ7/view?usp=drive_link) + - Must be OSU Sustainability paid employee to see above link + - Again, DO NOT run anything from here without editing to fit your needs. Make sure to highlight line by line in MySQL Workbench. Only highlight commands you intend to run + +## Unix Timestamps + +- Usually referred to as `time_seconds` value in data tables +- Useful reference / converter: https://www.unixtimestamp.com/index.php + - We are using millisecond precision for the webscrapers, to keep in mind for the Unix timestamps, your time_seconds values should have 10 digits +- Useful sandbox - [https://playcode.io/1457582](https://playcode.io/1457582) + +```js +const date = new Date('May 27, 2023 23:59:59 GMT+0'); + +// Calculate the Unix time in seconds +const unixTimeSeconds = Math.round(date.getTime() / 1000); + +console.log(unixTimeSeconds); +``` diff --git a/website/docs/faq.md b/website/docs/faq.md index 7a5d1b2..e882a36 100644 --- a/website/docs/faq.md +++ b/website/docs/faq.md @@ -42,4 +42,9 @@ Important AWS Services for OSU SO (non-exhaustive) - [Solar data missed upload](cloudwatch) - [Creating new Kilowatt Campaign](kilowatt_crackdown) - Anything else touching the database -- **WARNING**: Messing with the data in MySQL workbench **directly mutates the data** and I'm not sure about data backups. Use with care, although by default MySQL workbench should use "safe" settings, e.g. must specify primary key values for deleting values +- **See [Database document](database) for more detailed info** + +## Credentials + +- [Credentials Folder](https://drive.google.com/drive/u/1/folders/1geuKCp-aTIrde2WdJkE3f_L2TsF46_O3) + - Need to be OSU SO employee to see this link diff --git a/website/sidebars.js b/website/sidebars.js index d022265..23dddda 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -38,7 +38,14 @@ const sidebars = { slug: 'specific', }, - items: ['webscraper_tutorial', `energy_dashboard_graphs`, `lambda_common_layer`, `carbon_calculator`, 'kiosks'], + items: [ + 'database', + 'webscraper_tutorial', + `energy_dashboard_graphs`, + `lambda_common_layer`, + `carbon_calculator`, + 'kiosks', + ], }, { type: 'category', diff --git a/website/static/img/erd.png b/website/static/img/erd.png new file mode 100644 index 0000000..c957444 Binary files /dev/null and b/website/static/img/erd.png differ