Merge webinfra 1/1

EMAILS.md

@@ -0,0 +1,60 @@
+# Mass Emails
+
+## MIT Systems
+
+We're using mailman -- in particular, it handles unsubscribing for us.
+
+### Lists we maintain
+
+We currently maintain the following lists on Mailman:
+- `battlecode-interest-mm` is the general interest mailing list, linked to by the website homepage. This should be used for announcing new Battlecode contests.
+- `battlecode-competitors-2020-mm` is an empty mailing list created by accident. It can be used for testing.
+- `battlecode-competitors-2021-mm` contains all registered competitors of Battlecode 2021.
+
+### Creating a list
+
+1. Create a list at https://listmaker.mit.edu/lc/ and navigate to the admin page. You will receive an email containing the admin password.
+1. Update the administrator to the Battlecode contact email (this is the publicly viewable list owner), and set yourself and others to be moderators.
+   After doing this, you will need to re-authenticate (you're no longer the admin), using the password that was emailed to you.
+1. Set list parameters. Below are some examples.
+    - Description: MIT Battlecode 2021
+    - Subject prefix: [battlecode-21]
+    - Hide sender = Yes. All sent emails will appear to come from the list, not you.
+    - Strip Reply-To header = Yes, explicit Reply-To = the battlecode address. This configures replies to not go back to the list.
+    - In the privacy menu:
+        - Only list admin should view subscription list.
+        - In sender submenu, new members should be moderated. This prevents random subscribers from sending unmoderated mail.
+        - In recipient submenu, ceiling should be zero. This allows us to send mail even if the subscription list is huge.
+
+### Adding to a list
+
+Before doing this, turn off welcome messages in the general menu (`send_welcome_msg`). This ensures we don't spam people when they're added.
+
+We use mmblanche for this. It's easiest to use it installed on Athena. SSH in, then run:
+```
+bash  # if your default shell is not bash
+add consult
+mmblanche
+```
+to be able to use it. (`add consult` only works when running through bash, for some reason.)
+
+Get a list of emails, and convert it to the format as specified by mmblanche's `-f` argument: one email per line. (For example, if you're working with a Google Form, you can get a Google Sheet of responses. Create a new tab -- **not** the tab where form responses are being collected!. Convert it into only one column of values such that all the values are emails -- strip out timestamps, header row, and the like. Then download this as a csv. If you'd like to export emails from our database, you can use a nice interface to download a table as a csv, and work with it.)
+
+Move this file to your Athena locker; you can use `scp` for this (example [here](https://unix.stackexchange.com/questions/106480/how-to-copy-files-from-one-machine-to-another-using-ssh)).
+
+Finally run mmblanche! For example, `mmblanche [mailman-list-name] -al [path/to/file]`. Likely you'll want to use `-al`, as it simply adds any emails in the file to the list, skips over duplicates for you, and doesn't delete anything otherwise.
+
+You'll be prompted for the list admin password; find this somehow. You may wanna use the `-S` setting too, which will save admin passwords so you don't have to keep typing them.
+
+### Sending to a list
+
+If the set of emails you're trying to reach is continually growing (e.g. if you're messaging a large interest list through a  gsheet that a keeps growing), then you'll probably want to update the lists first. Follow the instructions in the above section.
+
+Send mail to the list as you would normally.
+- Ensure that you (yes, you specifically, not the Battlecode address) are a member of the list. Mark yourself as not-moderated in the Members list, or make sure you have a mod ready to release your email.
+- Do not use Bcc. Bcc'd mail will be blocked by the list.
+- Include trailing newlines. Mail without trailing newlines may show the message footer in the same line as your email signature.
+
+## Sendgrid
+
+(TODO dump some convos that I had w nate in Slack, as well as some messages that he sent in some channel.)

backend/api/views.py

@@ -28,21 +28,10 @@
 SUBMISSION_FILENAME = lambda submission_id: f"{submission_id}/source.zip"
 RESUME_FILENAME = lambda user_id: f"{user_id}/resume.pdf"
 
-# pub sub commands (from pub.py)
-def get_callback(api_future, data, ref):
-    """Wrap message data in the context of the callback function."""
-    def callback(api_future):
-        try:
-            print("Published message {} now has message ID {}".format(
-                data, api_future.result()))
-            ref["num_messages"] += 1
-        except Exception:
-            print("A problem occurred when publishing {}: {}\n".format(
-                data, api_future.exception()))
-            raise
-    return callback
-
-def pub(project_id, topic_name, data="", num_retries=5):
+# Methods for publishing a message to a pubsub.
+# Note that data must be a bytestring.
+# Adapted from https://github.com/googleapis/python-pubsub/blob/master/samples/snippets/quickstart/pub.py
+def pub(project_id, topic_name, data, num_retries=5):
     """Publishes a message to a Pub/Sub topic."""
 
     # Repeat while this fails, because the data is already in the
@@ -63,23 +52,9 @@ def pub(project_id, topic_name, data="", num_retries=5):
             # `projects/{project_id}/topics/{topic_name}`
             topic_path = client.topic_path(project_id, topic_name)
 
-            # Data sent to Cloud Pub/Sub must be a bytestring.
-            #data = b"examplefuncs"
-            if data == "":
-                data = b"sample pub/sub message"
-
-            # Keep track of the number of published messages.
-            ref = dict({"num_messages": 0})
-
             # When you publish a message, the client returns a future.
-            api_future = client.publish(topic_path, data=data)
-            api_future.add_done_callback(get_callback(api_future, data, ref))
-
-            # Keep the main thread from exiting while the message future
-            # gets resolved in the background.
-            while api_future.running():
-                time.sleep(0.5)
-                # print("Published {} message(s).".format(ref["num_messages"]))
+            api_future = client.publish(topic_path, data)
+            message_id = api_future.result()
         except:
             pass
         else:
@@ -708,14 +683,21 @@ def retrieve_file(self, request, team, league_id, pk=None):
     def compilation_pubsub_call(self, request, team, league_id, pk=None):
         # It is better if compile server gets requests for compiling submissions that are actually in buckets. 
         # So, only after an upload is done, the frontend calls this endpoint to give the compile server a request.
+
+        # Only allow if superuser, or on the team of the submission
+        # Also make sure that the admin is on a team! Otherwise you may get a 403.
         submission = self.get_queryset().get(pk=pk)
-        if team != submission.team:
-            return Response({'message': 'Not authenticated on the right team'}, status.HTTP_401_UNAUTHORIZED)
+        is_admin = User.objects.all().get(username=request.user).is_superuser
+        if not ((team == submission.team) or is_admin):
+            return Response({'message': 'Not authenticated on the right team, nor is admin'}, status.HTTP_401_UNAUTHORIZED)
 
         # If a compilation has already succeeded, keep as so; no need to re-do.
-        # (Might make sense to re-do for other submissions, however.)
+        # (Might make sense to re-do for other submissions though, for example if messages are accidentally taken off the compilation pubsub queue.)
         if submission.compilation_status == settings.COMPILE_STATUS.SUCCESS:
             return Response({'message': 'Success response already received for this submission'}, status.HTTP_400_BAD_REQUEST)
+        # Only allow the admin to re-queue submissions, to prevent submission spam.
+        if (submission.compilation_status != settings.COMPILE_STATUS.PROGRESS) and (not is_admin):
+            return Response({'message': 'Only admin can attempt to re-queue submissions'}, status.HTTP_400_BAD_REQUEST)
 
         # indicate submission being in a bucket
         submission.compilation_status = settings.COMPILE_STATUS.UPLOADED
@@ -759,6 +741,7 @@ def compilation_update(self, request, team, league_id, pk=None):
                     # Only if this submission is newer than what's already been processed,
                     # update the submission history. 
                     # (to prevent reverting to older submissions that took longer to process)
+                    # (The compile server should generally be processing submissions in the same order they were uploaded, anyways. But this check is still good in case of async conditions, re-queueing, etc)
                     if team_sub.last_1_id is None or submission.id > team_sub.last_1_id:
                         team_sub.last_3_id = team_sub.last_2_id
                         team_sub.last_2_id = team_sub.last_1_id
@@ -1033,7 +1016,7 @@ def set_outcome(self, request, league_id, team, pk=None):
                         return Response({'message': 'Must include both winscore and losescore in request.'},
                                         status.HTTP_400_BAD_REQUEST)
 
-                    if int(sc_winscore) < (float(sc_winscore) + float(sc_losescore))/2.0:
+                    if sc_winscore <= sc_losescore:
                         return Response({'message': 'Scores invalid. Winscore must be at least half of total games.'}, status.HTTP_400_BAD_REQUEST)
                     scrimmage.status = sc_status
                     scrimmage.winscore = sc_winscore
@@ -1081,6 +1064,7 @@ def set_outcome(self, request, league_id, team, pk=None):
                     scrimmage.status = sc_status
 
                     scrimmage.save()
+                    # Return 200, because the scrimmage runner should be informed that it successfully sent the error status to the backend
                     return Response({'status': sc_status, 'winscore': None, 'losescore': None}, status.HTTP_200_OK)
                 else:
                     return Response({'message': 'Set scrimmage to pending/queued/cancelled with accept/reject/cancel api calls'}, status.HTTP_400_BAD_REQUEST)

frontend/README.md

@@ -34,12 +34,22 @@ This automatically reloads the page on changes. To run the same thing without au
 
 When installing a new Node package, always `npm install --save <package>` or `npm install --save-dev <package>`, and commit `package.json` and `package-lock.json`. This should work even if we run it from Docker. If you don't have `npm` installed on your computer, you can `docker exec -it battlecode20_frontend_1 sh` and then run the commands above.
 
+Our local processes (including our dockerfile) use `npm start` and/or `npm run start`. These commands automatically use `.env.development`, and not `.env.production`. See here for more information: https://create-react-app.dev/docs/adding-custom-environment-variables/#what-other-env-files-can-be-used
+
 ## Deployment
 
 For production, build with `npm run build` for the full thing, and `npm run buildnogame` to build the site without any game specific information. This is handled automatically by calling `./deploy.sh deploy` or `./deploy.sh deploynogame` using Bash, respectively. Note that the former should ONLY be called after the release of the game, since it makes the game specs and the visualizer public.
 
+### access.txt
+
+During deployment, you'll need an up-to-date version of `frontend/public/access.txt`. This file is needed by game runners to run matches, and by competitors because it grants them access to downloading the GitHub package containing the engine. It's is really difficult to deploy; our solution is to have it deployed with the rest of the frontend code and onto our website, but have it never pushed to GitHub. Make sure you have an up-to-date copy! If you don't have one, check with the infra devs.
+
+### Assorted notes
+
 Notably, the servers that serve the deployed frontend never run npm (through Docker or otherwise). Instead, our deploy script runs npm locally to build the frontend, and then sends this compiled version to Google Cloud.
 
+Deployed code automatically builds using `.env.production`, since we call it with `npm run build`. See here for more information: https://create-react-app.dev/docs/adding-custom-environment-variables/#what-other-env-files-can-be-used
+
 ### One-time setup
 
 #### AWS
@@ -53,6 +63,8 @@ We first need to register the subdomain.
 
 This should create the subdomain `2021.battlecode.org` and point it to our load balancer.
 
+With this new subdomain registered, make sure to update the URLs in `.env.production`to this new URL.
+
 #### Google Cloud
 
 We now need to set up a new bucket, and set up the load balancer to point to the bucket when navigating to the right address.

frontend/src/api.js

@@ -40,21 +40,21 @@ class Api {
         console.log(data, status)
         $.post(`${URL}/api/${LEAGUE}/submission/` +Cookies.get('submission_id') + `/compilation_pubsub_call/`)
         .done((data, status) => {
-          Cookies.set('submission_upload_status', 1)
+          Cookies.set('upload_status_cookie', 11)
         })
         .fail((xhr, status, error) => {
           console.log("Error in compilation update callback: ", xhr, status, error)
-          Cookies.set('submission_upload_status', 3)
+          Cookies.set('upload_status_cookie', 13)
         })
       })
       .fail((xhr, status, error) => {
         console.log("Error in put request of file to bucket: ", xhr, status, error)
-        Cookies.set('submission_upload_status', 3)
+        Cookies.set('upload_status_cookie', 13)
       })
     })
     .fail((xhr, status, error) => {
       console.log("Error in post request for upload: ", xhr, status, error)      
-      Cookies.set('submission_upload_status', 3)   
+      Cookies.set('upload_status_cookie', 13)   
     });
 
   }