Workflows

Contents

• Git workflows
• Updating code on the production server
• Resolving an internal server error
• Running South migrations
• Creating South migrations
• Adding a user on a development server

Git workflows

See the page on Git Workflows.

Updating code on the production server

Requirements before proceeding:

• You will need a user account on your production server.
• You may need to use sudo to have permission to run certain commands. For example, if apachectl start yields a "permission denied" error, then sudo apachectl start might work. To use sudo, you have to be added as a sudoer on the server.
  • Using sudo on git commands, however, is generally a bad idea. See the below requirements to set up Git permissions with your production server username.
• To use Git on the production server, you must have a Git credential helper or SSH key set up on the production server. See GitHub's guide for more info (make sure to select the server's operating system at the top of the page).
• You will need write permission for the directories in the Git repository. To ensure that all project collaborators can write to the directories, you should make sure all of these directories have g+w (group write) permission and are under a group which all project collaborators are part of. Suppose your group is called admin, and your project root directory is CoralNet. Then you would run sudo chgrp -R admin CoralNet and sudo chmod -R g+w CoralNet.
• Make sure the computer vision backend is not running! The easiest way to check that is to see whether a file "robot_running_flag" exists in the /cnhome/images directory. If that file exists, it means that the backend is processing images. This can in extreme cases take 2 days to complete. Another way is to use the 'top' command at the linux command line and see if the is any MATLAB or svm_train or svm_precict processes running. If there are, the backend is still processing. A third way is to look at the log files in /cnhome/logs, and see if there are any recent entries to cvlog.txt or celery_out.log. A forth, and perhaps the best way is to check the django celery queue. Do this: >>>from celery.task.control import inspect >>> i = inspect() or >>> i.scheduled() or >>> i.active(). You may have to cancel the cronjob (sudo crontab -e) and then wait a day or two before the current batch of jobs competes. A good way to clear the queue is to restart celeryd (using the supervisord tool).

Now, whenever you need to update code on the production server, you'd do the following:

0. Did you check that the vision tasks were not running!!!!???
1. SSH into the production server machine, and locate the directory with the project code.
2. Put up the maintenance message. Set the maintenance time to be at least several minutes after the current time. That way, users have some advance warning before you actually start messing with the site. More info on the maintenance message: here
3. Wait until your specified maintenance time begins.
4. Stop the server with apachectl stop. This is done because updating code while the server is running can temporarily leave the server code in an inconsistent state, which can lead to some very weird internal server errors. (Note that this is only the case when settings.DEBUG = False, which it should be for production servers.)
5. Follow the Updating procedure on the Git workflows page. This'll be simple because you almost never develop code on the production server; you just update it. In particular, git fetch origin and git rebase origin/master are probably all you will need.
6. If there are any new Python packages or other packages to install, then install them.
7. If there are any new settings to specify in settings_2.py, or new task helper functions to create in images/task_helpers.py, then do that.
8. If any static files (CSS, Javascript, etc.) were added or changed, run python manage.py collectstatic to serve those new static files.
9. If there are any new South migrations to run, then run those.
10. Start the server again with apachectl start.
11. If possible, quickly test the new functionality to check that's it's working.
12. Take down the maintenance message.
13. If you have modified the tasks, restart celeryd (python manage.py supervisor shell).

Resolving an internal server error

Almost always, when you need to resolve an internal server error, you'll need to know the error traceback.

• Sentry logs errors and their tracebacks, so if an error occurred, check the Sentry client, which is at /sentry . Log in as a site admin, using the same username and password as an admin account on the website.
• Occasionally, there will be an error that Sentry cannot log, or an error that prevents the Sentry client from being reachable. In that case, you'll need to check the Apache error log. The Apache error log location is configured with the ErrorLog directive, and this directive's value should be in one of the Apache configuration files (ending in .conf).

Special cases:

• If an internal server error is only occurring on the production server (not on any development machines), and/or intermittently occurring, then it's possible that someone forgot to do step 3 in "Updating code on the production server". Try restarting Apache (apachectl restart) and see if that stops the error from happening again.

Running South migrations

The South documentation is pretty readable and is a great resource to understand what South does and how to use it.

To get at least the basics of what database migrations are and what South does about them, read: What are migrations?, and at least skim the Tutorial's Part 1 and Part 2. It's also recommended to read the whole Tutorial, though; it could spare you some confusion later on.

Here's a quick run-down on running South migrations:

0. Check which migrations have yet to be run by typing python manage.py migrate --list (or python manage.py migrate <appname> --list to just see migrations for a particular app).
1. Run the migrations that you haven't yet run. You can run these:
   • All at once: python manage.py migrate
   • One app at a time: python manage.py migrate <appname>
   • One migration at a time. If you've already run up to migration 0003 in your app and the migrations go up to 0005, you can use python manage.py migrate <appname> 0004 to just run migration 0004 and not 0005.
   • In a dry-run mode, meaning you'll "test" the migrations without actually applying them to your database: python manage.py migrate --db-dry-run. Note that certain migrations cannot be dry-run, however, in which case this command would do nothing for that migration.
   • As backwards migrations: if you've just run migration 0003 and you want to roll back, you can do python manage.py migrate <appname> 0002 to do so.

Notes:

• See the database-specific issues. CoralNet uses MySQL, so if something goes wrong during a migration, South will probably turn up a long and fairly scary-looking error message, saying that your database may now be in an inconsistent state that is "between" migrations. This can be a pain to sort out. If this happens, then before trying anything else, you may want to go into the MySQL command line and examine your tables manually to get a sense of what has happened (i.e. are you really in a state that's between migrations, or did the migration just fail at the start, meaning you can just figure out what's wrong and re-run the migration?).

Creating South migrations

If you haven't already, see the "Running South migrations" section for some basic info on migrations.

The following procedure applies mainly to schema migrations. For data migrations (not as common), see Part 3 of the South Tutorial. Also note that the following procedure only covers the most common kinds of schema migrations. Read the rest of the South Tutorial for more special cases, such as custom fields and migration dependencies.

0. Change one of the Django apps' models code.
   • You'll need to go through the following steps to create a migration only if your change requires corresponding changes at the database level. This includes adding or deleting a model, adding or deleting a field, or making a field have null=True or null=False. It does not include some things like changing help text, or changing whether a field is editable in Django ModelForms that use that model.
1. Run python manage.py schemamigration <appname> --auto for the app whose model code you changed. This will start the process of auto-generating a migration file.
2. If you're adding or removing a field, or making a field accept null or not accept null, you may be asked to provide a default value for that field, so do so. You'll probably also be asked if you want the default value to apply only when this migration is run (a "one-off" default value), or if you want that default value to apply to this field from now on. Answer according to your needs.
   • You may wonder why you need to supply a default value even when removing the field. The reason is that if you apply the backwards migration, you'll be re-adding that field, and adding the field requires a default value.
3. The newly generated migration file will be in <appname>/migrations.
4. Run the migration as described in the "Running South migrations" section.
5. If you want to make extra sure that things will go smoothly (not a bad idea with migrations), try the backwards migration, too. So a thorough test procedure would involve running forwards, backwards, then forwards again so you're up to date.
6. When you're sure you're going to keep this migration, don't forget to add this migration file to Git version control so that other developers (and the server) can run the migration as well.

Notes:

• MySQL may have issues during the migration if you change a field from accepting null values (null=True) to not accepting null values (null=False). So if you do this, be aware that you may have to deal with an "inconsistent state" error.

Adding a user on a development server

Chances are you don't have a mail server on your machine, so the activation email can't be sent out. However, there is a workaround.

• Open accounts/forms.py and comment out the send_activation_email ... line.
• Log in as a site admin/superuser.
• Click Add User at the top of the site, and fill out the form to add a user.
• Go to the admin site. It should be something like: 127.0.0.1:8000/admin/
• Under "Auth", click "Users".
• Click the newly added user.
• Under "Permissions", check the "Active" checkbox.
• Save.
• Click the user again, and click "change password form" under the Password field. Use this form to change the user's password. You need to do this because there is no way to get the auto-generated password; it is only accessible through the activation email.

The user should be good to go now. Don't forget to revert the change you made to accounts/forms.py.

(Note: Don't actually add the user through the admin site. The Add User form sets some special permissions that the admin site doesn't seem to provide an interface for.)