Setup

This page is about setting up your own machine to develop and run the CoralNet project (or a fork of the project).

Contents

• Git and GitHub
• MySQL
• Python
• Python packages
• Development environment
• Final setup steps for Django
• Celery

Git and GitHub

Setting up Git and GitHub

Follow GitHub's guide for setting up Git and GitHub. Make sure to select your operating system at the top of the page.

• As of 2012 October 5, the top of that guide suggests using GitHub's native app for setting up and using Git and GitHub. This hasn't yet been tried by anyone on the CoralNet team, but you're welcome to try it out. It's available for Windows and Mac.

Contributing

• Core collaborators, who can push directly to the main CoralNet repository, can only be added manually by the repository admin. In general, we don't accept requests for core team members.
• Anyone is welcome to fork the CoralNet repository and submit pull requests. For more information, see here.
• Lighter ways of contributing to the project include creating Issues and submitting patches.

Getting the code repository onto your machine

• Run a git clone command on the repository you're going to work on. For example, for the main CoralNet project, it's git clone https://github.com/DevangS/CoralNet.git.

MySQL

Setup guide on the MySQL site

Download

• Here
• We use MySQL 5.1 (as of 2012 October 5). All CoralNet developers should use the same version of MySQL.
• Some of the Windows packages are labelled as "Essentials" and some aren't (Essentials is a smaller package). Essentials will suffice for CoralNet development.
• If your have a 32-bit machine, get 32-bit MySQL. If you have a 64-bit machine, either 64 or 32 bit MySQL should be fine. 32 vs. 64 bit shouldn't raise any compatibility issues between developers.
• After picking a package to download, skip the registration stuff; just scroll down and click "No thanks, just start my download!".

Database server setup

There's a setup wizard for Windows, and other setup methods for other OSes. Whichever way you install, make sure you do these things:

• Enable InnoDB tables (if this is asked). It's very important that the database tables support transactions.
• Enable TCP/IP networking. This will make it easier for Django to connect to MySQL on a development machine.
• When you set the root password, remember what that password is.

Create a database and a user

• Now that you've installed and configured MySQL, start the MySQL Server command line.
• You'll be asked for the root password, so enter the root password. Then you'll be able to start entering MySQL commands.
• Create a database to use for your development server. If you wanted to name the database coralnet, then use: CREATE DATABASE coralnet;
• Create a user for Django to log in as to access the database. If you wanted the username usernamehere and the password passwordhere, then use: CREATE USER 'usernamehere'@'localhost' IDENTIFIED BY 'passwordhere';
• Give that user permission to do anything with the database: GRANT ALL ON coralnet.* TO 'usernamehere'@'localhost';
• When running Django unit tests, a separate test database is created, used for the duration of the test, and then destroyed after tests are finished. That test database has the same name as your main database, except it's prefixed with "test_". You'll want to grant permission to do anything with this test database, too: GRANT ALL ON test_coralnet.* TO 'usernamehere'@'localhost';

Python

Install Python

• CoralNet uses Python 2.7.1 (as of 2012 October 6). Anything 2.7.x should be fine for development, so if you already have Python and the version number is 2.7.0 through 2.7.5 (or whatever the latest 2.7.x is), then use that.
• 2.6.x could run into issues due to not supporting everything that 2.7.x has. For example, 2.6.x doesn't have the argparse module, and doesn't have the curly-brace set literal syntax.
• 3.x should be avoided for now, as some common functions will not work in 3 the same way it did in 2. See here for examples of Python 2 vs. 3 differences. Notable differences include print 'hello' becoming print('hello'), and dict.keys() returning a "view" instead of a list. Another reason to avoid 3.x for now is that some of the Python packages we use may outright refuse to install when seeing your Python is 3.x, because the package was coded for 2.x.
• Note that you may have multiple versions of Python on the same machine without problems. For example, on Windows you can have C:\Python26 and C:\Python27 side by side, or perhaps usr/lib/python27 and usr/lib/python33 on Linux. Just be sure to set your system path so that when you call python, easy_install, pip, etc., you're actually referring to the Python installation that you think you are.
• For Windows, if your have a 32-bit machine, get 32-bit Python. If you have a 64-bit machine, it's recommended to still get 32-bit Python. 64 bit is workable, but can make things tricky; also, certain Python utilities like PyYAML don't seem to have 64-bit Windows binaries.

Python packages

Pip

• There are two main tools used to install Python packages: pip, and easy_install. pip is newer and meant to improve on easy_install. It's recommended to use pip for most Python package installations.
• Installing pip - If you don't already have virtualenv, install pip by either using get-pip or installing from source, as described in the link (as of 2013 Jul 08).

Virtualenv (optional)

virtualenv - if you have other Python projects that may require different versions of the same packages that CoralNet uses, then you should strongly consider using virtualenv. It will allow you to keep separate sets of Python packages under separate directories.

• You can install virtualenv using pip.
• Create an env for CoralNet, and be sure to run the activate script for that env before installing any packages that you will use only for CoralNet.
• If a package can't be installed in a virtualenv for whatever reason, you can try the following:
  • Install the package into your global Python installation (e.g. something like C:\Python27 on Windows).
  • Go to your global Python installation's Lib/site-packages directory and locate the package's directory (and .pth file, if any). Copy these into the CoralNet env's Lib/site-packages directory. This should "install" the packages into that env.
• If you use the PyCharm IDE (described later), you may have to:
  • Point it toward the Scripts/python.exe of your CoralNet env. You would set this in Settings -> Project Settings -> Project Interpreter -> Python Interpreters.
  • Manually make Run/Debug configurations to make PyCharm find and run your unit tests.

Packages with specific installation instructions

• MySQL-Python adapter
  • On Windows, install with the binary (.exe). Make sure to get 32 bit, assuming your Python installation is 32 bit.
    • You can still use the binary even if you're installing to a virtualenv. Just download the .exe, activate the virtualenv, then do easy_install <path to .exe>.
  • On Mac OS X 10.8: the installation path is wrong for Mountain Lion. You'll have to resolve the path for where your installation of mySQL is in order for python to see it.
• PIL (Python Imaging Library) OR Pillow, a PIL fork
  • Django previously didn't have a preferred Python imaging library, but as of Django 1.6, Pillow is considered "the preferred image manipulation library to use with Django." Thus, it is probably best to install Pillow.
  • PIL has packaging issues that make it cumbersome to install or otherwise work with. This is one big reason why Pillow is preferred. (Pillow installation may not be 100% smooth sailing, but it is still easier than PIL.)
  • If installing Pillow, install version 2 or greater.
  • System-specific tips:
    • Pillow + Windows: As the Pillow page says (as of Pillow 2.1.0), using pip to install Pillow might not work in Windows. Instead, use easy_install Pillow.
    • Pillow + Ubuntu: You'll first have to apt-get libraries depending on which image formats you need to support; for example, libjpeg8-dev to support jpeg.
    • PIL + Windows: You can download a binary of the latest version, found in the PIL link above.
    • PIL + Mac OS X (10.8 Mountain Lion): You will need to install PIL and libjpeg. Install homebrew and install libjpeg BEFORE installing pil. Once that is finished, run pip install pil. If you see the confirmation that jpeg files are supported, then the installation worked. If not, uninstall pil and try again.
• NumPy
  • On Windows, install with a binary (.exe), which can be found on PyPI instead of numpy.org (as of 2013 Jul 08).
    • You can still use the binary even if you're installing to a virtualenv. Just download the .exe, activate the virtualenv, then do easy_install <path to .exe>.

Packages that don't require special steps to install with pip

Read about using pip - here are some commonly useful commands:

• Install the latest version of my-package: pip install my-package
• Upgrade an existing installation: pip install -U my-package
• Upgrade an installation, without touching dependency packages: pip install -U --no-deps my-package
• Install version 1.2.3 of my-package: pip install my-package==1.2.3
• Install the latest version of my-package that's less than 1.2: pip install my-package<1.2
• Uninstall my-package: pip uninstall my-package
• Install a package from an archive file (useful when a particular version isn't hosted on PyPi): pip install path/or/url/to/archive

For CoralNet, get the following:

• django==1.3.0
• django-userena==1.0.1
  • This will probably auto-install django-guardian and easy_thumbnails (listed below) as dependencies. This is fine.
• django-guardian
• easy_thumbnails
• South
• django-reversion==1.5.0
  • This should be updated if django is updated. See here.
• django-sentry<1.13
  • Make sure you install 'django-sentry', not 'sentry'.
  • 1.13 and above obsolete sentry.client, which we use, hence the <1.13.
  • If < is a special character in your shell, try: pip install "django-sentry<1.13"
• django-dajaxice==0.2
  • You'll probably need to specify the URL instead of auto-finding it with pip: Get it here
• django-dajax==0.8.4
• django-celery
• GChartWrapper
• pyyaml
  • This is required to run most of our Django unit tests, because the test fixtures (which specify initial database data to use in the tests) are written in YAML.
• django-supervisor

Notes:

• Many of the above packages will also auto-install other packages as dependencies, particularly django-sentry and django-celery.
• On Windows, if you get an error saying "Unable to find vcvarsall.bat", try this workaround. (Yes, unfortunately, it requires installing MinGW if you don't have it already.)

Development Environment

Here are some recommended programs to use for Django coding.

PyCharm IDE

Get it here.

It's not freeware (although you can get a 30-day free trial to try it out). See here for licensing options. In particular:

• If you're a student, you may be able to apply for a Classroom License for free, or an Academic License for $29. Or, check your school's computing services (or similar) to see if they already have a PyCharm license available for students to use.
• You could just get a Personal License for $99.
• As of 2012 October 5, we're not sure if CoralNet developers could get a free Open Source License, but it's something we should look into later.

Pros:

• Great support for Django, such as server debugging, template syntax highlighting, relevant code suggestions, and generally understanding the structure of a Django project.
• Syntax highlighting and code suggestions for client-side languages such as Javascript and CSS.
• Integration with version control, particularly Git, and a very good Diff tool.

Cons:

• Uses a lot of resources, so can be slow on less powerful machines (such as a netbook with an Atom processor and 2 GB memory).
• Code suggestions are occasionally a bit overzealous. However, the kinds of suggestions can be turned off individually.

Other IDEs

Wingware, Komodo IDE, Aptana Studio, PyScripter

Text Editors

Komodo Edit, Notepad++, JEdit, Vim, Emacs, Sublime Text

• Just make sure you have these configured so that Python indentation isn't a chore. Python doesn't like tab characters, and standard indentation is 4 spaces per indent. Here is an example of a possible Vim setup for working with Python.

Final setup steps for Django

Required files that aren't in the Git repository

The following files aren't in the Git repository, but must be created on each machine running CoralNet:

• settings_2.py: Create this on the project root level, and use settings_2_example.py as an example. This file is for Django settings that should differ between the production server and development machines, which is why it's not in the repository.
• images/task_helpers.py: Use images/task_helpers_example.py as an example. This is for calling system-specific external scripts/programs to perform certain asynchronous tasks.
• templates/maintenance_notice.html: Use templates/maintenance_notice_example.html as an example. This is for putting up a "under maintenance" message at the top of the website. maintenance_notice.html will be changed whenever site maintenance begins or ends, and these changes shouldn't be considered code updates, so the file isn't in the repository.
• Some directories and their sub-directories may need to be created prior to Django creating files under them. These directories include PROCESSING_ROOT, TEST_PROCESSING_ROOT, MEDIA_ROOT, TEST_MEDIA_ROOT, and STATIC_ROOT. An example of sub-directories that may need to be created: data/original/ under MEDIA_ROOT and TEST_MEDIA_ROOT. You'll know later if you need to create these directories if Django gets a "permission denied" error when trying to create files under them.

Creating the database tables

• Make sure you've installed South (the Python package) before proceeding.
• Make sure that in settings_2.py, the DATABASES setting's NAME, USER, and PASSWORD correspond to the database name, username, and user password that you created when configuring MySQL. Using the example given in the MySQL setup section, NAME would be 'coralnet', USER would be 'usernamehere', and PASSWORD would be 'passwordhere'. If you don't configure DATABASES correctly, you'll probably get some kind of "permission denied" error on the following steps.
  • You might also have to set the HOST as '127.0.0.1'.
• From the project root directory, run python manage.py syncdb to create the database tables that aren't under South control.
  • At this point, you'll probably be asked to create a superuser, which is a website user with all available privileges. Create one, and write down the username and password. Later, you can log in to your website as this superuser.
• Then run python manage.py migrate to create the database tables that are under South control. Note that this is done by creating some initial tables and then applying a bunch of changes to them (these changes corresponded to how CoralNet's database tables changed over time), so it can take a little while to complete.

Collecting static files

From the project root directory, run python manage.py collectstatic and reply 'yes' to the prompt. This will collect static files (CSS, Javascript, etc.) to the STATIC_ROOT directory. When DEBUG is False (this includes all runs of unit tests), static files must be collected to this directory in order to be served to the web browser.

For more info, see Django notes - Media and static files.

Running

Try running the server.

• From the project root directory, run python manage.py runserver and wait for the server to start. (Or run the server in your IDE.)
• Visit http://127.0.0.1:8000/ in a web browser. If things work, that should take you to the website homepage.
• Browse the website to see if the various pages work.
• Try creating labels, sources, labelsets, uploading images, and annotating images.

Try running some unit tests.

• For more info, see Django notes - Running unit tests.

Celery

TODO

• Setting up Celery probably won't be necessary immediately. It'll only be necessary if you are working on anything related to asynchronous tasks.