This repository contains the Django project and applications for working with LGRs.
lgr_web
is the main Django project that ties everything together. It has the settings, main url routes, most of the templates as well as static resources like CSS, Javascript and images.
The following modules are Django apps for clarity but may depend on each other, therefore using them in another project would require some work:
lgr_advanced
contains the advanced tools to manipulate LGR. It is split into the following sub-modules:lgr_editor
contains the code related to the LGR web-editor.lgr_validator
contains the code related the label validation module.lgr_tools
contains the code related to all utilities: testing LGR and label sets, LGR comparisons, etc.
lgr_auth
contains the authentication part that is used by some apps.lgr_basic
contains the code related to the simple LGR interface.lgr_idn_table_review
contains the IDN table review tools. It is split into the following sub-modules:icann_tools
contains the IDN table ICANN review that launches a review on all the tables stored in IANA registry,idn_tool
contains a tool allowing reviewing IDN tables against the references LGRs managed by admin,
lgr_manage
contains the LGRs and users management part.lgr_models
contains common models for other applications.lgr_renderer
contains the code and templates used to generate the static exports of the LGR (HTML only for now).lgr_session
defines a session object that allows manipulating some session objects and accessing and displaying storage.lgr_tasks
provides asynchronous tasks management.lgr_utils
provides common shared utilities for lgr apps.
This toolset was implemented by Cofomo (formerly Viagenie) (Audric Schiltknecht, Julien Bernard, Guillaume Blanchet, Michel Bernier, David Drouin, Vincent Gonzalez and Marc Blanchet) and Wil Tan on an ICANN contract.
Copyright (c) 2015-2016 Internet Corporation for Assigned Names and Numbers (“ICANN”). All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of the ICANN nor the names of its contributors
may be used to endorse or promote products derived from this
software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY ICANN AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL ICANN OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- Operating system: Tested on Linux and Mac OS X
- Python >=3.6
- LibXML2 [MIT License] used by the lxml Python bindings
- ICU4C [ICU License]
- Python modules (listed in
etc/requirements.txt
, or dependencies thereof)- Django version 1.8.x [BSD License]
- django-widget-tweaks [MIT License]
- Celery [BSD License]
- lxml [BSD License]
- lgr-core [BSD License]
- picu [MIT/X License]
- munidata [BSD License]
- natsort [MIT License]
- django-redis-cache [BSD License]
- django-autocomplete-light [MIT License]
- django-cleanup [MIT License]
- django-celery-results [BSD License]
- django-celery-beat [BSD License]
- social-auth-app-django [BSD License]
- A redis server for cache and asynchronous computations
For documentation generation:
For testing
You need at least one version of ICU4C library installed.
If your distribution ships ICU, install it using the package manager. Otherwise, you can download the ICU library in various formats (source, binary packages for some distributions). The easiest way to get up and running is to use the binaries.
For example, to install ICU 52 (Unicode 6.3):
wget http://download.icu-project.org/files/icu4c/52.1/icu4c-52_1-RHEL6-x64.tgz
tar xvf icu4c-52_1-RHEL6-x64.tgz -C / --wildcards usr/local/lib/*.52*
Using Homebrew
brew install icu4c
If you wish to install a specific version, browse the commit history of the formula and install using raw URL of the file at the given version (commit hash).
For example, to install ICU 52.1 (Unicode 6.3):
brew install https://github.com/Homebrew/homebrew/raw/de95d0b8/Library/Formula/icu4c.rb
The package will be installed in /usr/local/Cellar/icu4c/
Once you have installed one or more versions of ICU4C, you need to configure the paths to them in Django.
The parameter is called UNICODE_DATABASES
in src/lgr_web/settings/local.py
.
If used Homebrew to install as above, the default paths for Unicode 6.3.0 might actually work.
We highly recommend using virtualenv and pip for development. If you don't have them, install them first before running the following commands.
On fresh check out, do the following to create a new virtualenv, install dependencies and configure the project:
$ ./bin/bootstrap
If the above did not return an error, the dependencies and basic configuration would have been set up.
Before working on the project each time, activate the virtualenv:
$ . ./venv/bin/activate
Start the development server:
$ (venv) python manage.py runserver
The Django project uses a composite settings module style such that the DJANGO_SETTINGS_MODULE
is lgr_web.settings
.
To customize the settings, edit src/lgr_web/settings/local.py
which overrides any value defined in default.py
(found in the same directory).
Most uploaded LGR files are stored as session data. See Django documentation on
configuring the session engine.
The default setting is to use the configured database, which defaults to sqlite
.
For example, to use files as a storage mechanism, the following can be added to src/lgr_web/settings/local.py
:
SESSION_ENGINE = 'django.contrib.sessions.backends.file'
Tools also produce result files which are stored on the server.
Both the session engine and result files may need a periodical cleaning. Consider calling periodically the following command to clear expired data:
$ (venv) python manage.py cleanstorage
User accounts and files uploaded in IDN admin mode are stored in a database. See the official documentation to configure your database.
Default database is sqlite
, to use MariaDB, install mysqlclient
:
$ (venv) pip install mysqlclient
And set the following settings:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'lgr',
'USER': 'lgr',
'PASSWORD': '1234',
'HOST': 'localhost',
'PORT': '',
}
}
In order to set up the database for the above configuration, do the following:
$ mysql -u root -p
MariaDB [(none)]> CREATE DATABASE lgr CHARACTER SET UTF8;
MariaDB [(none)]> CREATE USER lgr@localhost IDENTIFIED BY '1234';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON lgr.* TO lgr@localhost;
MariaDB [(none)]> FLUSH PRIVILEGES;
Celery is used to handle asynchronous processing for long operations in background. A broker is used to transport messages. See Celery documentation on broker configuration. The default broker used is the redis broker.
To launch celery, in a venv-enabled console:
$ (venv) ./venv/bin/celery --app=lgr_web --workdir=./src worker --concurrency=2
Celery beat is used to trigger Celery periodic tasks. The task schedule is configured in the project settings:
CELERYBEAT_SCHEDULE = {
"calculate_index_variant_labels_tlds": {
"task": "lgr_tasks.tasks.calculate_index_variant_labels_tlds",
"schedule": 3600 * 24,
}
}
CELERYBEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
The beat scheduler can be launched, in a venv-enabled console:
$ (venv) ./venv/bin/celery --app=lgr_web --workdir=./src beat
Some notes for developers
- The
manage.py
script adds thesrc
directory to the Pythonsys.path
.
src/lgr_icann/templates
contains ICANN staff tools specific templatessrc/lgr_idn_table/templates
contains common templates for IDN review apps and IDN tool specific templatessrc/lgr_manage/templates
contains admin tools specific templatessrc/lgr_web/templates
contains the base templatessrc/lgr_auth/templates
contains authentication templatessrc/lgr_basic/templates
contains simple interface templatessrc/lgr_renderer/templates
contains LGR renderers (HTML output) templatessrc/lgr_tasks/templates
contains tasks monitoring templatessrc/lgr_advanced/templates
contains common template for LGR advanced appssrc/lgr_advanced/lgr_validator/templates
contains LGR validator specific templatessrc/lgr_advanced/lgr_tools/templates
contains LGR tools specific templatessrc/lgr_advanced/lgr_editor/templates
contains LGR editor specific templates
The static assets are stored in the following directories, using conventions supported by Django static files mechanism.
src/lgr_web/assets
src/lgr_advanced/lgr_editor/static
ICU4C is used to provide Unicode database such as character names, age and other properties.
The UNICODE_DATABASES
setting variable tells munidata how to instantiate the implementation
of each Unicode version that we support.
Keys are the Unicode version, like '6.3.0' Values are a dict of the kwargs to pass to the munidata.manager.register
function along with the version.
Django comes with a management tool that we can use to create users with admin role.
$ (venv) ./manage.py createsuperuser [email protected]
Please see Installation_Guide
To generate the documentation, go to the doc
directory and run the following command:
$ (venv) make html
The generated documentation is available in doc/_build/html/index.html
.
To generate and update translation for locales (1 and 2) run the following commands:
./manage.py makemessages -l <locale1> -l <locale2> -i venv
./manage.py compilemessages -l <locale1> -l <locale2> -i venv