Merge pull request #1404 from open-data/i18n/pd-types

PD Type Translations

README.md

@@ -11,7 +11,7 @@ Features:
 Installation:
 
 * Use [open-data fork of CKAN](https://github.com/open-data/ckan),
-  branch canada-v2.8
+  branch canada-v2.9
 
 From a clean database you must run:
 
@@ -40,6 +40,10 @@ before loading any data.
   package processing between CKAN and Solr (requires
   `Scheming extension`, see below)
 
+`canada_security`
+  extra security processing (requires
+  `Security extension`, see below)
+
 ## Related Open Source Repos
 
 Project | Github group/repo | Our Contribution
@@ -64,14 +68,14 @@ The CKAN ini file needs the following settings for the registry server:
 ```ini
 ckan.plugins = dcat dcat_json_interface googleanalytics canada_forms canada_internal
         canada_public datastore recombinant
-        canada_datasets scheming_organizations fluent
+        canada_datasets scheming_organizations canada_security fluent
 ```
 
 For the public server use only:
 
 ```ini
 ckan.plugins = dcat dcat_json_interface googleanalytics canada_forms
-        canada_public canada_datasets scheming_organizations fluent
+        canada_public canada_datasets scheming_organizations canada_security fluent
 
 canada.portal_url = http://myserver.com
 
@@ -149,17 +153,36 @@ You will need to rebuild your search index using:
 ckan search-index rebuild
 ```
 
-## Compiling the updated French localization strings
+## Localization strings
+
+To update strings in the translation files:
+
+```bash
+python setup.py extract_messages
+```
+
+Extract messages will gather `gettext` calls in Python, JS, and Jinja2 files. It will also use th custom PD extractor to get specific strings for the Recombinant YAML files.
+
+To update the English and French catalog files:
+
+```bash
+python setup.py update_catalog
+```
+
+This will update both English and French PO files. You will need to confirm that there are NO `fuzzy` translations in either of the PO files.
+
+After updating the PO files and ensuring that there are no fuzzies, you may commit the two PO files along with the POT file.
+
+### Compiling localization strings
 
 Each time you install or update this extension you need to install the
 updated translations by running:
 
 ```bash
-bin/build-combined-ckan-mo.sh
+python setup.py compile_catalog
 ```
 
-This script overwrites the ckan French translations by combining it with
-ours.
+
 
 ## Migrating Proactive Disclosure (recombinant) data
 

ckanext/canada/extract.py

@@ -0,0 +1,73 @@
+from yaml import load
+from yaml.loader import SafeLoader
+from six import string_types
+
+
+class SafeLineLoader(SafeLoader):
+    def __init__(self, stream):
+        super(SafeLineLoader, self).__init__(stream)
+        self.line_numbers = {}
+
+    def construct_scalar(self, node):
+        if node.value in self.line_numbers:
+            self.line_numbers[node.value].append(node.start_mark.line + 1)
+        else:
+            self.line_numbers[node.value] = [node.start_mark.line + 1]
+        return super(SafeLineLoader, self).construct_scalar(node)
+
+    def get_single_data(self):
+        data = super(SafeLineLoader, self).get_single_data()
+        return self.line_numbers, data
+
+
+def extract_pd(fileobj, keywords, comment_tags, options):
+    """Extract messages from XXX files.
+
+    :param fileobj: the file-like object the messages should be extracted
+                    from
+    :param keywords: a list of keywords (i.e. function names) that should
+                     be recognized as translation functions
+    :param comment_tags: a list of translator tags to search for and
+                         include in the results
+    :param options: a dictionary of additional options (optional)
+    :return: an iterator over ``(lineno, funcname, message, comments)``
+             tuples
+    :rtype: ``iterator``
+    """
+    encoding = options.get('encoding', 'utf-8')
+    line_numbers, chromo = load(fileobj.read().decode(encoding), Loader=SafeLineLoader)
+
+    pd_type = chromo.get('dataset_type', 'unknown')
+
+    # PD Type Title
+    title = chromo.get('title')
+    if isinstance(title, string_types):
+        for line_number in line_numbers.get(title, [0]):
+            yield (line_number, '', title, ['Title for PD Type: %s' % pd_type])
+
+    # PD Type Short Label
+    label = chromo.get('shortname')
+    if isinstance(label, string_types):
+        for line_number in line_numbers.get(label, [0]):
+            yield (line_number, '', label, ['Label for PD Type: %s' % pd_type])
+
+    # PD Type Description
+    notes = chromo.get('notes')
+    if isinstance(notes, string_types):
+        for line_number in line_numbers.get(notes, [0]):
+            yield (line_number, '', notes, ['Description for PD Type: %s' % pd_type])
+
+    # PD Type Resources
+    resources = chromo.get('resources', [])
+    for resource in resources:
+        resource_title = resource.get('title')  # resource title
+        if isinstance(resource_title, string_types):
+            for line_number in line_numbers.get(resource_title, [0]):
+                yield (line_number, '', resource_title, ['Resource Title for PD Type: %s' % pd_type])
+
+        resource_trigger_strings = resource.get('trigger_strings')  # resource sql error messages
+        if resource_trigger_strings:
+            for k, v in resource_trigger_strings.items():
+                if isinstance(v, string_types):
+                    for line_number in line_numbers.get(v, [0]):
+                        yield (line_number, '', v, ['SQL Trigger String for PD Type: %s' % pd_type])