Merge branch 'master' of https://github.com/openoereb/pyramid_oereb i…

…nto 1392_testcov_interlis_23

doc/source/configuration.rst

@@ -3,112 +3,288 @@
 Configuration
 =============
 
-You are looking at a highly configurable piece of software. To get the right understanding of the server it
+You are looking at a highly configurable piece of software. To get a good understanding of the server it
 is recommended to read this part carefully.
 
-Since the swiss confederation's definition and the specification for the `OEREB Data Extract
-<https://www.cadastre.ch/content/cadastre-internet/de/manual-oereb/publication/publication.download
-/cadastre-internet/de/documents/oereb-weisungen/OEREB-Data-Extract_de.pdf>`__ is really straight,
-we had very narrow margins to develop the code. Using this pyramid plugin, you will get a running server
-providing the services to satisfy the federal specification. But to get this extract, you need to bind your
-data to this server. And this is basically what you need to configure.
+The specifications of the oereb data model, the web services, the data extracts and the print layout 
+by the Swiss Confederation are very precise `OEREB specifications
+<https://www.cadastre.ch/de/manual-oereb/publication/instruction.html>`__ . Code development was guided 
+by the exact implementation of the specifications and the user requirements. Using this pyramid plugin, 
+you will get a running server providing all the services defined by the federal specifications. 
+The binding of cantonal and federal data to the server is done by the configuration options.
 
-This section describes the different possibilities to adapt the application on different data structures or
-even custom data sources. If you are planning to use such modifications, we suggest to check all possible
+This section describes the different possibilities to adapt the application to work with various data structures or
+even custom data sources. If you are planning to implement such modifications, we suggest to check all possible
 solutions first, as the necessary effort can vary significantly depending on your specific needs.
 
+.. _configuration-initial-setup:
+
+Create the inital database setup
+--------------------------------
+
+Out of the box the pyramid_oereb server supports three different topic configurations in the database:
+
+  - the **pyramid_oereb standard model**
+  - the **interlis 2.3 OeREBKRM transfer model**
+  - the **OEREBlex model**
+
+Pyramid_oereb Standard Model
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This schema and table structure is based on the initial topic structure used in the pyramid_oereb
+v1.x versions. It's mainly used for cantonal topics which are not (yet) stored in the interlis 2.3 OeREBKRM-
+Transfer structure and by cantons that do not use OeREB-Lex to manage the legal documents.
+
+Interlis 2.3 OeREBKRM Transfer Model
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+All the federal data sets are provided in this data structure. So this is the schema and table model you
+want to use for all the federal topics unless you want to transform the data to a specific database structure.
+If your cantonal data is also stored based on this model, then you probably want to use this structure 
+for all topics to homogenize your database content.
+The `Ili2pg Oereb Data Import Manual <https://github.com/openoereb/ili2pg_oereb_data_import_manual>`__
+explains how to use the ili2pg tool to create the corresponding schema and how to import the XML data.
+
+
+OEREBlex Topic Model
+^^^^^^^^^^^^^^^^^^^^
+
+This third model is usefull if you maintain your legal documents using the OEREBlex application and you
+have a specific cantonal model for your data. It is similar to the pyramid_oereb standard model, but all
+the document related tables are omitted. Instead the documents are linked by the geolink attribute. 
 
 .. _configuration-additional-topics:
 
-Add additional topics
----------------------
+Add additional standard topics
+------------------------------
+
+If you like to add one or more additional topics based on the *pyramid_oereb standard* database structure
+you can use the internal command below creating an SQL script to establish the topic schema.
 
-If you only need to add one or more additional cantonal topics and want to use the standard database structure
-for these too, you can run a provided script to generate the required models.
+But before creating any new topic structure you have to add its configuration. Open the configuration file
+(pyramid_oereb.yml) and copy the section from one of the existing **standard** topics which usually 
+looks like this:
+
+.. code-block:: yaml
+
+    - code: ch.NE.Baulinien
+      geometry_type: LINESTRING
+      thresholds:
+        length:
+          limit: 1.0
+        area:
+          limit: 1.0
+      language: fr
+      federal: false
+      view_service:
+        layer_index: 1
+        layer_opacity: 0.75
+      source:
+        class: pyramid_oereb.contrib.data_sources.standard.sources.plr.DatabaseSource
+        params:
+          db_connection: *main_db_connection
+          # model_factory: pyramid_oereb.contrib.data_sources.standard.models.theme.model_factory_integer_pk
+          # uncomment line above and comment line below to use integer type for primary keys
+          model_factory: pyramid_oereb.contrib.data_sources.standard.models.theme.model_factory_string_pk
+          schema_name: road_building_lines
+      hooks:
+        get_symbol: pyramid_oereb.contrib.data_sources.standard.hook_methods.get_symbol
+        get_symbol_ref: pyramid_oereb.core.hook_methods.get_symbol_ref
+      law_status_lookup:
+        - data_code: inKraft
+          transfer_code: inKraft
+          extract_code: inForce
+        - data_code: AenderungMitVorwirkung
+          transfer_code: AenderungMitVorwirkung
+          extract_code: changeWithPreEffect
+        - data_code: AenderungOhneVorwirkung
+          transfer_code: AenderungOhneVorwirkung
+          extract_code: changeWithoutPreEffect
+      document_types_lookup:
+        - data_code: LegalProvision
+          transfer_code: Rechtsvorschrift
+          extract_code: LegalProvision
+        - data_code: Law
+          transfer_code: GesetzlicheGrundlage
+          extract_code: Law
+        - data_code: Hint
+          transfer_code: Hinweis
+          extract_code: Hint
+
+Apply the necessary modifications/replacements for the new topic. This should at least be the schema name, 
+code, geometry type and of course the models property within the source parameters:
+Make sure that this source class is `pyramid_oereb.contrib.data_sources.*standard*.sources.plr.DatabaseSource`
+and not interlis_2_3. - The same goes for the model_factory and the get_symbol element. It should be set to
+*standard*.
+
+Also set the language of the data and if it's a federal (true) or cantonal topic (false). You also want to
+define what lookup codes are for the law_status and document types.
+
+Once the configuration set, run the following command:
 
 .. code-block:: shell
 
-   create_standard_model -c <YOUR_NEW_TOPIC_CODE> -g <GEOMETRY_TYPE> -p <TARGET_PATH> -k TRUE
+   create_standard_tables -c <YOUR_YAML_CONFIGURATION> -T [flag used to skip schema creation] 
+    --sql-file=<PATH_AND_SQL_SCRIPTNAME> -w [to over-write existing sql instead of append]
 
-The first parameter is the code of your new topic and has to be defined in camelcase. The geometry type for
-the theme can be one of the following:
+The first parameter ``-c or --configuration=YAML`` is the path to your YAML configuration file. 
+By default it's *pyramid_oereb.yml*
 
-   - `POINT`,
-   - `LINESTRING`,
-   - `POLYGON`
-   - `MULTIPOINT`
-   - `MULTILINESTRING`
-   - `MULTIPOLYGON` or
-   - `GEOMETRYCOLLECTION`.
+The second optional parameter ``-s or --section=SECTION`` allows you to specify the section containing
+the configuration part to use. Default is *pyramid_oereb*.
 
-The geometry collection is only meant for topics that can consist of different geometry types. We **strongly**
-recommend to use it **only if a simple geometry cannot be used** and to put **only one geometry** in each
-record. Do not use it as a bucket for everything! The third parameter defines the output directory in which
-the new file will be created, e.g. a module directory of your application.
+The parameter ``-T or --tables-only`` skips the schema creation and creates only the tables.
 
-To use strings as primary key type, add the '-k TRUE' parameter. If not supplied, the default primary key type
-of integer will be used instead.
+The option ``--sql-file=SQL_FILE`` generates an SQL file containing the schema and table creation 
+commands. *SQL_FILE* should be the name or the absolute path of the file. E.g: my_sql_script.sql
 
-After creating the models you have to create its configuration. Open the configuration file
-(pyramid_oereb_standard.yml) and copy the section from one of the existing topics which usually looks like
-this:
+If your yaml file uses the c2ctemplate style (starting with vars) you need to add the
+``--c2ctemplate-style`` parameter.
 
-.. code-block:: yaml
+The option ``-w or --over-write`` allows you to overwrite an existing sql file. Default is append.
 
-   - code: ch.BaulinienNationalstrassen
-     geometry_type: LINESTRING
-     thresholds:
-       length:
-         limit: 1.0
-         unit: 'm'
-         precision: 2
-       area:
-         limit: 1.0
-         unit: 'm2'
-         precision: 2
-       percentage:
-         precision: 1
-     text:
-       de: Baulinien Nationalstrassen
-     language: de
-     federal: true
-     standard: true
-     source:
-       class: pyramid_oereb.lib.sources.plr.DatabaseSource
-       params:
-         db_connection: postgresql://postgres:password@localhost:5432/pyramid_oereb
-         models: pyramid_oereb.standard.models.motorways_building_lines
-     get_symbol_method: pyramid_oereb.standard.methods.get_symbol
+Now you have set up an empty additional topic in your database and you can proceed with deploying 
+your data into it.
 
+Add additional interlis topics
+------------------------------
 
-Apply the necessary modifications for the new topic. This should at least be the the name, code and text
-definitions and of course the models property within the source parameters. It should point to the module
-with the generated models of the former step.
+Follow the `Ili2pg Oereb Data Import Manual <https://github.com/openoereb/ili2pg_oereb_data_import_manual>`__
+to create a new topic schema based on the OeREBKRM Transfer model and about how to import the XML data.
 
-Now you have set up an empty additional topic and you can continued with deploying your data into it.
+Once the schema is created do not forget to add the corresponding topic configuration in the *pyramid_oereb.yml*
 
+.. code-block:: yaml
 
-Add an oereblex topic
+    - code: ch.BaulinienNationalstrassen
+      geometry_type: LINESTRING
+      thresholds:
+        length:
+          limit: 1.0
+          unit: 'm'
+          precision: 2
+        area:
+          limit: 1.0
+          unit: 'm²'
+          precision: 2
+        percentage:
+          precision: 1
+      language: de
+      federal: true
+      view_service:
+        layer_index: 1
+        layer_opacity: 0.75
+      source:
+        class: pyramid_oereb.contrib.data_sources.interlis_2_3.sources.plr.DatabaseSource
+        params:
+          db_connection: *main_db_connection
+          # model_factory: pyramid_oereb.contrib.data_sources.standard.models.theme.model_factory_integer_pk
+          # uncomment line above and comment line below to use integer type for primary keys
+          model_factory: pyramid_oereb.contrib.data_sources.interlis_2_3.models.theme.model_factory_integer_pk
+          schema_name: motorways_building_lines
+      hooks:
+        get_symbol: pyramid_oereb.contrib.data_sources.interlis_2_3.hook_methods.get_symbol
+        get_symbol_ref: pyramid_oereb.core.hook_methods.get_symbol_ref
+      law_status_lookup:
+        - data_code: inKraft
+          transfer_code: inKraft
+          extract_code: inForce
+        - data_code: AenderungMitVorwirkung
+          transfer_code: AenderungMitVorwirkung
+          extract_code: changeWithPreEffect
+        - data_code: AenderungOhneVorwirkung
+          transfer_code: AenderungOhneVorwirkung
+          extract_code: changeWithoutPreEffect
+      document_types_lookup:
+        - data_code: Rechtsvorschrift
+          transfer_code: Rechtsvorschrift
+          extract_code: LegalProvision
+        - data_code: GesetzlicheGrundlage
+          transfer_code: GesetzlicheGrundlage
+          extract_code: Law
+        - data_code: Hinweis
+          transfer_code: Hinweis
+          extract_code: Hint
+
+Make sure to set the schema name to the one you defined using ili2pg, also set code, geometry type
+and of course the models property within the source parameters:
+Here the source class is `pyramid_oereb.contrib.data_sources.*interlis_2_3*.sources.plr.DatabaseSource`
+and not standard. - The same goes for the model_factory and the get_symbol element. It should be set to
+*interlis_2_3*.
+
+Also define the language of the data and if it's a federal (true) or cantonal topic (false). You also want to
+define that it is *NOT* the standard structure (false) and what lookup codes are used for the law_status 
+and document types.
+
+Add an OEREBLex Topic
 ---------------------
 
-If you want to use oereblex for a topic, you can proceed as described in the previous section,
+If you want to use the OEREBlex structure for a topic, you can proceed as described in the previous section,
 but using a different script to generate the required models.
 
 .. code-block:: shell
 
-   create_oereblex_model -c <YOUR_NEW_TOPIC_CODE> -g <GEOMETRY_TYPE> -p <TARGET_PATH> -k TRUE
+   create_oereblex_tables -c <YOUR_NEW_TOPIC_CODE> -g <GEOMETRY_TYPE> -p <TARGET_PATH> -k TRUE
+
+For all topics
+--------------
 
+Do not forget to add the availability information in the *pyramid_oereb_main.availability* table to activate (or not)
+the topic for a municipality.
 
 .. _configuration-adapt-models:
 
 Adapt existing models
 ---------------------
 
 Another option to modify the standard configuration, is to adapt the existing models to fit another database
-structure. This method is recommended if you are using an existing database supported by GeoAlchemy 2 and
+structure. This method is recommended if you are using an existing database supported by GeoAlchemy2 and
 already containing all the necessary data but in a different structure. In this case you should check, if it
 is possible to transform the data by extending the existing models with a mapping to fit your structure.
 
+The easiest example is a simple mapping of table and column names, if you use a different language. Using the
+possibilities of SQLAlchemy, you could extend the existing
+pyramid_oereb.core.models.motorways_building_lines.office
+:ref:`api-pyramid_oereb-core-models-motorways_building_lines-office` like this:
+
+.. code-block:: python
+
+   from pyramid_oereb.lib.standard.models import motorways_building_lines
+
+   class Office(motorways_building_lines.Office):
+       """
+       The bucket to fill in all the offices you need to reference from public law restriction,
+       document, geometry.
+
+       Attributes:
+           id (int): The identifier. This is used in the database only and must not be set manually.
+               If you don't like it - don't care about.
+           name (dict): The multilingual name of the office.
+           office_at_web (str): A web accessible url to a presentation of this office.
+           uid (str): The uid of this office from https
+           line1 (str): The first address line for this office.
+           line2 (str): The second address line for this office.
+           street (str): The streets name of the offices address.
+           number (str): The number on street.
+           postal_code (int): The ZIP-code.
+           city (str): The name of the city.
+       """
+       __table_args__ = {'schema': 'baulinien_nationalstrassen'}
+       __tablename__ = 'amt'
+       id = sa.Column('oid', sa.Integer, primary_key=True)
+       office_at_web = sa.Column('amt_im_web', sa.String, nullable=True)
+       line1 = sa.Column('zeile1', sa.String, nullable=True)
+       line2 = sa.Column('zeile2', sa.String, nullable=True)
+       street = sa.Column('strasse', sa.String, nullable=True)
+       number = sa.Column('hausnr', sa.String, nullable=True)
+       postal_code = sa.Column('plz', sa.Integer, nullable=True)
+       city = sa.Column('ort', sa.String, nullable=True)
+
+       (...)
+
+The only thing, you have to care about, if you want to stay using the standard sources, is to keep the class
+name, the names of the properties and their data types.
+
 After extending the models, do not forget to change the models module in the configuration of the topic's
 source.
 

doc/source/index.rst

@@ -54,7 +54,10 @@ API). These need to be configured and prepared by the integrator of this package
 of documentation.
 
 For the moment we provide an adapter to get the PDF static extract from mapfishprint and
-`oereb_xml_pdf_service <https://github.com/Geocloud-AG/oereb_xml_pdf_service>`__.
+
+=======
+For the moment we provide an adapter to get the PDF static extract from mapfishprint and
+`xml2pdf <https://github.com/Geocloud-AG/oereb_xml_pdf_service>`__.
 
 .. image:: ../images/overview.png
    :align: center

pyramid_oereb/contrib/data_sources/oereblex/sources/document.py

@@ -199,12 +199,16 @@ def _get_document_records(self, document, language):
             raise AssertionError('Missing authority for document #{0}'.format(document.id))
         if oereblex_doctype is None:
             raise AssertionError('Missing type for document #{0}'.format(document.id))
+        if document.authority_url is None:
+            authority_url = None
+        else:
+            authority_url = {language: document.authority_url}
 
         # Get document type
         document_type = Config.get_document_type_by_data_code(self._code, oereblex_doctype)
 
         # Create related office record
-        office = OfficeRecord({language: authority}, office_at_web=document.authority_url)
+        office = OfficeRecord({language: authority}, office_at_web=authority_url)
 
         # Get files
         records = []

tests/core/sources/test_document_oereblex.py

@@ -297,6 +297,9 @@ def test_read_related_notice_as_main(oereblex_test_config):
         assert isinstance(document.responsible_office, OfficeRecord)
         assert document.responsible_office.name == {'de': '-'}
         assert document.responsible_office.office_at_web is None
+        assert source.records[0].responsible_office.office_at_web == {
+            'de': "http%3A%2F%2Fwww.zihlschlacht-sitterdorf.ch"
+        }
         assert document.published_from == datetime.date(1970, 1, 1)
         assert document.index == 40