Write pointer files: PREMIS-rw and METS validation #27

jrwdunham · 2017-09-27T18:39:05Z

Makes it easier to use mets-reader-writer to create Archivematica METS pointer files for compressed/encrypted Archival Information Packages (AIPs) by doing the following.

Adds a PREMIS reader/writer plugin (and supporting dependency injection infrastructure).
Adds METS validation, both via XMLSchema (.xsd) and Schematron.

See this draft Architectural Decision Record for unifying METS interactions in Archivematica.

The PREMIS-RW functionality introduced here has significant overlap with the work in PR #20, which introduces a different approach to PREMIS-related functionality. These two pull requests should be reconciled. This one has the benefit of providing a plugin/dependency injection system. It should be easy to modify the PREMIS functionality of PR #20 (or that of uchicagoldr-premiswork) to work with this dependency injection system. See issue #28.

These changes were motivated by the desire to make it easier to read and write METS files, specifically Archivematica METS pointer files, in the context of the larger AIP replication project where interactions with METS files are increasingly being spread across several Archivematica-related tools/repos. Saliently, the Storage Service will now be responsible for creating pointer files. As a result, there is a greater need for establishing standard practices for interacting with METS files in Archivematica-related work. It is hoped that the changes introduced here can help to achieve that goal. See DRAFT Architectural Decision Record (ADR) 1: Unification of METS Creation in Archivematica
.

jrwdunham

See #28

jrwdunham · 2017-09-30T03:42:36Z

metsrw/fsentry.py

@@ -285,7 +309,9 @@ def serialize_structmap(self, recurse=True):
        """
        if not self.label:
            return None
-        el = etree.Element(utils.lxmlns('mets') + 'div', TYPE=self.type, LABEL=self.label)
+        el = etree.Element(utils.lxmlns('mets') + 'div',
+                           TYPE=self.mets_div_type or self.type)


This seems redundant, given how self.mets_div_type is set during initialization.

jrwdunham · 2017-09-30T03:43:57Z

metsrw/mets.py

+                file_id_prefix = utils.FILE_ID_PREFIX
+                # If the file is an AIP, then its prefix is not "file-" but the
+                # name of the AIP. Therefore we need to get the extension-less
+                # basename of the AIP's path and remove its UUID suffix to ge


jrwdunham · 2017-09-30T03:44:15Z

metsrw/mets.py

+                # name of the AIP. Therefore we need to get the extension-less
+                # basename of the AIP's path and remove its UUID suffix to ge
+                # the prefix to remove from the FILEID attribute value.
+                if entry_type.lower() == 'archival information package':


this string should be a constant

jrwdunham · 2017-09-30T03:47:04Z

metsrw/plugins/premisrw/premis.py

+        >>> premis_obj = PREMISObject(
+            identifier_type='UUID',
+            identifier_value='9bf6bcf8-4d77-4623-a9fb-b703365d0ffe',
+            ...)


Docstring should make clear that the tuple passed to data kwarg is the source of truth for the PREMIS element, under both construction approaches.

jrwdunham · 2017-09-30T03:54:24Z

metsrw/plugins/premisrw/premis.py

+    @property
+    def attrs_to_paths(self):
+        """Return a dict that maps valid getter attributes to the simplified
+        XPaths needed to get the corresponding values from ``self.data``.


Improve docstring to explain what this property returns/generates. It analyzes self.schema and sets self._attrs_to_paths to a dict that maps implicit getters like 'agent_identifier_value' and 'identifier_value' to the XPaths implicit in self.schema, in this case the XPath 'agent/agent_identifier/agent_identifier_value'. That same path also implies the getters 'agent_identifier' and 'identifier', which both map to the XPath 'agent/agent_identifier' and which should return a tuple (or list thereof) instead of a string.

jrwdunham · 2017-10-02T04:42:08Z

metsrw/plugins/premisrw/premis.py

+    return tuple(ret)
+
+
+def premis(data):


The existence of this function is just confusing. _data_to_lxml_el should probably just be used directly throughout (and made "public").

jrwdunham · 2017-10-02T04:43:23Z

metsrw/plugins/premisrw/premis.py

+
+
+def data_find(data, path):
+    """Find and return the element-as-tuple in tuple ``data`` using simplified


Find and return the first element-as-tuple ...

jrwdunham · 2017-10-02T04:45:10Z

metsrw/plugins/premisrw/premis.py

+    return data_find_all(data, path)
+
+
+def get_event_type(data):


Is this ever used?

jrwdunham · 2017-10-02T04:50:58Z

metsrw/plugins/premisrw/premis.py

+
+def get_attrs_to_paths(data, attrs_to_paths=None, path=None):
+    """Analyze PREMIS-element-as-tuple ``data`` and return a dict that maps
+    attribute names to the simplified XPaths needed to retrieve them, e.g.,::


The first positional argument here should be schema not data. data is a relic from the previous implementation.

jrwdunham · 2017-10-02T04:54:29Z

tests/test_mets.py

@@ -379,4 +382,213 @@ def test_full_mets(self):
        mw.append_file(sip)
        mw.write('full_metsrw.xml', fully_qualified=True, pretty_print=True)

+        # TODO: this should be passing.


Deal with this.

sevein

I've learned quite a lot reading your code. It's very smart!

I would have liked to see more than one PR though, e.g. validation was its own thing and I think the dependency injection could also be a PR alone easily.

sevein · 2017-10-24T20:54:47Z

tests/test_mets.py

 import os
+import pprint


sevein · 2017-10-24T20:59:14Z

tests/test_mets.py

@@ -378,5 +381,201 @@ def test_full_mets(self):

        mw.append_file(sip)
        mw.write('full_metsrw.xml', fully_qualified=True, pretty_print=True)
+        self.assert_mets_valid(mw.serialize())


In fafaf71 this fails but you seem to have deleted it later.

Yes, it fails because the METS file in this test is not really valid (according to the schematron file) because it contains fake metadata elements such as <premis>agent</premis>. The validation validates not only the METS but also the embedded PREMIS elements.

sevein · 2017-10-24T21:05:21Z

metsrw/fsentry.py

+    # Dependencies that must be injected. This means that an
+    # ``FSEntry`` instance must be able to call ``self.premis_object_class`` and
+    # get a class with methods ``fromtree`` and ``serialize``.
+    premis_object_class = Dependency(


sevein · 2017-10-24T21:06:18Z

tests/test_mets.py

@@ -381,10 +381,6 @@ def test_full_mets(self):

        mw.append_file(sip)
        mw.write('full_metsrw.xml', fully_qualified=True, pretty_print=True)
-        self.assert_mets_valid(mw.serialize())


This was added in fafaf71 and deleted here. Why?

Yes, it fails because the METS file in this test is not really valid (according to the schematron file) because it contains fake metadata elements such as <premis>agent</premis>. The validation validates not only the METS but also the embedded PREMIS elements.

sevein · 2017-10-24T21:12:47Z

metsrw/plugins/premisrw/premis.py

+    'PREMIS:OBJECT': PREMISObject,
+    'PREMIS:EVENT': PREMISEvent,
+    'PREMIS:AGENT': PREMISAgent
+}


How is this used?

Yes, this should be removed I think. It's a vestige from the initial way I was doing the plugin system.

sevein · 2017-10-24T21:13:51Z

metsrw/plugins/premisrw/premis.py

+- PREMISAgent
+
+"""
+__metaclass__ = type


I've learned something new!

- Schematron files have been added to the distribution i.e., they are present in pip installs. - Validation reports synthesizing XMLSchema and Schematron reports can be generated. - Can perform XMLSchema (.xsd file) validation using the ``validate`` function. - Can perform Schematron validation using the XML files in metsrw/resources/. - Using schematron files created by copying and modifying https://github.com/artefactual/mets-validator/blob/master/src/main/resources/archivematica_mets_schematron.xml - Tests added, including two new METS files (AIP METS and pointer METS) as fixtures, which were created via a production AM qa/1.x deploy. - Encryption added to known premis event types in pointer file schematron. - Weakened assertions for Archivematica pointer METS files (which are different from Archivematica METS files). Notably, now only requiring that there be at least one PREMIS:AGENT for each PREMIS:EVENT, not two as previously. - Unrelated: ``metsrw::FSEntry`` can now create transformFile elements under mets:file.

The mets_div_type will override the type of the FSEntry in the structmap's <mets:div>'s TYPE attribute. This allows, for example, a METS pointer file to have a user-defined type for the AIP that is more specific than 'Archival Information Package'.

Adds a dependency injection system which allows metsrw classes (i.e., FSEntry) to explicitly declare their dependencies on classes (with a declared API) for reading and writing metadata elements, e.g., PREMIS. Also adds a PREMIS reader/writer plugin which contains classes that conform to the API required by FSEntry.

- F403 'from .premis import *' used; unable to detect undefined names - F405 'VAR' may be undefined, or defined from star imports

…und 0

The ``PREMISRW_PLUGINS`` was unused.

jrwdunham added the work-in-progress label Sep 27, 2017

jrwdunham changed the title ~~Dev/issue 10662 write pointer files~~ Write pointer files: PREMIS-rw and METS validation Sep 27, 2017

nickwilkinson requested a review from sevein October 2, 2017 18:56

jrwdunham mentioned this pull request Oct 2, 2017

Problem: API for defining metadata plugins is ill-defined #28

Closed

jrwdunham commented Oct 2, 2017

View reviewed changes

jrwdunham mentioned this pull request Oct 2, 2017

Mirror locations and AIP replicas artefactual/archivematica-storage-service#241

Merged

jrwdunham mentioned this pull request Oct 17, 2017

Problem: filesystem and database are not treated as injectable dependencies artefactual/archivematica-storage-service#249

Open

jrwdunham force-pushed the dev/issue-10662-write-pointer-files branch 3 times, most recently from cfbb162 to 9b071a8 Compare October 23, 2017 22:32

jrwdunham removed the work-in-progress label Oct 23, 2017

jrwdunham mentioned this pull request Oct 24, 2017

Add normative structmap and PREMIS 3 support #32

Merged

sevein approved these changes Oct 24, 2017

View reviewed changes

jrwdunham added 15 commits October 25, 2017 09:15

Flake8: fix E713 test for membership should be "not in"

7225b5c

Flake8: fix E201 whitespace after "("

c473e17

Flake8: fix E302 expected 2 blank lines, found 1

0228c5a

Flake8: fix E731 do not assign a lambda expression, use a def

6534a41

Flake8: fix E402 module level import not at top of file

5968bda

Flake8: fix F403 (star import) and related F405

672d494

- F403 'from .premis import *' used; unable to detect undefined names - F405 'VAR' may be undefined, or defined from star imports

Flake8: fix F401 "VAR" imported but unused

0767fb7

Flake8: fix E265 block comment should start with "# "

e35b236

Flake8: fix E301 expected 1 blank line, found 0

d4ec2a6

Flake8: fix E306 expected 1 blank line before a nested definition, fo…

4da05c4

…und 0

Flake8: fix E128 continuation line under-indented for visual indent

a64f319

PREMISRW: removed unused constant

5c2b832

The ``PREMISRW_PLUGINS`` was unused.

jrwdunham force-pushed the dev/issue-10662-write-pointer-files branch from 9b071a8 to 5c2b832 Compare October 25, 2017 17:25

sevein approved these changes Oct 25, 2017

View reviewed changes

jrwdunham merged commit 5c2b832 into master Oct 25, 2017

jrwdunham deleted the dev/issue-10662-write-pointer-files branch October 25, 2017 17:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write pointer files: PREMIS-rw and METS validation #27

Write pointer files: PREMIS-rw and METS validation #27

jrwdunham commented Sep 27, 2017 •

edited

Loading

jrwdunham left a comment

jrwdunham Sep 30, 2017

jrwdunham Sep 30, 2017

jrwdunham Sep 30, 2017

jrwdunham Sep 30, 2017

jrwdunham Sep 30, 2017

jrwdunham Oct 2, 2017

jrwdunham Oct 2, 2017

jrwdunham Oct 2, 2017

jrwdunham Oct 2, 2017

jrwdunham Oct 2, 2017

sevein left a comment

sevein Oct 24, 2017

sevein Oct 24, 2017

jrwdunham Oct 25, 2017

sevein Oct 24, 2017

sevein Oct 24, 2017

jrwdunham Oct 25, 2017

sevein Oct 24, 2017

jrwdunham Oct 24, 2017

sevein Oct 24, 2017



		def data_find(data, path):
		"""Find and return the element-as-tuple in tuple ``data`` using simplified

Write pointer files: PREMIS-rw and METS validation #27

Write pointer files: PREMIS-rw and METS validation #27

Conversation

jrwdunham commented Sep 27, 2017 • edited Loading

jrwdunham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sevein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrwdunham commented Sep 27, 2017 •

edited

Loading