Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem: Storage service qa/0.x and metsrw-0.3.8 are incompatible #732

Closed
ross-spencer opened this issue Jun 7, 2019 · 5 comments
Closed

Problem: Storage service qa/0.x and metsrw-0.3.8 are incompatible #732

ross-spencer opened this issue Jun 7, 2019 · 5 comments

Comments

@ross-spencer
Copy link
Contributor

Expected behaviour

Pointer files can be created in the Storage Service using metsrw-0.3.8.

Current behaviour

There is an error in the Store AIP microservice when upgrading from 0.3.1 to 0.3.8.

Steps to reproduce

Modify the requirements in the storage service to metsre=0.3.8 and then run a trial transfer generating a compressed AIP. An error will be recorded in Store AIP:

storeaip_v0.0: WARNING   2019-06-07 15:42:55,533  archivematica.mcp.client.storeAIP.store_aip:245  SIP creation failed: Failure storing file: <type 'exceptions.AttributeError'>: Instance of <class 'metsrw.plugins.premisrw.premis.PREMISEvent'> has no attribute compression_details. Valid attributes are
date_time
detail
detail_extension
event_date_time
event_detail
event_detail_extension
event_detail_information
event_detail_information__event_detail
event_detail_information__event_detail_extension
event_identifier
event_identifier__event_identifier_type
event_identifier__event_identifier_value
event_identifier_type
event_identifier_value
event_outcome
event_outcome_detail
event_outcome_detail_note
event_outcome_information
event_outcome_information__event_outcome
event_outcome_information__event_outcome_detail
event_outcome_information__event_outcome_detail__event_outcome_detail_note
event_type
identifier_type
identifier_value
linking_agent_identifier
linking_agent_identifier__linking_agent_identifier_type
linking_agent_identifier__linking_agent_identifier_value
linking_agent_identifier_type
linking_agent_identifier_value
outcome
outcome_detail_note
schema_location
type
version
xsi_schema_location.
SIP creation failed: Failure storing file: <type 'exceptions.AttributeError'>: Instance of <class 'metsrw.plugins.premisrw.premis.PREMISEvent'> has no attribute compression_details. Valid attributes are
date_time
detail
detail_extension
event_date_time
event_detail
event_detail_extension
event_detail_information
event_detail_information__event_detail
event_detail_information__event_detail_extension
event_identifier
event_identifier__event_identifier_type
event_identifier__event_identifier_value
event_identifier_type
event_identifier_value
event_outcome
event_outcome_detail
event_outcome_detail_note
event_outcome_information
event_outcome_information__event_outcome
event_outcome_information__event_outcome_detail
event_outcome_information__event_outcome_detail__event_outcome_detail_note
event_type
identifier_type
identifier_value
linking_agent_identifier
linking_agent_identifier__linking_agent_identifier_type
linking_agent_identifier__linking_agent_identifier_value
linking_agent_identifier_type
linking_agent_identifier_value
outcome
outcome_detail_note
schema_location
type
version
xsi_schema_location. See logs for more details.Traceback (most recent call last):
  File "/src/MCPClient/lib/job.py", line 111, in JobContext
    yield
  File "/src/MCPClient/lib/clientScripts/store_aip.py", line 353, in call
    args.sip_type,
  File "/src/MCPClient/lib/clientScripts/store_aip.py", line 246, in store_aip
    raise Exception(errmsg + " See logs for more details.")
Exception: SIP creation failed: Failure storing file: <type 'exceptions.AttributeError'>: Instance of <class 'metsrw.plugins.premisrw.premis.PREMISEvent'> has no attribute compression_details. Valid attributes are
date_time
detail
detail_extension
event_date_time
event_detail
event_detail_extension
event_detail_information
event_detail_information__event_detail
event_detail_information__event_detail_extension
event_identifier
event_identifier__event_identifier_type
event_identifier__event_identifier_value
event_identifier_type
event_identifier_value
event_outcome
event_outcome_detail
event_outcome_detail_note
event_outcome_information
event_outcome_information__event_outcome
event_outcome_information__event_outcome_detail
event_outcome_information__event_outcome_detail__event_outcome_detail_note
event_type
identifier_type
identifier_value
linking_agent_identifier
linking_agent_identifier__linking_agent_identifier_type
linking_agent_identifier__linking_agent_identifier_value
linking_agent_identifier_type
linking_agent_identifier_value
outcome
outcome_detail_note
schema_location
type
version
xsi_schema_location. See logs for more details.

Your environment (version of Archivematica, OS version, etc)

Docker compose, qa/0.x, 5fb80c554d3685dd3a81d182d64e70092ddcaa13 with metsrw manually upgraded to 0.3.8 (couldn't quite get pip-tools to work)

ross-spencer@artefactual:~/git/artefactual-labs/am/compose$ sudo docker-compose exec --user=root archivematica-storage-service bash
[sudo] password for ross-spencer: 
root@716e2a0d74d3:/src/storage_service# python
Python 2.7.16 (default, Mar 27 2019, 09:43:28) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import metsrw
>>> print metsrw.__version__
0.3.8

For Artefactual use:
Please make sure these steps are taken before moving this issue from Review to Verified in Waffle:

  • All PRs related to this issue are properly linked 👍
  • All PRs related to this issue have been merged 👍
  • Test plan for this issue has been implemented and passed 👍
  • Documentation regarding this issue has been written and it has been added to the release notes, if needed 👍
@ross-spencer
Copy link
Contributor Author

NB This is currently a blocker for #660

@ross-spencer ross-spencer mentioned this issue Jun 7, 2019
Closed
@ross-spencer
Copy link
Contributor Author

@cole I see you're upgrading metsrw for https://github.com/artefactual/archivematica-storage-service/pull/466/files have you spotted the same thing in your work?

@cole
Copy link

cole commented Jun 7, 2019

@ross-spencer I think this should be fixed now with the merging of artefactual/archivematica#1432, if you update Archivematica. It is a very strange error with a misleading message, but hopefully I can provide a bit of explanation...

It seems that the store_aip client script in MCPClient sends a number of PREMIS events to SS via the API, for inclusion in the pointer METS. It looks like in the qa/1.x branch of Archivematica, when we updated to PREMIS 3, we moved the version of these events as well, but didn't add the required eventDetailInformation wrapper, so we were sending invalid PREMIS 3 events over the wire to SS.

Back in SS-land, metsrw is trying to parse compression information out of event details, but metsrw 0.3.1 isn't aware of the eventDetail changes in PREMIS 3. Support was added in 0.3.7. However, the lack of support in 0.3.1 worked fine with the invalid PREMIS events coming from Archivematica, since they were still using the PREMIS 2.2 format. When updating SS to 0.3.8, metsrw sees that the event PREMIS version is 3, and looks for the eventDetailInformation wrapper, which isn't present, so it raises an AttributeError.

To add some extra confusion, metsrw implements a __getattr__ method, which is called when an attribute can't be found. That method is returning the exception you're seeing, which references compression_details, but actually the original error comes via a missing event_detail_information dynamic attribute, accessed from parsed_event_detail, which is accessed from compression_details. It would be nice if there was a way to make __getattr__ raise the original exception, but I'm not sure we really can without removing the dynamic attribute functionality.

Hopefully that is somewhat clear 😁 TLDR; try latest Archivematica qa/1.x + metsrw 0.3.8.

@ross-spencer
Copy link
Contributor Author

Thanks @cole 👍 my branch was further behind than I thought. Updating seems to have done the trick.

@ross-spencer
Copy link
Contributor Author

Indeed, closing this ticket as a result!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cole @ross-spencer