Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New volumes aren't working (tasks aren't available) #843

Closed
mialondon opened this issue Dec 19, 2018 · 34 comments
Closed

New volumes aren't working (tasks aren't available) #843

mialondon opened this issue Dec 19, 2018 · 34 comments
Labels
priority.high
Milestone
Jan 2019 fixes

Comments

@mialondon
Copy link
Member

Reported by @christianalgar

I have added four volumes today:

168 – 2 vols - A collection of playbills from miscellaneous London theatres 1807-1857.
[London : s.n.], 1807-1857.

285 – 2 vols - A collection of playbills from miscellaneous theatres: Stroud - Tullamore 1788-1848. – Swansea; Taunton; Tiverton; Teignmouth; Tewkesbury; Truro etc

I set up the following projects / tasks:

Mark Titles: Miscellaneous London theatres 1807-1857 (Vol. 1)
Mark Titles: Miscellaneous London theatres 1807-1857 (Vol. 2)
Transcribe Dates: Miscellaneous London theatres 1807-1857 (Vol. 1)
Transcribe Dates: Miscellaneous theatres: Stroud - Tullamore 1788-1848 (Vol. 1)
Transcribe Dates: Miscellaneous theatres: Stroud - Tullamore 1788-1848 (Vol. 2)
Transcribe Dates: Miscellaneous London theatres 1807-1857 (Vol. 2)

No projects are currently available to for ‘Transcribe Titles’ projects – Templates appear only for Transcribe dates / Transcribe genres / Mark Titles.

I used the same procedure to load all volumes and compose new projects but only one volume / task has succeeded: Transcribe Dates: Miscellaneous theatres: Stroud - Tullamore 1788-1848 (Vol. 2)

All other volumes launch the confetti ‘Hooray’ screen.

I received an email to confirm that the volume that is working had imported tasks successfully – see email below.
@mialondon
Copy link
Member Author

The email he received said: '241 new tasks were imported successfully to your project Transcribe Dates: Miscellaneous theatres: Stroud - Tullamore 1788-1848 (Vol. 2)!'

https://community.libcrowds.com/d/44-your-requests-for-help-questions-about-the-project/22 reports 'Sadly I get the confetti when I try to access any of the new miscellaneous London volumes.'

@mialondon
Copy link
Member Author

And my own notes from a while ago:

There are some strange issues on the playbills site at the moment, with a bunch of volumes giving the 'hooray' message when they've only just started. I can't seem to delete the 'broken' volumes so at the moment people are having a bad experience on the site, especially as the first two volumes are throwing that error.

Volumes not working for me (in early October):
• https://www.libcrowds.com/collection/playbills/projects/156/iiif-annotation
• https://www.libcrowds.com/collection/playbills/projects/154/iiif-annotation
• https://www.libcrowds.com/collection/playbills/projects/144/iiif-annotation Transcribe Titles: Miscellaneous Nottingham theatres 1806-1831
• https://www.libcrowds.com/collection/playbills/projects/141/iiif-annotation Transcribe Titles: New Theatre Royal, Hull 1810-1811
• https://www.libcrowds.com/collection/playbills/projects/131/iiif-annotation Mark Titles: Miscellaneous Birmingham theatres 1774-1800
• https://www.libcrowds.com/collection/playbills/projects/127/iiif-annotation Mark Titles: Portsmouth 1781-1782
• https://www.libcrowds.com/collection/playbills/projects/126/iiif-annotation Transcribe Dates: Chideock to Deptford 1790-1853 (Vol.1)
• https://www.libcrowds.com/collection/playbills/projects/124/iiif-annotation Transcribe Dates: Portsmouth 1781-1782
• https://www.libcrowds.com/collection/playbills/projects/122/iiif-annotation Transcribe Dates: Derby to Drogheda 1792-1852 (Vol. 2)
• https://www.libcrowds.com/collection/playbills/projects/121/iiif-annotation Transcribe Dates: Derby to Drogheda 1792-1852 (Vol. 1)
• I also get it for https://www.libcrowds.com/collection/playbills/projects/109/iiif-annotation https://www.libcrowds.com/collection/playbills/projects/101/iiif-annotation and https://www.libcrowds.com/collection/playbills/projects/96/iiif-annotation but it's possible that I really have completed all the tasks for those volumes as they're at 99% complete.

@mialondon mialondon added this to the Jan 2019 fixes milestone Jan 17, 2019
@mialondon
Copy link
Member Author

mialondon commented Feb 26, 2019 •
edited
Loading

Summary: volumes added to the site via the project admin interface appear to save successfully but volunteers can't access tasks on them.

Steps to reproduce:

  1. find a sample manifest to add (e.g. https://api.bl.uk/metadata/iiif/ark:/81055/vdc_100022588967.0x000002/manifest.json ) by searching for 'A collection of playbills' in the Catalogue box in the top right-hand corner of https://bl.uk
  2. add a new volume via https://www.libcrowds.com/admin/collection/playbills/volumes/new
    @christianalgar can you further describe or link to the process for adding a volume here?

Expected results: a new volume should be available on the projects page https://www.libcrowds.com/collection/playbills/projects. When a volunteer follows the link, they should have access to the task.

Actual result: When a volunteer follows the link, they get the 'confetti' message, 'Hooray! You have completed all available tasks for this project. As we have more than one person to complete each task to ensure high quality results, we still need more contributions before the project is marked as complete, so please spread the word!'

Further information

We will provide you with an admin account so you can test the process.

Volumes are listed at https://www.libcrowds.com/admin/collection/playbills/volumes

The 'hooray' screen might be called 'User completion' internally.

Sample screenshot
20190226 take part i libcrowds https___www libcrowds com_collection_playbills_projects

@geoffroy-noel-ddh
Copy link
Collaborator

geoffroy-noel-ddh commented Aug 22, 2019 •
edited
Loading

Hi Mia,

I have re-imported the project "Transcribe Dates: A collection of playbills from Haymarket Theatre 1781-1783". Please have a look on the live site and let me know if it now works as expected for you.

What I can see from my analysis of the database is that some projects didn't have any tasks and it seems to correlate with a missing manifest in the parent volumes. I'm not sure how this actually happened, whether it is an omission during creation or the admin interface being slightly buggy and loosing some inputs when things are not done in a proper sequence.

Here is how I managed to create a volume, a project and import the tasks:

  1. create a new volume in the admin by first selecting playbill collection then going to Volumes page and clicking the New button;
  2. fill in the form (Name, short name and importer = iiif) and submit
  3. go to the "Import Source" tab and insert the link to the manifest (please try that link in another browser tab first to make sure it actually returns a json file)
  4. click "update import source" button and wait for the success message
  5. Just to make sure, you can reload the page and go to the "Import Source" tab to verify that the value isn't empty (many problematic volumes do have an empty value)

6 . Create a new project in the usual way (/admin/project/new/playbills)
7. On the creation form select in the template (e.g. Dates), the newly created volume, ... and press Create
8. It should show a success message
9. After the screen reloads, tag the Task, Entity and Location then Click Update

After step 8, Libcrowds will create a background job of Medium priority to import the tasks from the manifest. This job is queued and can therefore take several minutes before it comes at the top. You should be able to see that job in the Admin:

  • select to "Background Tasks" on the left-hand side panel
  • look at the Queue table and check if you have a Medium Job,
  • it should say 1 (or more) if the import job was created, 0 if the import is already finished

You can also see those import tasks on your admin Dashboard, under the New Task section.

Could you try the above on a new volume and manifest and let me know how it goes? If it doesn't work please let me know at which point it failed and what you see on screen so I can try to reproduce?

@geoffroy-noel-ddh
Copy link
Collaborator

As you noted in your message above there is a bug in the admin with the removal of the Volumes that have no tasks. I have fixed that bug locally and will share that soon with you on the repository.

However when that bug is fixed, Libcrowds will show you a message telling you that you can't remove a Volume that contains Projects. So that won't help much. The good news is that you don't need to remove those Volumes.

Instead you can

  1. use Pybossa to remove the projects that have no task attached to them,
  2. then insert the missing manifest link in the Volume and
  3. recreate the project according to the instructions I left above.

Here's an example for this volume: "Miscellaneous London theatres 1807-1857 (Vol. 2)".

  1. removing the projects
  • go to https://backend.libcrowds.com/project/category/playbills/ (make sure you are logged-in)
  • in the project list find the date or title project for that volume and click "More info" to go to that project page (e.g. https://backend.libcrowds.com/project/mark_titles_miscellaneous_london_theatres_1807-1857_vol_2/)
  • on the left hand side menu, select "Settings"
  • on the right hand side, click the DELETE link to arrive on the deletion screen
  • now please verify that the name of the project is correct and also that it contains no task ("Overall progress: 0% completed", "Tasks (0)"); if you see anything else then you might not want to delete it.
  • if you are sure it contains no task, then click "Yes, delete it"
  • go back to the project list and remove the remaining projects for this volume
  1. when a volume has no project left you should be able to edit its Manifest URI field

Please let me know if you have any question.

@geoffroy-noel-ddh
Copy link
Collaborator

In case you get an error message about a missing column in the database when deleting a Project, you may want to ask a developer / sysadmin on your side to update the database schema according to the PyBossa documentation:

https://docs.pybossa.com/installation/guide/#updating-pybossa-core-and-migrating-the-database-table-structure

It is the following command:
alembic upgrade head

@geoffroy-noel-ddh
Copy link
Collaborator

Finally I'd strongly advise you to take a backup of your database before doing any project deletion or updating the database schema in case something doesn't work well. If you have any doubt you should try all those operations on your staging site first.

This was referenced Aug 22, 2019
Closed
Closed
geoffroy-noel-ddh added a commit that referenced this issue Sep 3, 2019
@geoffroy-noel-ddh
gh-843: minor fix, clicking the delete button on the volume listing p…
a83c50c 
…age was causing a javascript error. Cause: access to response.data.length instead of response.length.
geoffroy-noel-ddh referenced this issue Sep 3, 2019
@geoffroy-noel-ddh
Merge branch '843-new-volumes' into 2019-fixes
0b5e97f
@geoffroy-noel-ddh
Copy link
Collaborator

Just to confirm that the volume deletion bug in the LibCrowds admin as mentioned above is now fixed in the 2019-fixes branch.

@mialondon
Copy link
Member Author

Thanks @geoffroy-noel-ddh ! @harryjmoss - can you check these over ahead of deploying the changes?

@mialondon
Copy link
Member Author

@geoffroy-noel-ddh we've been working through your fixes (finally!).

Before we put the fixes on live, we wanted to check - is there some kind of filtering on dev that means that previous projects don't show in the admin or public interface? On playbills we can see the volumes we'd added or updated following your fixes; on convert-a-card we can see two volumes (if we play with the filter), but I'd expect to see more of the past projects (some of which would have been completed or in progress and some of which might be missing info).

@mialondon
Copy link
Member Author

Other notes

Adding a volume via your steps works, hoorah! We'll need to update the documentation to reflect this (there wasn't much before, which might be how steps were missed)

On live some buttons (navigating around items, submitting) seem to be offset but that's not an issue on dev so I assume updating will fix that

We couldn't update the 'filter' options for the existing volume we fixed on either dev or live. However, it does work on IE (!). In Chrome it does seem to save the changes but not update the drop-down to show the selected filter tag.

@mialondon
Copy link
Member Author

We're happy to push the changes to live if we can understand why previous projects aren't showing in dev.

@geoffroy-noel-ddh
Copy link
Collaborator

Hi @mialondon , glad you've managed to add new valid volumes to the system!

I also noticed the filter options bug in the admin form. My understanding is that it's a visual bug only, if you select the filters the drop downs look unchanged but the option you've selected are actually set and will be recorded when you submit the form. This is an existing bug, it was already there before we made any change to the code. Perhaps you could add this into a separate github issue and depending on the remaining budget we could add it to the milestone after the current set of changes have been released live and tested.

The issue with the filtering of the cards could also be placed in a separate github ticket if you don't mind as I don't think it is related to recent changes. This might be a pre-existing bug. On the live site I only see two projects in the list:

https://www.libcrowds.com/collection/convertacard/projects

Could you tell me which other volumes you'd expect to see in the list?

On annual leave next week but I could have a quick look at it when I'm back. However it would be easier if we had the same data on both side so it's easier for us to reproduce and investigate the bug. If this bug also currently occurs on the live site then that'd be easier for us as our database matches that one.

@mialondon
Copy link
Member Author

Hi @geoffroy-noel-ddh - thanks for getting back to us!

Yes, I don't think the filter drop-down issue is new. I wonder if there's something bespoke going on - Alex might have re-used the code for adding tags about images (which was definitely rushed and buggy).

I've asked my colleague if she knows how many projects were live on Convert-a-Card recently. Only two are still in progress on https://www.libcrowds.com/admin/project so that might well be fine.

@mialondon
Copy link
Member Author

It seems weird that the projects are all either 100% or 0%, but perhaps some people were still managing to get past the confetti message and the submit issue and they completed them over the past few months?

We're going to do some tasks to move the dial off 0% for the new / fixed projects.

@mialondon
Copy link
Member Author

The volume that @geoffroy-noel-ddh repaired in late August https://www.libcrowds.com/collection/playbills/projects/164/iiif-annotation (Transcribe Dates: A collection of playbills from Haymarket Theatre 1781-1783) was showing as 100% complete, which seemed a bit unlikely, but I downloaded the results from https://www.libcrowds.com/collection/playbills/data and what's in transcribe_dates_a_collection_of_playbills_from_haymarket_theatre_1781-1783_task_run.csv seems legit.

@christianalgar
Copy link

christianalgar commented Oct 4, 2019 via email •
edited
Loading

@mialondon
Copy link
Member Author

@christianalgar is that a confession?

@mialondon
Copy link
Member Author

Also, there was a bug that made hitting the 'submit' button hard except on certain browsers but that won't have affected everyone.

I'll take a quick look at the Google Analytics.

@mialondon
Copy link
Member Author

More visits than I'd have thought to the tasks page (i.e. URLs containing 'project')

Analytics All Web Site Data Overview 20190701-20191003.pdf

This was referenced Oct 4, 2019
Open
Open
@mialondon
Copy link
Member Author

Looks like it's working. Happy Friday, everyone!

@christianalgar
Copy link

christianalgar commented Oct 4, 2019 via email •
edited by mialondon
Loading

@mialondon
Copy link
Member Author

mialondon commented Oct 4, 2019 via email

@christianalgar
Copy link

christianalgar commented Oct 4, 2019 via email •
edited by mialondon
Loading

@mialondon mialondon reopened this Apr 16, 2020
@mialondon
Copy link
Member Author

@harryjmoss I've just started getting the dreaded confetti on all volumes, though they're new and I haven't even looked at most of them. @christianalgar had mentioned some issues in adding new volumes so I don't know if there's a glitch in how they went in, but when you have time could you investigate?

@mialondon
Copy link
Member Author

mialondon commented Apr 20, 2020 via email •
edited
Loading

@christianalgar
Copy link

christianalgar commented Apr 20, 2020 via email •
edited by mialondon
Loading

@mialondon
Copy link
Member Author

mialondon commented Apr 20, 2020 via email

@mialondon
Copy link
Member Author

I’ve added two new tasks, the first works, the second throws confetti.
Birmingham 1801-1805 vol 2 Mark Titles works https://www.libcrowds.com/collection/playbills/projects/203/iiif-annotation
York 1782-1849 vol 2 does not https://www.libcrowds.com/collection/playbills/projects/204/iiif-annotation

Both of these work for me on my home Mac.

The attached doc highlights the vols that are throwing confetti.

It looks like GitHub doesn't process email attachments, you'd need to email them directly or log into GitHub. (I was only doing it via email as I was on my phone at Barts while waiting for an appointment)

@mialondon
Copy link
Member Author

I’ve added two new tasks, the first works, the second throws confetti.
Birmingham 1801-1805 vol 2 Mark Titles works https://www.libcrowds.com/collection/playbills/projects/203/iiif-annotation
York 1782-1849 vol 2 does not https://www.libcrowds.com/collection/playbills/projects/204/iiif-annotation

Both of these work for me on my home Mac.

Mark Titles: A collection of playbills from Theatre Royal, York 1782-1849 (Vol. 1) https://www.libcrowds.com/collection/playbills/projects/196/iiif-annotation didn't work for me, while Vol 2 did...

@mialondon
Copy link
Member Author

Bouncing the process for reviewing broken volumes back up: #843 (comment)

@christianalgar
Copy link

I've deleted the confetti volumes. Thanks for leading me to the water, Mia : )
Adding more project tasks so we can let people know there's fresh material.

@mialondon
Copy link
Member Author

I'm glad it worked, and thanks for deciphering my very hurried comment!

@mialondon mialondon mentioned this issue Jun 23, 2020
Open
2 tasks
@mialondon
Copy link
Member Author

Closing this as we're taking up related issues in #862

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority.high
Projects
None yet
Milestone
Jan 2019 fixes
Development

No branches or pull requests
3 participants
@mialondon @geoffroy-noel-ddh @christianalgar