Action items for submission

[Benjamin-Lee]

Hi everyone!

I hope everyone has been staying safe and productive these last few months. It's certainly been adventure for me as I have graduated college and started on my doctorate over the last six months. We're just shy of the two year anniversary of this project, which is a bittersweet milestone. While I'm glad that this project has continued making progress, it has also taken longer than anticipated. I bear the fault and for that I apologize.

I'm aiming to get this manuscript over the finish line as soon as possible, since what we need is momentum rather than even more time. Here are the steps (in approximate order) I propose we follow to get the manuscript submitted within a month:

• Finish reviewing PRs for minor changes such as Close #210 #213, Remove paragraph on simulating data from first tip #218, Add examples of private bio/biomed data (closes #188) #223, and Add a mention of testing non-ML tools as a baseline #224. I'm working through a backlog of minor suggestions from the issue tracker, though none of the PRs I intend to submit will change the structure or main content.
• Lock changes to content the manuscript on Friday, October 9 Monday, October 12 Wednesday, October 14 Monday, October 19. Grammatical and typographic mistakes identified during review are fine to fix.
• Confirm ICMJE approval and accountability for all authors who have met the other three criteria by Friday, October 23 Friday, October 30.
• Finalize author list and order
• Finish the cover letter (Add cover letter #225)
• Update the cover letter list of tips before submission
• Decide on four proposed reviewers (Reviewer suggestions #227)
• Check for conflicts of interest with reviewers and finalized author list
• Generate final PDF with line numbers
• Submit on Friday, October 30 Friday, November 6 (close Finish manuscript and submit to journal #174 at last!)

I look forward to hearing everyone's suggestions and submitting soon!

[SiminaB]

I'm reading through the manuscript now. I wonder if it would help to start an issue for each tip, to discuss them individually (eg "should we expand on X?" "should we remove Y?" etc)

[Benjamin-Lee]

This sounds like it would be a good idea. If you want to do it, please feel free. Otherwise, I'll do it as soon as I'm back to my computer later today.

[Benjamin-Lee]

I'm pushing the deadline for substantive changes back to Monday since we still have some unresolved PRs and new issues.

[Benjamin-Lee]

Pushed it back again by four more days. The submission date has now slipped to November 6 at the earliest. Once we merge @mdkessler's edits, add a section on ethical interpretation, and fix a few other minor issues, I think we should be able to freeze the paper. I know @mdkessler and @rasbt have expressed interest in doing final passes before submission. Does this timeframe work for you?

[rasbt]

Sorry this Thu and Fri will almost entirely be consumed by midterms & grading. 4 days would be a Sunday. Can we push it back to Monday? I will put it on my calendar for Mon afternoon then.

EDIT: Nevermind, I think the freeze is okay with me. I have no plans to add anything really, more like going over everything and providing feedback on what we have.

[Benjamin-Lee]

@rasbt Ok we'll go ahead with it then. I should clarify that I got to the four day figure by subtracting the new date I set in the issue description from today's date. Monday is included and it's not a very hard deadline either so don't feel pressured if you're too busy.

The part on AI ethics is the last piece of content that I think needs written. It was raised by @SiminaB in #252 and should be addressed, especially since it probably wouldn't be on the minds of readers. I've asked @juancarmona, a bioethicist (among his many hats), to contribute to that section; he said he'd work on it today.

[juancarmona]

Hello, everyone! I'm delighted that Ben has kindly invited me to write about ethical implications, which will connect nicely to many of the themes already mentioned in this manuscript. A little bit about myself: I completed my graduate education within the Harvard system, where I met Ben. I have a PhD in cell and developmental biology, a MPH in family and community health, and a MBE in ethics of genomic data in industry and healthcare. I presently work at Philips Research North America, the R&D arm of Philips Healthcare, a health technology company focused on developing clinical algorithms and tools for hospitals. Regarding the ethics section, I will certainly need time to reflect and write--as well as harmonize my text with what others have produced. Could we perhaps aim for a submission in November, which will grant me time to be thorough and effective? Thank you for your time and kind consideration. I look forward to collborating with you!

[Benjamin-Lee]

Welcome to the project @juancarmona! I'm looking forward to hearing your thoughts on what ethical considerations we should include (thread on the issue tracker coming soon). Regarding the deadline, given the fact that @rasbt is swamped, you need some time to get up to speed, and my idea to ask Nature Computational Science if they're interested (#270), let's see where we are and decide on new content freeze and submission deadlines on November 1.

[rasbt]

Sounds good! I won't be stressing out going over the article tomorrow then :), and postpone this for a week until we discussed the plans reg. ethics sections and the Nature Computational Science possibility.

[Benjamin-Lee]

Just circling back on this. @juancarmona and @rasbt do you have any idea when you can have your PR(s) in? That way we can see out new final deadline for the content.

[rasbt]

I hope to be able to review it on Monday. It's another hectic week with recording lectures, PhD committees, and wrapping up the second midterm for next week.

[juancarmona]

I will set Fri Nov 20th as a target to share a draft of my component. I too am swamped at the moment given COVID-related outbreaks at hospital customers. Thank you for your time and kind consideration.

[Benjamin-Lee]

@rasbt, thanks for starting on the review in #281. @juancarmona, focus on COVID work and feel free to take your time if you're too busy. I know Phillips is involved in everything from ventilators to monitoring so I'm sure you have your hands full of important work. Thanks again for that and for helping out with the manuscript.

[juancarmona]

@Benjamin-Lee thanks for your kind note. I am swamped with COVID-related work at hospital customers. I will use these next few days to recalibrate and get back to you regarding a date... I will very likely not be able to make the anticipated Fri Nov 20th target. Thank you in advance for your kind consideration and patience.

[Benjamin-Lee]

Hi @juancarmona, just checking back in on the ethics section. From looking over the repo, we have to merge @rasbt's comments and do some final normalizations once the content is done. Do you have a timeline on the ethics section?

[juancarmona]

@Benjamin-Lee, yes, thanks for your message. I hope that you and others here had a safe Thanksgiving holiday. I've been getting caught up emails after the holiday. My goal is to get you a draft by latest Mon Dec 21st--or sooner. I'll check-in with you 1-on-1 prior to that date.

[Benjamin-Lee]

Are there ethics topics that you can point us to so we can help you move the draft along? I'm happy to learn about ethical considerations and help with the writing to lighten your load. This is the last part we need to write and review before the paper is ready to go (pending final author sign off).

[Benjamin-Lee]

Now that the ethics section is in and pending some clean up (#304, #306). I think we are ready to freeze the manuscript. If anyone objects, say it now. Otherwise, once #304 is closed, we will freeze the manuscript to content changes. As mentioned above, only grammatical and typographic mistakes identified during review will be subject to merging. Once frozen, we will immediately begin soliciting final approvals. Once final approvals are in, we'll use those who signed off as authors and finalize the list. Then send it off!

[Benjamin-Lee]

@rasbt @agitter Are you good to freeze the manuscript? If so, we'll declare it and begin the submission process.

[rasbt]

Are you good to freeze the manuscript?

Yes, it looks good to me and I am okay with freezing it and starting the submission process.

[agitter]

Are you good to freeze the manuscript?

Yes, I'm ready to freeze content and ask authors for final review and approval.

Do you have thoughts on author order? Different models I've seen include:

• Fully ordered by approximate contribution (biology style)
• Alphabetical
• Random
• Ordered by approximate contribution with similar contributions ordered randomly or alphabetically

[rasbt]

To me, the last option with the random sorting sounds good

Ordered by approximate contribution with similar contributions ordered randomly or alphabetically

[mdkessler]

Looks good overall, but I'm giving it one more read through and finding a few minor typos - I can push them shortly.

I think an argument can be made for any of the authorship schemes...I kind of like choices 1 and 4, especially given that the target audience is used to that ordering (given the biology field standard), and so it may given them more intuition on who to contact in follow-up.

[rasbt]

I am also open to option 1, following the convention in bio.

[cgreene]

I don't think this part is necessary because this is a review article and there is no research grant / funding involved. I.e., we don't really have a PI here.

Just to note - we should ask folks to update their funding information before submission. People may have funding that supported the time that they put into this that they need to acknowledge under the terms of their grants.

Edit: also no strong feeling on ordering.

[rasbt]

Sounds good. Do we have a document for this? If not, we should maybe create a file similar to 14.acknowledgements.md where we could add the funding references.

[fmaguire]

Can manubot automatically create a funder acknowledgement section from funders details put in the author metadata?

…

On Thu., Jan. 28, 2021, 17:44 Sebastian Raschka, ***@***.***> wrote: Sounds good. Do we have a document for this? If not, we should maybe create a file similar to 14.acknowledgements.md <https://github.com/Benjamin-Lee/deep-rules/blob/master/content/14.acknowledgements.md> where we could add the funding references. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#226 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAM6WRM5NWN47MS6F4U7P7TS4HLELANCNFSM4R5G3B3Q> .

[agitter]

The author information in metadata.yaml is a good place to track funders. Any author who has funding to acknowledge can add it as a list: https://github.com/manubot/rootstock/blob/main/USAGE.md#manuscript-metadata

Currently Manubot will list the funders alongside author affiliations, but we can customize that later.

We can also track conflicts of interest in the metadata file and should ask all contributors to report those or declare they have none to report.

[evancofer]

I have no strong opinion regarding ordering.

[SiminaB]

I'm also OK with ordering whichever way. Exciting that this is so close to submission!!

[cgreene]

I added funding info and updated affiliations. @Benjamin-Lee this would be a good time to start the submission and see what author fields are required at the target journal. This way we can make sure everyone has info that's complete enough in metadata for you to submit.

[Benjamin-Lee]

I added funding info and updated affiliations. @Benjamin-Lee this would be a good time to start the submission and see what author fields are required at the target journal. This way we can make sure everyone has info that's complete enough in metadata for you to submit.

Definitely. I've already gotten it started! Additional information required is the zip code. Here's the form for reference:

[rasbt]

Shall we ask authors to add this to the metadata yaml file or would you be able to look it up (should be quick for university affiliations since the address is usually listed on the departmental Contact and Location webpage)?

[Benjamin-Lee]

Just to make it clear for each author, let's make a checklist of what each person has to do. Once we solidify it, I will email all authors (in case they miss the GH notifications) and take charge of tracking people down. What they need to do:

1. Indicate final approval and accountability by editing the contributors.md document. Edit: thinking logistically, it may be better to state up front that editing the metadata.yaml file to state that no conflicts exists also signals your final approval and agreement for accountability. The reason is that having people update two files (many of whom may not want to use their text editor to do the work but would rather stick with the GitHub UI) will get complicated. A single step may be better. Thoughts?
2. Update their metadata with funding and conflict fields.

@rasbt you beat me to the punch! Your comment came in as I was typing. Can metadata.yaml handle arbitrary fields? If so, we can have them add it there. Since people already need to indicate conflicts, it would save a lot of time (and potential errors) if they added the zip field for themselves. If someone forgets, it should be easy enough to track down. I'm also not 100% sure what to do with the department field, since not everyone has listed it. We can also include that as an instruction.

---

For people who contributed to the discussion but not the drafting, I suggest we list them in alphabetical order. We also need to solicit their approval to be listed, but that can be done via email I think. I don't want to make someone who left a few useful comments a year ago to have to make a PR so we can thank them.

[rasbt]

That sounds like a good plan forward.

Can metadata.yaml handle arbitrary fields? If so, we can have them add it there.

I think so, because the conflicts field was also not explicitly included in manubot I think. So, I think we can put any field (e.g, "zip") in there without breaking it. @agitter may know more though

[Benjamin-Lee]

What about adding "accountable" and "approval" fields? That way it's a paste three lines sort of PR but still explicit approval.

[rasbt]

Yeah, we can add that too, why not. Btw what does "accountable" mean in this context? Based on the https://github.com/Benjamin-Lee/deep-rules/blob/master/contributors.md I wouldn't be accountable for example

[mdkessler]

Yea what does accountable mean?

[Benjamin-Lee]

Sorry, I was unclear. Since the criteria for authorship we chose requires authors to agree to be held "accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved", I'd like some way to have it be explicitly tracked. The reason we used the table in markdown was so that we could track progress towards achieving author status, with final accountability and approval to be solicited once the manuscript is done.

The question that's on my mind is the easiest way to confirm those criteria have been met. To be concrete, we need some way for @rasbt to indicate he's agreed to be accountable. One way is to have him PR to contributors.md which is what other some other authors have done when they added PRs touching the manuscript satisfying the "drafting" field. The alternative I'm proposing is to have him PR his zip code, which we need all the authors to submit anyway, along with the final two criteria (accountable and final approval) fields to metadata.yaml.

The main thing I want to avoid is burdening the coauthors with PRs touching multiple files for something simple like signing off on the manuscript. Hope that clarifies what I was getting at.

[mdkessler]

Thanks for clarifying - that makes sense to me!

[rasbt]

Thanks for clarifying! I didn't touch the contributors.md reg. accountability and authorship previously because I thought this might be best filled out by the main organizer / @Benjamin-Lee. I can update it though if necessary under the alternative

The alternative I'm proposing is to have him PR his zip code, which we need all the authors to submit anyway, along with the final two criteria (accountable and final approval) fields to metadata.yaml.

Other than that, I've read the whole manuscript when I made the edits back in December and have been following all the PRs after that. So, I am happy to give approval and be accountable. I can add this to the author metadata.

[agitter]

Can metadata.yaml handle arbitrary fields? If so, we can have them add it there.

I think so, because the conflicts field was also not explicitly included in manubot I think. So, I think we can put any field (e.g, "zip") in there without breaking it. @agitter may know more though

Yes, @rasbt is correct that we can add arbitrary fields to the metadata.yaml file. Each author's data is read in as a dict, which you can see on the output branch

deep-rules/variables.json

Lines 45 to 62 in 351ebd3

	"authors": [
	{
	"github": "Benjamin-Lee",
	"name": "Benjamin D. Lee",
	"initials": "BDL",
	"orcid": "0000-0002-7133-8397",
	"email": "[email protected]",
	"affiliations": [
	"In-Q-Tel Labs",
	"National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health",
	"Nuffield Department of Medicine, University of Oxford"
	],
	"affiliation_numbers": [
	1,
	2,
	3
	]
	},

If we want to do anything with these new fields for each author, like build a table listing conflicts of interest, I can help edit the manuscript to do that using template variables.

[agitter]

@Benjamin-Lee are you ready to submit this soon? Do you need someone to help enter information into the submission system?

[rasbt]

Oh I thought it was already submitted. Yeah, if not, please let us know if you need help with the process! Happy to help!

[Benjamin-Lee]

It’s on my radar! Sorry, just in the middle of a move

…

On Apr 18, 2021, at 10:19 AM, Sebastian Raschka ***@***.***> wrote: Oh I thought it was already submitted. Yeah, if not, please let us know if you need help with the process! Happy to help! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#226 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADH4YHI2RH3D3ZLOCZRKSPTTJLTAPANCNFSM4R5G3B3Q>.

[juancarmona]

@Benjamin-Lee, good luck with the move and please keep us posted :)

[Benjamin-Lee]

Submitted! 🥳

[rasbt]

Awesome news. Thanks for putting this together!

[SiminaB]

Any updates from the journal?

[agitter]

I had the same question and emailed Benjamin about this last week. There still hadn't been any updates from the journal, so he was going to contact the editor.

[Benjamin-Lee]

Got a response earlier today:

Dear Dr Lee ,

Thank you for your email. We apologise for the delay.

Your manuscript is currently undergoing peer review. The first reviewer was invited on 21 June 2021, but unfortunately, we have had some difficulty in obtaining the right reviewers to cover all aspects of your manuscript. However, one reviewers completed and so hope to be able to move your manuscript on soon. Once the required reviews are submitted, the editor will assess them and make a decision as to how to proceed. You will be informed of the decision at this stage.

Given the disruptions resulting from the ongoing COVID-19 pandemic, please expect delays to the editorial process. We apologise in advance for any inconvenience caused and will do our best to minimize impact as far as possible. Please be assured that the editorial team are aware of the delay and are keeping a close eye on the progress of your manuscript. We will do everything possible to ensure that the manuscript moves smoothly through the remaining process.

I hope this information is useful, but please do not hesitate to get in touch if I can be of further assistance.

Kind regards,

Anita Gerenscer
Journal Assistant, Editorial Office Ltd.
On behalf of PLOS Computational Biology