-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support ITU-T Meeting "Contribution" document type #514
Comments
@taddhar could you please help provide a sample Contribution document for templating purpose? Thanks. |
Hi all, sure thing. |
Here is the base template |
The most important are the fields in the header and in particular one must register a number before submitting So the Cnnnn because say C614 |
Now there are different types of Contributions. There are the general ones, someone wants to make a simple text about something. Here is an example: |
Then a very common type of Contribution is a contribution for a new baseline revised text. Here is an example: |
And one that is one of the most complex one is a contribution for a New Work Item: |
This one is a New Work Item for a Recommendation and note that the table in the middle is refering to a so-called A.1 template which is here |
When a contribution is for a Technical Paper, a Technical Report or for a non normative text one needs to use an A.13 template which is here: |
Here is an example Contribution for a Technical Paper or Report |
Please let me know for anything |
@taddhar As I am sure you will appreciate, Metanorma relies on rigorous and parsimonious structure. What you have provided is not enough to work from, and I require a more rigorous presentation of possible contribution types. I am trying to work out what the variation is between the different artefacts you have provided, and I should not be having to analyse a quite possibly limited number of documents to work out what is going on. That is your responsibility to provide. I am particularly unhappy that you have not provided a definition of "baseline revised text", and with zero Google hits for +ITU +"baseline revised text", I really am being forced to guess what this means. You are familiar with ITU processes. I am not. It is not proper use of my time for me to have to spend hours Googling and guessing. metanorma-itu has implemented ITU-T Editing Guidelines, which applies to ITU-T Recommendations. That is how contributions are going to be styled by default. If different styling applies, you need to tell me. As far as I can work out, contributions can follow the following structure:
The target document shall be entered as a separate document, which shall be embedded into the contribution document as an annex; e.g. SG17-C-0639
(Embedding is a Metanorma-specific means of including a document which preserves that document's metadata, and keep it a separate document.) A revised draft document is NOT presented as an annex; we can realise that inconsistency, but it is an inconsistency with New Work Item proposals SG17-C-0660
The doctype shall be consistently So:
Forms A1 and A13 are pretty much the same table, with some different introductory guidance on populating them; these are going to be realised as the same table, with some Liquid templating to change what gets rendered, and the introductory comment are going to form part of a Metanorma template. So mn-template-itu, contribution.adoc will contain the following Liquid template, activated only in case of a new Work Item:
@ronaldtse I'm sure would like to see the entire A.1/A.13 table automatically populated from metadata. I don't think that is a good use of programming at all, especially if differences creep in. Once the target documents are decoupled from the contribution as embedded documents, and the A1/A13 forms are streamlined and partly populated in Liquid, the contribution becomes reasonably tractable. |
@ronaldtse Would be useful for us to have the documents that @taddhar supplied converted sample documents for us to test contributions. But I don't need them immediately, and I will be experimenting with how best to structure these anyway. Templates are going to do a lot of the heavy lifting here for forms... |
@opoudjis I read with interest your elaborated comments above and I completely understand your perspective. Sadly I only published #520 now which I wanted to do earlier as I realized quickly by myself that this was not correct and I am not the right person for what you need. I can only see what I need from my relatively small perspective and will work to bring here the right people with the right authority so that you get the right input and do not need to do undue search or guessing. This is indeed neither your job, nor a good use of your time, and this is simply not the right way. Of course, I know by experience that even if you have the right people in this repo, there will be a grey area on a number of things so my own 'pragmatism' might turn to not be too bad at the end but let's make this properly. |
So first YES I absolutely recognize and appreciate that
this is a strong benefit that I completely understand and want this principle to stay bold. |
Ahh yes, the 'baseline revised text' ... what is it in a non rigourous description. So say you have a contribution proposing to establish a new work item say X.bla with an initial proposed baseline text for a Recommendation and therefore an A.1 justification table. This initial baseline text must follow the skeleton template for an ITU-T Recommendation (I guess i forgot to send this one right? or you have it already as you are already able to produce Recommendations ?) This contribution is produced and submitted for a certain meeting of a certain study group, say SG17. The contribution is published say as C650 before a certain cut date (12 days before SG17 meeting starts). And unless exceptions, once the cut date was passed C650 cannot be revised! If it is revised AFTER the cut date it can be considered as a 'late contribution' and in principle should be deferred to the NEXT SG17 meeting. Then the meeting starts and a certain Question of SG17, say Q15, reviews the contribution C650 and agrees to establish the new work item X.bla. At that point during the meeting the Rapporteur of the Question publishes a TD, say TD2100 which contains:
Then Q15 sends TD2100 to its upper Working Party that discusses again this TD2100 in Working Party Closing Plenary and if they agree to establish it it is sent to the SG17 closing plenary (the last day). At the end of the SG17 meeting, in closing plenary, SG17 say agrees then to establish the new work item X.bla as part of the work program of Q15. At this moment X.bla is part of the work program and TD2100 is used as its initial 'text' including an initial baseline text which is the one in the Annex of TD2100 which means the annex of C650 modified by Q15 in TD2100. Then X.bla will appear in the work program. I imagine you can access to the current work program of Q15 but not to the content of the files because you don't have membership access I imagine: For example you may be able to see the one I produced as X.icd-schemas using metanorma and liquid templates thanks to all your help! Now at the next SG17 meeting someone wants to propose to improve the current baseline text which is in TD2100. Typically you can do this with a new Contribution at the next meeting. In which case typically what I would do is create a new contribution using contribution template, then I would load the initial baseline text or the current baseline text, I would approve all the modifications, would put now in new revision mark mode and one could see all the new changes proposed. Then I submit a Contribution say C850 which is NOT for a new work item but simply a contribution to revise the baseline text of an existing work item, in our case X.bla ... In fact others could do other contributions in parallel and in this case you end up in the meeting with 1 or multiple contributions to revise the new baseline text. When Q15 will open these contributions, the EDITORS of the work item, need to coordinate all the contributions and agree what is the final modifications. Then they produce a new TD, say TD3500 and then we agree at SG17 closing plenary that this is the new baseline text for this work item etc. etc. And this continues until it is decided that the text is stable and then we go for the next step of the process: consent if AAP process, determination if TAP process. AAP = Accelerated Approval Process and TAP = Traditional Approval Process Does it help you 'visualising' the baseline text improvements? |
Ahh in fact there is an authoritative source. Am not sure if you are aware of the A.1 Recommendation: https://www.itu.int/rec/T-REC-A.1-201909-I/en This may answer a LOT of your questions, including on groups, sub groups, etc. |
Ahh yes:
Yes I would believe that there are different styles for documents that are not Recommendations. First of all you have other published documents example:
If you take a step back the way I see it is that you could group all the documents in 3 first categories:
|
Yes then I understand your approach on how you try to make a categorisation, great job and i see the need to 'translate a bit':
Ahh yes we didn't review that case ... Hmm, there are here some specialisation of New Work Items for example to do:
Yes this is the whole point of my early long coment
Yes and it has a TD number ... I hope you can at least see the metadata in the example of the work program of Q15 as per abobe comment
Yes Contributions that constitue proposals as NEW WORK ITEM for a specific target document (a Recommendation, a Technical Paper, etc. INCLUDING a Revision, an Amemdment, a Corrigendum, etc. ... yes I forgot that case actually)
Yes I agree on your remark therefore my own complaint that we do not even have the right templates in word ... so each time one needs to take a Contribution template, then add justification, gap analysis, etc. then pick either A.1 or A.13, then add an initial baseline text ... how annoying.
Am not sure what you mean here |
Well if you only use Word (which we do now in ITU) we cannot do that BUT HERE this would be a GREAT benefit of Metanorma: "the lego brick approach" I would love it.
Ok I am not too sure what it means yet because my knowledge of metanorma is too limited but I believe I understand and like your intention.
@opoudjis I believe I follow your thinking and I understand the problem you raise for Contribution for new baseline text on an existing work item and my proposal could be to do some tries and see what is the best possible implementation. In fact this is the part that today requires that one takes the initial document, makes modification and at the end you must do a WORD comparison, this is what I had to do for X.icd-schemas as I had sooooo many changes, it led to a very big Word file to open in revision mark ... and until I realized I could use simple rev marks, with full rev marks, my Mac CPU was litterally BURNING. But this was due to the specific of this work item. |
finally I am in Frankfurt on my way to Australia for IETF 119 so I do my best now that I am insulated from family, colleagues, etc. :-) I hope this helps a little bit @opoudjis |
That helps, thank you @taddhar, although I will confess to still being a little hazy on how this will all come together. What I am going to try to put in motion, after my analysis and your explanation, is indeed going to be somewhat Lego-brick-like:
I'm going to start with the General contributions, since they don't involve A.1/A.13 or embedded documents, but I know what to do for the first iteration of this work. The devil is always in the detail with this kind of work, and the Word output won't necessarily look identical to the samples; but it is still a more tractable workflow than what is in place now... |
The things to be entered into the base template of ITU-T contributions are as follows; we are actually already capturing all of them, since they are expected for technical reports.
Contributions don't have subgroups and workgroups. The abstract is marked up as normal, The remaining text in the template will go to the Metanorma template. |
I've generated the generic contribution, without embedded document, for SG17-C-0223. I attach it, rendered in English and (for jollies) Russian. You will note that the initial table is entirely populated from document attributes, and that those document attributes were already defined for technical reports, which have the same structure. I have other tasks to tend to, so I will come back to embedded documents next week; but I'm happy with where this got to. This is not 100% replication of the source document format, but frankly, that's the effort this format of document merits, as an initial draft, and it's already taken up much of the week. The PDF and HTML are going to be less precise, but I don't think they warrant the additional effort at this stage. |
We've done Output documents to date, per your taxonomy. I'm now implementing:
That at least is a document structure taxonomy that I can work with. I sort of understand the process of proposals and TDs and revisions and whatnot that you laid out, but you are asking me to make sense of a very convoluted process, that only tangentially seems to affect what I will end up doing with document formatting; so I'm disinclined to tackle TDs. These sound like they will be edits of Word documents as Input documents, whatever else happens in Metanorma, and the amount of detail tracking of edits that TDs require sounds to me intractable and prolix. Appendix I of ITU A-1, the Rapporteur progress report format, is very discursive, and I'm not seeing how it can even be put into a template, let alone be machine-populated. At any rate, I have no idea what they even look like, and whether they structurally are any different from edited input documents at all (just adding more text to the input document preface). If they are not structurally different, then editing input documents really will be enough. Next I am going to do a baseline revised text contribution. The text of the contribution is going to be C-0660, but the embedded document is not going to be X.arch, because I don't have Asciidoctor source for that, and I don't need it for a sample document; I am grabbing T-REC-P.1203-201710-I as a brief recommendation from mn-samples-itu. We'll touch base on this when you're in town, but if things go well, I'll be well on the way to realising what is in this ticket by the time you are. |
Just opened 660... ... Metanorma does not realise Word Track Changes, and will not: that is a HUGE undertaking to realise, and one that may well force us to reimplement Metanorma Word output from scratch. As you can understand, I am unwilling to entertain that possibility. If you want Word Track Changes in anything that Metanorma generates as a baseline revised text contribution, you will need to:
If you're going to do that, of course, it calls into question generating revised text contribution through Metanorma at all. The embed mechanism doesn't really make sense, if that's what you expect: you might as well generate the before and after as two separate output documents, and then paste the comparison into the Contribution generated without embedding. So Metanorma only makes sense for the automatically generated preface template, in that case. Of course, this is not an issue for New Work Items: the embedding approach there makes all the sense in the world. But if we are undertaking to support a very Microsoft Word-bound process, you need to do analysis on what that will end up looking like. It won't look identical to the existing Microsoft Word-bound process, because Metanorma is not Word... Anyway, proceeding with 660, but there will be no Track Changes in what we generate, and I've explained why. |
We're going to have trouble with the fact that New Work Items treat the embedded document as an appendix. Our embedding model to date has relied on the fact that any embedding content is prefatory to the embedded document, which is why the embedded document's numbering is preserved intact: it is the main document. Putting Appendixes will instantly destroy that, and in fact shows the bankruptcy of treating the embedding + embedded document as a unitary document: the embedded document is an Appendix, but its clauses are numbered as if it is a self-standing document (and of course, it is: its content is merely cut and pasted into the embedding document, clause numbers and all). That's going to be a major change to the handling of embedded documents, and it will likely force structural changes to Metanorma. |
We're even getting that change forced in the case of baseline revised text contributions. Embedding does not generate a title page for the embedded document (title, keywords, summary), because embedding expects to attach prefatory content to a document, and render it as a unitary document. This embedding expects a title page on the embedded document; and embedded documents usually will. Likewise, there is no table of contents for the embedded document: Metanorma expects that it is getting a unitary document, and that tables of contents are for that entire document. The table of contents here is specific to the embedded document, since it is after all a cut and paste of the embedded document into the embedding template. This leads to an unpleasant situation: the Semantic XML of such documents is going to end up containing self-standing documents, which need to be processed separately. In fact, these are not embeddings at all: they are document collections, because document collections are how we keep documents self-standing in Metanorma. And we have never attempted to create a single Word document out of a document collection until now. Embedding almost gives us the right output, as the attachment shows. It is "only" missing the title and table of contents of the embedded document. But I no longer think it will give us the full right answer, especially once appendixes start turning up. Making that title and table of contents turn up is harder than it looks, and it would be a kludge. I don't think I should be tweaking embedding to enable separate processing of embedded documents, or contaminating the Metanorma document model with embedding, when that's what document collections were designed for. So I am going to start looking into what it will take to get a document collection working instead. |
We have had a similar instance to this in JIS commentaries, which are included in the same Word document as a standard but paginated differently, and with a different header. But we need to formalise this in Metanorma for Word documents specifically, because this is going to keep coming up, and we need a coherent approach rather than ad hoc solutions (such as we arguably did for JIS). In Metanorma, we have a concept of a document: something that follows the Metanorma document schema, with a preface, clauses, and annexes in that order. The clauses are numbered consecutively from beginning to end. There is a single cover page, a single instance of introductory boilerplate, and a single table of contents. We have instances where what were originally separate documents are combined into a single document, and are processed as a single document. That is embedded documents. So whatever the numbering of the clauses in the source document, the resulting artefact is a single document, with a single numbering of clauses from beginning to end, and a single cover and intro page. The way they have been implemented to date, the embedded document is the main set of clauses, and any wrapping content is additional prefatory content (or potentially additional annexes). That's how there can still be a single numbering of clauses: there is a single document, with stuff added before and after it. We also have had instances where multiple documents reference each other, and are compiled as a document collection, dynamically updating those cross-references as hyperlinks. These are treated as a document collection: they end up all being aggregated into a single XML artefact, and the cross-references are preprocessed across the entire collection, so that they render consistently as hyperlinks. But each individual document in the collection is processed as a standalone document: the only thing that links the documents together is the hyperlinks and an overall cover page and index. Document collections to date have been implemented as HTML and PDF, not Word. We have implemented the following to date:
That brings us to ITU contributions. On the face of it, they are Embedded documents. They add prefatory material to an separately circulated output document, presenting it as a single document. They maintain the same document header, and consecutive page numbering. But:
I can tweak the embedded solution to cope with these requirements—automatically extracting a separate cover page somehow, and inserting a table of contents into the embedded document. But the embedding happens as an Asciidoctor include, during document preprocessing. The metadata of the embedded document is preserved in the metadata of the embedding document, as metadata about what the document is based on. But as far as Metanorma processing is concerned, this really is a single document; and telling Metanorma to jump back from annex to preface is going to break its processing model. That's why the real solution to rendering these has to be a document collection. @ronaldtse is going to be unhappy about that, because document collections are more complicated to set up. I have no sympathy for that objection: document collections are going to be preconfigured in the mn-templates-itu, and I'm not going to destroy the Metanorma processing model to force what is clearly two separate documents to be processed as a single document, but end up looking like two separate documents anyway. These are two separate documents, and Metanorma needs to render one document at a time. This is going to require added a lot of implementation effort. First, we have never dealt with document collections as a single Word document at all—although making that happen will not be prohibitive. Second, we need directives on what the cover page and intro page of each document looks like in Word, which will be a truncation of the standalone cover and intro pages, passed as custom templates by the collection. Each Word document is structured as Title section (no page numbers), Preface section (usually Roman number pagination), Main section (Arabic number pagination). The workflow is going to be:
By default, each section (preface and main section of each document) will restart page numbering from 1, but I can introduce configuration to continue numbering. In the case of ITU New Work Items, the custom title document will be:
In other words, I am not proposing to mark up the embedded document as an annex at all, but to keep the annex as an artefact of the cover page. As I said above, the embedded document is not rendered as a true annex: it really is just treated as a separate document. So I'm not feeling guilty about that kind of subterfuge. |
@taddhar I am not expecting feedback on these comments, and I am not inviting extensive discussion on it. This is an implementation challenge I need to surmount, and I am laying out the considerations, since any course I take is going to have serious consequences for the structure of the implementation of Metanorma overall. And I cannot keep kludging solutions to embedding autonomous content. This is harder than I'd have preferred, and I'm going to investigate what I can do about it. |
I'm going to create a separate ticket for the generation of Metanorma collections in Word. Once I've got a satisfactory outcome on that, I'll resume work on this ticket. |
Resuming work on this, after a lot of distractions. We have a framework for generating document collections in Word. We now need to set up templates in mn-templates-itu for:
|
There's an issue that the coverpage of the embedded target document has keywords on its coverpage, but standalone documents already have keywords after the table of contents; so there is an inconsistency between the Collection Embedding positioning of keywords (in the ad hoc coverpage) and the standalone document (whose formatting I am not overriding, other than its cover page.) This is not worth fixing right now, and if anything, I'd argue that the keywords should be removed from the place they've ended up in the embedding cover page. |
Populating A.1/A.13 from document metadata is raising a problem. Do we author them as:
The former allows us to interleave metadata and document content. But if we are going to use document metadata in the form, we have to either duplicate the attribute processing done in Presentation XML (which is a coding smell), or duplicate the document metadata in a YAML data source. The latter is how we populate cover pages already, including the metadata of the contribution cover page. It gives us more flexibility in formatting, and it lets us reuse parsed data. But it requires us to enter all its information as document metadata (document attributes), not as document text. For example, we already provide question information in document metadata, like this:
This is broken up into separate questions, and separate identifiers and titles, but that happens in Metanorma XML generation. It has not happened during preprocessing, which only knows about the raw "question" attribute. I don't want to write the parsing of the question attribute in two places, or enter it in two places (the document attribute and some YAML file). But if I do the forms as programmatically generated text, they have to be fully programmatically generated: I can't do that as half programmatic, half document, like this:
The "2020-12" and "AAP" have to be document attributes as well. Which is fine, except we also have big chunks of open-ended formatted text in the same table, which does not make sense as document attributes:
I can't do this, because the processed metadata like The way out of this is to enter the textual bits of Annex A with identified fixed subheadings, and assemble the table out of a combination of those subheadings and the processed metadata. This is similar to the approach taken in participant lists for IEEE. So:
The output form A.1/A.13 is going to be populated in Presentation XML, as a combination of those labelled subheadings in the Annex A, and the document metadata. It's somewhat clunky, but the alternatives are clunkier. |
Will allow type on annex, since we will set |
This is done. There is a contribution-new-item and a contribution-revision in mn-templates-itu, both generated as collections, and the latter incorporates A.1/A.13, identified to be populated as |
Similar to #375
The text was updated successfully, but these errors were encountered: