Problem Scope: What needs to be marked up? How?

[tombogle]

There are several types of text that can potentially be regarded as speech or quoted text. These types of text can be identified in several different ways:

• Presence of quotation mark punctuation in the text itself, included both paired quotation marks and start-only punctuation, such as a dialogue dash or a colon.
• Special-purpose character styles that apply to the entire quote (e.g., in USFM, \wj or \qt).
• "Paragraph" styles, which are usually formatted using block indentation (.e., in USFM, [\pi](https://ubsicap.github.io/usfm/paragraphs/index.html#pi, \q, etc.)
• "Milestones" to mark the start/end of speech, quotes, etc.
• Special headings (often used in Song of Solomon)
• Natural language. We tend to think of this as "unmarked" because it is typically requires knowledge of the language and is difficult/impossible to conclusively identify the direct speech using automated means. Some languages indicate both the start and end of the speech/quote, while others only indicate the start. (There might even be languages that only indicate the end.) Note: This is how most speech in the original Hebrew/Greek text is identified.
• Implicit speech/thought. Nothing in the text directly identifies the speaker or even that someone is "speaking" but the words themselves imply that they are being spoken, written or thought by a particular character. This is common in prophetic books, where a exchange is taking place between the prophet and God. It is also employed in some books (e.g., Nehemiah) where the author sometimes writes in first person.
• External/standoff markup. Information about speech by biblical characters is stored in some external file that is not necessarily published with the source text. For example, Glyssen creates its own copy of the source text and the work done to identify speakers is not necessarily processed in a way that allows that information to be fed back into the original USFM (though there are some Voice Marking Tools designed to make that possible).
• Direct formatting. This is not usually employed in Scripture and it is not considered a best practice.
  In many cases, more than one of these methods may be employed for the same speech. It is also very much the case that within a translation project these methods might be used inconsistently.

Depending on the intended use(s) of a text, not all "speech" needs to be marked up, and it may be acceptable to have it marked up in different ways. However, using lots of different ways to mark up speech does tend to make it more difficult to do various kinds of automated processing or parsing of the text. To the extent that it is feasible to maximize the extent to which speech is identified and the richness of the associated information and minimize the complexity of how that is done, current and future tools will be better able to access that information to help users with tasks. Ideally, we want to make it as easy as possible for translators and downstream editors to supply the needed information in a consistent way.

Currently, my operating belief is that milestones offer the best hope of useful markup because they do not interfere with the text itself (i.e., they can be used whether or not the script or language requires/allows quotation punctuation, formatting or particular wording to identify speech) and because they allow for extra information to be supplied that cannot always be easily inferred from the text. However, I'm open to being shown why this not necessarily the best way...

[RobH123]

Can a USFM milestone occur inside the text of a paragraph without interrupting the flow? (I think not.) For example:

\c 1
\v 1
\p Jesus answered and said, "You must be born again."

becomes

\c 1
\v 1
\p Jesus answered and said, "
\someStartMilestone
You must be born again.
\someEndMilestone
"

[RobH123]

One thing that I've noticed with heavily-tagged whole-book USFM is that it's large to download and especially it's slow to parse, and hence we usually need one of the following:

1. User software with fancy prefetching, fast parsing (without any expensive error checking but that isn't good for WIP), and then caching
2. Having it prechecked and preparsed already by the server, and then we just fetch data structures (e.g., JSON or bit-compressed data in the case of Proskomma).

It depends if the software is just display software (so we assume no errors, or just ignore them) or an editor (so we can't guarantee correctness at any point in time) where we have to rewrite all the edited data.

All to say that I've been leaning more towards stand-off markup lately (so that the actual text is short and quickly parsed), but the markup needs to be not too fragile to break on small text edits and the cost is that a system is needed to ensure that both text and markup stay in step.

[tombogle]

Obviously, the big downside to milestones (and even special character markers like \wj) is that they get in the way. If the editor "hides" them, then the chances are higher that they will get mangled or end up in the wrong place. If the editor shows them, they interrupt the flow of the text and prevent the user from reading the text in a straightforward way. Probably the best thing is an editor that is smart enough to display the effects of the milestones without actually showing the markup (e.g., through background shading), but even then, when editing right at the character positions right on the boundary of where the milestone is, it's not easy to ensure a perfect result. In some ways, this is no different from a normal word processor with styles or formatting applied to a bit of text. It will have rules about when that formatting applies to text typed at the beginning or the end of the run of text, but it won't always be what the user intended. (And depending on how astute the user is and how subtle the formatting differences, they might or might not notice that they got the wrong result.)
The performance issues are probably going to be a bigger deal in web-based JS apps than in an installed app running compiled code. In any case, there would be relatively few quote milestones, compared with the \w markers for every word, so it might not be as big of a problem in this case. Probably a test would be in order to see.
Although standoff markup completely solves the problem of interfering with the user's work when editing and at least relegates performance issues to cases where that data is needed, it exacerbates the problem of keeping the markup in sync with the text when it is being edited. For scripts/languages that use quotation punctuation fairly rigorously to mark up direct speech, it will likely continue to be true that identifying speakers will not be a high priority during drafting and this task will be left to a later stage in the project, if done at all. For projects that do not mark up quotes adequately with punctuation, even if we can succeed at having tools that make the job of identifying speech and speakers easy, having the markup "out of sight" is likely to also keep it "out of mind." In general, I think the earlier in drafting we can get users to provide the information about who says what, the more likely they will do it consistently and the richer the data will be, but also the more annoying it will be and the more likely to get broken.
Another downside of standoff markup is that it is more likely to get lost, forgotten, or ignored.
One of the great things about standoff markup is that it does not require universal acceptance. Software tools that don't want to think about this can theoretically ignore it.
Paratext already uses standoff markup for its notes feature. To do this, the "quoted text" is kept as part of the note, and then Paratext attempts to find that text in the verse, even if it has moved from its original location. In the case of a note, the note can still potentially be meaningful even if the original text has changed to where the quoted text can no longer be located. It's up to the user to make that determination. In the case of speaker identification, where a computer will have to do the work of finding the relevant text and deciding if the markup is still meaningful, it's a bit trickier. Still, I suspect that in most cases, a heuristic could fairly easily determine what's what. The problem space is generally fairly simple. The most complicated cases would be:

1. where a verse has two speakers in dialogue and the text associated with both of them changes.
2. an early translation decision is made about who is speaking, and later the verse is translated differently in a way that changes who is speaking (e.g., in MAT 14:5, where some translations attribute the words to Herod and others to the people)
3. scripts that do not use explicit quotation marks at all; even if there is a single known speaker, if the start or end of the direct speech is edited much, it will probably become impossible to know which part of the text is speech.

One concern I have about standoff markup is that since the USFM standard has already been enhanced to specify a means for marking up speech, if we come up with another competing approach (even if it is superior), tools may then have to be developed to support both approaches. (In the famous words of Nathan Miles, "The great thing about standards is that there are so many of them to choose from.")

[RobH123]

@tombogle said: the USFM standard has already been enhanced to specify a means for marking up speech? Seems he was talking about https://ubsicap.github.io/usfm/milestones/index.html#qt-s-qt-e (new in USFM 3.0).

From the example there, it doesn't seem that it was intended to also mark the narrator, although perhaps all text outside markers can always be assigned to the narrator by default?

[tombogle]

I'm pretty sure that marking narrator speech was not at the forefront of anyone's mind when devising the USFM milestone markup approach. But it is still possible to do. And there are situations where it makes sense. In my implementation in Glyssen for processing the milestone markers, I do attempt to handle both normal narrator quotes (e.g., "scare quotes") and narrator interruptions, such as -- let the reader understand --.