Skip to content

Commit

Permalink
Merge pull request #349 from cellar-wg/update-to-xml2rfc-version-3-pr…
Browse files Browse the repository at this point in the history
…ocessing

Update to xml2rfc version 3 processing
  • Loading branch information
dericed authored Oct 27, 2019
2 parents bc7dafe + 9d4596c commit 20e8df3
Show file tree
Hide file tree
Showing 18 changed files with 523 additions and 272 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ ebml_matroska_elements.md
ebml_matroska_elements4rfc.md
matroska_tagging_registry.md
matroska_xsd.xml
metadata.min.js
22 changes: 15 additions & 7 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$(info RFC rendering has been tested with mmark version 1.3.4 and xml2rfc 2.5.1, please ensure these are installed and recent enough.)
$(info RFC rendering has been tested with mmark version 2.1.1 and xml2rfc 2.30.0, please ensure these are installed and recent enough.)

VERSION_MATROSKA := 04
VERSION_CODEC := 03
Expand All @@ -10,6 +10,14 @@ OUTPUT_MATROSKA := $(STATUS_MATROSKA)ietf-cellar-matroska-$(VERSION_MATROSKA)
OUTPUT_CODEC := $(STATUS_CODEC)ietf-cellar-codecs-$(VERSION_CODEC)
OUTPUT_TAGS := $(STATUS_TAGS)ietf-cellar-tags-$(VERSION_TAGS)

XML2RFC_CALL := xml2rfc
MMARK_CALL := mmark

-include runtimes.mak

XML2RFC := $(XML2RFC_CALL) --v3
MMARK := $(MMARK_CALL)

all: $(OUTPUT_MATROSKA).html $(OUTPUT_MATROSKA).txt $(OUTPUT_MATROSKA).xml $(OUTPUT_CODEC).html $(OUTPUT_CODEC).txt $(OUTPUT_CODEC).xml $(OUTPUT_TAGS).html $(OUTPUT_TAGS).txt $(OUTPUT_TAGS).xml

matroska_xsd.xml: transforms/schema_clean.xsl ebml_matroska.xml
Expand All @@ -21,23 +29,23 @@ check: matroska_xsd.xml
ebml_matroska_elements4rfc.md: transforms/ebml_schema2markdown4rfc.xsl matroska_xsd.xml
xsltproc transforms/ebml_schema2markdown4rfc.xsl matroska_xsd.xml > $@

$(OUTPUT_MATROSKA).md: index_matroska.md diagram.md matroska_schema_section_header.md ebml_matroska_elements4rfc.md ordering.md chapters.md attachments.md cues.md streaming.md menu.md notes.md
$(OUTPUT_MATROSKA).md: index_matroska.md diagram.md matroska_schema_section_header.md ebml_matroska_elements4rfc.md ordering.md chapters.md attachments.md cues.md streaming.md menu.md notes.md rfc_backmatter_matroska.md
cat $^ | grep -v '^---' > $@

$(OUTPUT_CODEC).md: index_codec.md codec_specs.md subtitles.md block_additional_mappings_intro.md block_additional_mappings/*.md
$(OUTPUT_CODEC).md: index_codec.md codec_specs.md subtitles.md block_additional_mappings_intro.md rfc_backmatter_codec.md
cat $^ > $@

$(OUTPUT_TAGS).md: index_tags.md tagging.md matroska_tagging_registry.md tagging_end.md
$(OUTPUT_TAGS).md: index_tags.md tagging.md matroska_tagging_registry.md tagging_end.md rfc_backmatter_tags.md
cat $^ > $@

%.xml: %.md
mmark -xml2 -page $< | awk '/<?rfc toc=/ && !modif { printf("<?rfc tocdepth=\"6\"?>\n"); modif=1 } {print}' > $@
$(MMARK) $< | awk '/<?rfc toc=/ && !modif { printf("<?rfc tocdepth=\"6\"?>\n"); modif=1 } {print}' > $@

%.html: %.xml
xml2rfc --html $< -o $@
$(XML2RFC) --html $< -o $@

%.txt: %.xml
xml2rfc $< -o $@
$(XML2RFC) $< -o $@

matroska_tagging_registry.md: matroska_tags.xml transforms/matroska_tags2markdown4rfc.xsl
xsltproc transforms/matroska_tags2markdown4rfc.xsl $< > $@
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This repository holds content related to the official Matroska specification and

## About this repository

Local versions of the specification can be generated based on code in the `Makefile` directory and related dependencies. The dependencies required are `mmark`, `xml2rfc` and `xsltproc`. `mmark` is a Markdown processor written in Go, available [here](https://github.com/miekg/mmark) or, for Homebrew users, can be installed with `brew install mmark`. Installation instructions for `xml2rfc` (an XML-to-IETF-draft translator written in Python) are available on the [IETF Tools page](https://tools.ietf.org/tools/). `xsltproc` is a command line tool for applying XSLT stylesheets to XML documents.
Local versions of the specification can be generated based on code in the `Makefile` directory and related dependencies. The dependencies required are `mmark`, `xml2rfc` and `xsltproc`. `mmark` is a Markdown processor written in Go, available [here](https://github.com/mmarkdown/mmark) . Installation instructions for `xml2rfc` (an XML-to-IETF-draft translator written in Python) are available on the [IETF Tools page](https://tools.ietf.org/tools/). `xsltproc` is a command line tool for applying XSLT stylesheets to XML documents. A bootstrap and Makefile are provided to gather dependencies and generate the RFC documents.

To create local copies of the RFC in `.txt`, `.md`, and `.html` format, run `make`.

Expand Down
128 changes: 128 additions & 0 deletions bootstrap
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
#!/bin/sh
#
# bootstrap script to get the tools needed to build the specs within a UNIX shell

export LC_ALL=
FOUND=
NEEDED=

BOOTSTRAP_MAKE=bootstrap.mak
RUNTIMES_MAKE=runtimes.mak

check_version() {
gotver=$2
gotmajor=`echo $gotver|cut -d. -f1`
gotminor=`echo $gotver|cut -d. -f2|cut -d+ -f1`
gotmicro=`echo $gotver|cut -d. -f3|cut -d+ -f1`
[ -z "$gotmicro" ] && gotmicro=0
needmajor=`echo $3|cut -d. -f1`
needminor=`echo $3|cut -d. -f2`
needmicro=`echo $3|cut -d. -f3`
[ -z "$needmicro" ] && needmicro=0
if [ "$needmajor" -ne "$gotmajor" \
-o "$needmajor" -eq "$gotmajor" -a "$needminor" -gt "$gotminor" \
-o "$needmajor" -eq "$gotmajor" -a "$needminor" -eq "$gotminor" -a "$needmicro" -gt "$gotmicro" ]
then
echo "$1 too old (got $gotver, needed $3)"
NEEDED="$NEEDED $1"
else
FOUND="$FOUND $1"
echo "found $1 version $2 (needed $3)"
fi
}

check() {
if ! $1 --version >/dev/null 2>&1 && ! $1 -version >/dev/null 2>&1
then
echo "$1 not found"
NEEDED="$NEEDED $1"
else
# found, need to check version ?
if [ -z "$2" ];then
FOUND="$FOUND $1"
echo "found $1"
else
gotver=`$1 --version | head -1 | sed s/'.* '//`
check_version $1 $gotver $2
fi
fi
}

# check make
check mmark 2.1.1
check xml2rfc 2.30.0
# apt install xsltproc
check xsltproc 1.1.0
# apt install libxml2-utils
check xmllint 20903

cat > $BOOTSTRAP_MAKE << EOF
PREFIX=\$(abspath ./build)
EOF

echo "# calls the local or installed tool " > $RUNTIMES_MAKE

for t in $FOUND; do
echo ".$t:" >> $BOOTSTRAP_MAKE
echo "$t" | awk '{ tool = sprintf("%s_CALL := %s", toupper($0), $0); print tool }' >> $RUNTIMES_MAKE
done

for t in $NEEDED; do
echo .$t: .build$t >> $BOOTSTRAP_MAKE
PACKAGES="$PACKAGES $t"
TARGETS="$TARGETS .build$t"
if [ $t = "xml2rfc" ]; then
# installed in the Python local user dir
PYTHON_USER_PATH=$(python -m site --user-base)
echo PYTHON_USER_PATH=$PYTHON_USER_PATH >> $RUNTIMES_MAKE
echo "$t" | awk '{ tool = sprintf("%s_CALL := $(PYTHON_USER_PATH)/bin/%s", toupper($0), $0); print tool }' >> $RUNTIMES_MAKE
else
echo "$t" | awk '{ tool = sprintf("%s_CALL := ./%s", toupper($0), $0); print tool }' >> $RUNTIMES_MAKE
fi
done

[ -n "$PACKAGES" ] && echo "To-be-built packages: $PACKAGES"

case `uname` in
Linux)
MMARK_OS=linux
;;
Darwin)
MMARK_OS=darwin
;;
MINGW32*|MINGW64*|*MSYS*)
MMARK_OS=windows
;;
*)
echo Unsupported build OS `uname`
exit 1
;;
esac
case `uname -m` in
x86_64)
MMARK_MACHINE=amd64
;;
arm64)
MMARK_MACHINE=arm64
;;
arm)
MMARK_MACHINE=arm
;;
*)
echo Unsupported build CPU `uname -m`
exit 1
;;
esac

cat >> $BOOTSTRAP_MAKE << EOF
all: $TARGETS
@echo "You are ready to build Matroska specifications"
MMARK_VERSION=2.1.1
MMARK_OS=$MMARK_OS
MMARK_MACHINE=$MMARK_MACHINE
include tools.mak
EOF

make -f $BOOTSTRAP_MAKE
2 changes: 1 addition & 1 deletion chapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ These other `Elements` belong to the Matroska DVD menu system and are only used
- Soft Linking: In this complex system `Ordered Chapters` are REQUIRED and a `Chapter CODEC` MUST interpret the `ChapProcess` of all chapters.
- Medium Linking: `Ordered Chapters` are used in a normal way and can be combined with the `ChapterSegmentUID` element which establishes a link to another Matroska file/Segment.

See [the section on the `Linked Segments`](#linked-segments)) for more information about `Hard Linking`, `Soft Linking` and `Medium Linking`.
See [the section on the Linked Segments](#linked-segments)) for more information about `Hard Linking`, `Soft Linking` and `Medium Linking`.

## Menu features

Expand Down
6 changes: 3 additions & 3 deletions codec/av1.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ Within a protected `Block`, the following constraints apply to all the OBUs it c

* Encrypted partitions MUST span all complete 16-byte blocks of the __[decode_tile]__ structure (including any trailing bits).

* Bytes at the beginning of the __[decode_tile]__ that do not fit in the 16-bytes encrypted partitions SHOULD be added to the preceding unprotected partition. As a result the Encrypted partitions MAY NOT start at the first byte of the __[decode_tile]__ structure, but some number of bytes following that.
* Bytes at the beginning of the __[decode_tile]__ that do not fit in the 16-bytes encrypted partitions SHOULD be added to the preceding unprotected partition. As a result the Encrypted partitions might not start at the first byte of the __[decode_tile]__ structure, but some number of bytes following that.


# More TrackEntry mappings
Expand Down Expand Up @@ -217,7 +217,7 @@ The `BitsPerChannel` corresponds to the __[BitDepth]__.
### MatrixCoefficients
EBML Path: `\Segment\Tracks\TrackEntry\Video\Colour\MatrixCoefficients` | Mandatory: No

The `MatrixCoefficients` corresponds to the __[matrix_coefficients]__. Some values MAY NOT map correctly to values found in Matroska.
The `MatrixCoefficients` corresponds to the __[matrix_coefficients]__. Some values might not map correctly to values found in Matroska.

### ChromaSitingHorz
EBML Path: `\Segment\Tracks\TrackEntry\Video\Colour\ChromaSitingHorz` | Mandatory: No
Expand Down Expand Up @@ -245,7 +245,7 @@ The `TransferCharacteristics` corresponds to the __[transfer_characteristics]__.
### Primaries
EBML Path: `\Segment\Tracks\TrackEntry\Video\Colour\Primaries` | Mandatory: No

The `Primaries` corresponds to the __[color_primaries]__. Some values MAY NOT map correctly to values found in Matroska.
The `Primaries` corresponds to the __[color_primaries]__. Some values might not map correctly to values found in Matroska.

## Metadata OBU-based values

Expand Down
20 changes: 11 additions & 9 deletions codec_specs.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,9 @@ The following XML depicts the nested Elements of a `BlockGroup Element` with an
<BlockAdditions>
<BlockMore>
<BlockAddID>1</BlockAddID>
<BlockAdditional>{alpha channel encoding to supplement the VP9 frame}</BlockAdditional>
<BlockAdditional>
{alpha channel encoding to supplement the VP9 frame}
</BlockAdditional>
</BlockMore>
</BlockAdditions>
</BlockGroup>
Expand Down Expand Up @@ -667,39 +669,39 @@ Codec ID: S_TEXT/UTF8

Codec Name: UTF-8 Plain Text

Description: Basic text subtitles. For more information, please look at [the `Subtitles` section](#subtitles).
Description: Basic text subtitles. For more information, please look at [the Subtitles section](#subtitles).

### S_TEXT/SSA

Codec ID: S_TEXT/SSA

Codec Name: Subtitles Format

Description: The [Script Info] and [V4 Styles] sections are stored in the codecprivate. Each event is stored in its own Block. For more information, see [the SSA/ASS section in `Subtitles`](#subtitles).
Description: The [Script Info] and [V4 Styles] sections are stored in the codecprivate. Each event is stored in its own Block. For more information, see [the SSA/ASS section in Subtitles](#subtitles).

### S_TEXT/ASS

Codec ID: S_TEXT/ASS

Codec Name: Advanced Subtitles Format

Description: The [Script Info] and [V4 Styles] sections are stored in the codecprivate. Each event is stored in its own Block. For more information, see [the SSA/ASS section in `Subtitles`](#subtitles).
Description: The [Script Info] and [V4 Styles] sections are stored in the codecprivate. Each event is stored in its own Block. For more information, see [the SSA/ASS section in Subtitles](#subtitles).

### S_TEXT/USF

Codec ID: S_TEXT/USF

Codec Name: Universal Subtitle Format

Description: This is mostly defined, but not typed out yet. It will first be available on the USF specification [in `Subtitles`](#subtitles).
Description: This is mostly defined, but not typed out yet. It will first be available on the USF specification [in Subtitles](#subtitles).

### S_TEXT/WEBVTT

Codec ID: S_TEXT/WEBVTT

Codec Name: Web Video Text Tracks Format (WebVTT)

Description: Advanced text subtitles. For more information, see [the WebVTT section in `Subtitles`](#subtitles).
Description: Advanced text subtitles. For more information, see [the WebVTT section in Subtitles](#subtitles).

### S_IMAGE/BMP

Expand All @@ -715,7 +717,7 @@ Codec ID: S_DVBSUB

Codec Name: Digital Video Broadcasting (DVB) subtitles

Description: This is the graphical subtitle format used in the Digital Video Broadcasting standard. For more information, see [the Digital Video Broadcasting (DVB) section in `Subtitles`](#subtitles).
Description: This is the graphical subtitle format used in the Digital Video Broadcasting standard. For more information, see [the Digital Video Broadcasting (DVB) section in Subtitles](#subtitles).

### S_VOBSUB

Expand All @@ -733,15 +735,15 @@ Codec ID: S_HDMV/PGS

Codec Name: HDMV presentation graphics subtitles (PGS)

Description: This is the graphical subtitle format used on Blu-rays. For more information, see [HDMV text presentation in `Subtitles`](#subtitles).
Description: This is the graphical subtitle format used on Blu-rays. For more information, see [HDMV text presentation in Subtitles](#subtitles).

### S_HDMV/TEXTST

Codec ID: S_HDMV/TEXTST

Codec Name: HDMV text subtitles

Description: This is the textual subtitle format used on Blu-rays. For more information, see [HDMV graphics presentation in `Subtitles`](#subtitles).
Description: This is the textual subtitle format used on Blu-rays. For more information, see [HDMV graphics presentation in Subtitles](#subtitles).

### S_KATE

Expand Down
30 changes: 15 additions & 15 deletions diagram.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,21 +152,21 @@ The `Chapters Element` lists all of the chapters. Chapters are a way to set pred
| | | EditionFlagDefault |
| | |--------------------|
| | | EditionFlagOrdered |
| | |--------------------------------+
| | | ChapterAtom | ChapterUID |
| | | |------------------|
| | | | ChapterStringUID |
| | | |------------------|
| | | | ChapterTimeStart |
| | | |------------------|
| | | | ChapterTimeEnd |
| | | |------------------|
| | |---------------------------------+
| | | ChapterAtom | ChapterUID |
| | | |-------------------|
| | | | ChapterStringUID |
| | | |-------------------|
| | | | ChapterTimeStart |
| | | |-------------------|
| | | | ChapterTimeEnd |
| | | |-------------------|
| | | | ChapterFlagHidden |
| | | |---------------------------------+
| | | | ChapterDisplay | ChapString |
| | | | |--------------|
| | | | | ChapLanguage |
+--------------------------------------------------------------------+
| | | |-------------------------------+
| | | | ChapterDisplay | ChapString |
| | | | |--------------|
| | | | | ChapLanguage |
+------------------------------------------------------------------+
```
Figure: Representation of the `Chapters Element` and a selection of its `Descendant Elements`.

Expand Down Expand Up @@ -210,7 +210,7 @@ Figure: Representation of a `Cluster Element` and its immediate `Child Elements`
| | Data | Frame |
+----------------------------------+
```
Figure: Representation of the `Block Element` structure.
Figure: Representation of the `Block Element` structure.

Each `Cluster` MUST contain exactly one `Timestamp Element`. The `Timestamp Element` value MUST be stored once per `Cluster`. The `Timestamp Element` in the `Cluster` is relative to the entire `Segment`. The `Timestamp Element` SHOULD be the first `Element` in the `Cluster`.

Expand Down
Loading

0 comments on commit 20e8df3

Please sign in to comment.