Skip to content

Commit

Permalink
[reduced-operator-config-file.md] Add documentation about globally al…
Browse files Browse the repository at this point in the history
…lowed types.
  • Loading branch information
edgchen1 authored Feb 6, 2024
1 parent bd63813 commit 03cf68e
Showing 1 changed file with 33 additions and 6 deletions.
39 changes: 33 additions & 6 deletions docs/reference/operators/reduced-operator-config-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,24 +62,51 @@ Additionally, the ONNX operator specs for [DNN](https://github.com/onnx/onnx/blo

## Type reduction format

### Per-operator type information

If the types an operator implementation supports can be limited to a specific set of types, this is specified in a JSON string immediately after the operator name in the configuration file.

**It is highly recommended that you first generate the configuration file using ORT format models with type reduction enabled in order to see which operators support type reduction, and how the entry is defined for the individual operators.**

The required types are generally listed per input and/or output of the operator. The type information is in a map, with 'inputs' and 'outputs' keys. The value for 'inputs' or 'outputs' is a map between the index number of the input/output and the required list of types.

For example, both the input and output types are relevant to ai.onnx:Cast. Type information for input 0 and output 0 could look like this:
`{"inputs": {"0": ["float", "int32_t"]}, "outputs": {"0": ["float", "int64_t"]}}`

which is added directly after the operator name in the configuration file.
e.g.
`ai.onnx;12;Add,Cast{"inputs": {"0": ["float", "int32_t"]}, "outputs": {"0": ["float", "int64_t"]}},Concat,Squeeze`
```
{"inputs": {"0": ["float", "int32_t"]}, "outputs": {"0": ["float", "int64_t"]}}
```

which is added directly after the operator name in the configuration file. E.g.:

```
ai.onnx;12;Add,Cast{"inputs": {"0": ["float", "int32_t"]}, "outputs": {"0": ["float", "int64_t"]}},Concat,Squeeze
```

If, for example, the types of inputs 0 and 1 were important, the entry may look like this (e.g. ai.onnx:Gather):
`{"inputs": {"0": ["float", "int32_t"], "1": ["int32_t"]}}`

```
{"inputs": {"0": ["float", "int32_t"], "1": ["int32_t"]}}
```

Finally some operators do non-standard things and store their type information under a 'custom' key.
ai.onnx.OneHot is an example of this, where the three input types are combined into a triple.
`{"custom": [["float", "int64_t", "int64_t"], ["int64_t", "std::string", "int64_t"]]}`

```
{"custom": [["float", "int64_t", "int64_t"], ["int64_t", "std::string", "int64_t"]]}
```

For these reasons, it is best to generate the configuration file first, and manually edit any entries if needed.

### Globally allowed types

It is also possible to limit the types supported by all operators to a specific set of types. These are referred to as *globally allowed types*. They may be specified in the configuration file on a separate line.

The format for specifying globally allowed types for all operators is:

```
!globally_allowed_types;T0,T1,...
```

`Ti` should be a C++ scalar type supported by ONNX and ORT. At most one globally allowed types specification is allowed.

Specifying per-operator type information and specifying globally allowed types are mutually exclusive - it is an error to specify both.

0 comments on commit 03cf68e

Please sign in to comment.