[ENG-3848][ENG-3861]Shiki Code block Experimental

[ElijahAhianyo]

High-level API

rx._x.code_block(
        """
     console.log('hewwo') 
     console.log('hello')
     console.log('goodbye') 
  """,
       language="typescript",
       theme="github-dark",
)

With copy button

rx._x.code_block(
        """
     console.log('hewwo') 
     console.log('hello')
     console.log('goodbye') 
  """,
       language="typescript",
       theme="github-dark",
       can_copy=True
 )

Using transformers

rx._x.code_block(
        """
    console.log('hewwo') // [!code --]
     console.log('hello') // [!code ++]
     console.log('goodbye') // [!code highlight]
  """,
       language="typescript",
       use_transformers=True,
       theme="github-dark",
 )

Low-level API

Shikijs transformer

The shiki transformer is provided via code_block.transformers API.

rx._x.code_block.root(
        """
    console.log('hewwo') // [!code --]
    console.log('hello') // [!code ++]
    console.log('goodbye') // [!code highlight]
  """,
       language="typescript",
       transformers=[rx._x.code_block.transformers.shikijs(fns=["transformerNotationDiff", "transformerNotationHighlight"])],
       theme="github-dark",
 )

Custom transformer

In using custom transformers from third-party libs, we provide a code_block.create_transformer API that allows you to specify the library and functions to call from the lib.

rx._x.code_block.root(
        """
    console.log('hewwo') // [!code --]
    console.log('hello') // [!code ++]
    console.log('goodbye') // [!code highlight]
  """,
       language="typescript",
       transformers=[
                rx._x.code_block.create_transformer( # using a transformer from a third party library
                          library="shiki-transformer-color-highlight",
                          fns=["transformerColorHighlight"], 
                          style={...}
                  ),
                 {
                      code(node) { # lifted this off https://shiki.matsu.io/guide/transformers#transformers
                        this.addClassToHast(node, 'language-js')
                      },
                      line(node, line) {
                        node.properties['data-line'] = line
                        if ([1, 3, 4].includes(line))
                          this.addClassToHast(node, 'highlight')
                      },
                      span(node, line, col) {
                        node.properties['data-token'] = `token:${line}:${col}`
                      },
                },
         ],
         theme="github-dark",
 )

Some Notes:

• The styling for provided transformer(code_block.transformers.shikijs ) needs to be improved. The bg colors for the diffs overflow out of the div as well as the line numbers.
• Styles for other transformer fns in the provided transformer need to be provided (eg, for highlighting, etc)
• copy button needs to be improved

[Alek99]

Could we just include transformers internally in the component so they dont have to define this. Just have all the transformers in there

That way we can avoid this line.
transformers=[rx._x.code_block.transformers.shikijs(fns=["transformerNotationDiff", "transformerNotationHighlight"])],

Or have transformers be a boolean where it's disabled by default but they can enable it with some default styles. Then have a separate prop called custom_transformer where they can add customized ones. I think the current code is a a bit too low level for an end user

[linear]

ENG-3848 Change Syntax Highlighter to Shiki

ENG-3861 Showing Diffs + Code Highlighting

[ElijahAhianyo]

This is more of a hack to get reflex web to run. somehow, a list is sometimes passed as the value of code. Have a tracking ticket to look into that https://linear.app/reflex-dev/issue/ENG-3893/get-shiki-to-work-with-reflex-web

[picklelo]

Does the entire style dictionary get compiled in every time you call it? I wonder if we need to do some optimizations around that