Skip to content

Commit

Permalink
Fix etl-pipeline.adoc and prompt.adoc issues.
Browse files Browse the repository at this point in the history
  • Loading branch information
1993heqiang authored and Mark Pollack committed Nov 6, 2024
1 parent f111aae commit 4ad7650
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -515,7 +515,7 @@ The `TextSplitter` an abstract base class that helps divides documents to fit th


=== TokenTextSplitter
The `TokenTextSplitter` is an implementation of `TextSplitter` that splits text into chunks based on token count, using the `CL100K_BASE encoding.
The `TokenTextSplitter` is an implementation of `TextSplitter` that splits text into chunks based on token count, using the CL100K_BASE encoding.

==== Usage

Expand Down Expand Up @@ -593,6 +593,7 @@ for (Document doc : splitDocuments) {
* Metadata from the original documents is preserved and copied to all chunks derived from that document.
* The content formatter (if set) from the original document is also copied to the derived chunks if `copyContentFormatter` is set to `true` (default behavior).
* This splitter is particularly useful for preparing text for large language models that have token limits, ensuring that each chunk is within the model's processing capacity.

=== ContentFormatTransformer
Ensures uniform content formats across all documents.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ More than just an answer or reaction, it's crucial for maintaining the flow of t
By tracking the AI's previous responses (its 'Assistant Role' messages), the system ensures coherent and contextually relevant interactions.
The Assistant message may contain Function Tool Call request information as well.
It's like a special feature in the AI, used when needed to perform specific functions such as calculations, fetching data, or other tasks beyond just talking.
* Tool/Function Role: The Too/Function Role focuses on returning additional information in response to Tool Call Assistant Messages.
* Tool/Function Role: The Tool/Function Role focuses on returning additional information in response to Tool Call Assistant Messages.

Roles are represented as an enumeration in Spring AI as shown below

Expand Down

0 comments on commit 4ad7650

Please sign in to comment.