Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maximum length for the markdown exporter #33

Open
dolfim-ibm opened this issue Sep 26, 2024 · 0 comments
Open

Maximum length for the markdown exporter #33

dolfim-ibm opened this issue Sep 26, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@dolfim-ibm
Copy link
Contributor

When stuffing content to LLMs you want to get the biggest chunk that fits in the maximum context window.

A trivial implementation could simply do content_md = doc.export_to_markdown()[:MAX_LEN]. However, this could break sentences, paragraphs, tables, etc.

If we introduce a max_size in the export_to_markdown() function, we could control it by adding a document item only if it doesn't go above the limit.

@dolfim-ibm dolfim-ibm added the enhancement New feature or request label Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant