New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: improve sentence splitting #72

Merged

lsorber merged 1 commit into main from ls-better-sentences

Jan 2, 2025

Member

lsorber commented Dec 30, 2024

Changes:

Improve sentence splitting with better treating of Markdown headings. The start and end of a Markdown heading are now properly marked as the start of a new sentence, while its body is marked as not containing any sentence boundaries. These are supplied as hints to spaCy to improve overall output quality.
Sentence splitting is now explicitly covered by tests.


          feat: improve sentence splitting

a365ece

lsorber self-assigned this

lsorber merged commit c41c6f8 into main

2 checks passed

lsorber deleted the ls-better-sentences branch

January 2, 2025 14:32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet