Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove <br> tags in html #618

Merged
merged 2 commits into from
Oct 1, 2024
Merged

Remove <br> tags in html #618

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d4de1ae
remove <br> tags in html
addie9800 Sep 24, 2024
701556b
remove html modification
addie9800 Oct 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions src/fundus/publishers/na/the_namibian.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@
from datetime import datetime
from typing import List, Optional, Pattern

import lxml.html
from lxml.etree import XPath

from fundus.parser import ArticleBody, BaseParser, ParserProxy, attribute
from fundus.parser import ArticleBody, BaseParser, ParserProxy, attribute, function
from fundus.parser.base_parser import Precomputed
from fundus.parser.utility import (
extract_article_body_with_selector,
generic_author_parsing,
generic_date_parsing,
get_ld_content,
get_meta_content,
)


Expand Down Expand Up @@ -44,5 +48,10 @@ def authors(self) -> List[str]:

class V1_1(V1):
VALID_UNTIL = datetime.today().date()
_paragraph_selector = XPath("//div[contains(@class, 'entry-content')]/p[position()>1]")
_summary_selector = XPath("//div[contains(@class, 'entry-content')]/p[position()=1]")
_paragraph_selector = XPath("//div[contains(@class, 'entry-content')]/p[(text() or strong) and position()>1]")
_summary_selector = XPath("//div[contains(@class, 'entry-content')]/p[(text() or strong) and position()=1]")

def _base_setup(self, html: str) -> None:
html = re.sub(r"(<br>)+", "<p>", html)
doc = lxml.html.document_fromstring(html)
self.precomputed = Precomputed(html, doc, get_meta_content(doc), get_ld_content(doc))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things here:

  1. The intended way to do something like this would be to use the function decorator instead of overwriting the _base_setup
  2. html should still reflect the original HTML. We have to find some way around this.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to only modify the html for the purpose of extracting the body. This way it's also not necessary to overwrite the HTML.

3 changes: 2 additions & 1 deletion tests/resources/parser/test_data/na/TheNamibian.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,8 @@
"He said he, Mbumba and Geingob planned to settle at Swakopmund and to sometimes walk barefoot by the seaside and have coffee at a nearby café.",
"GOING HOME TO DIE",
"Political analyst Henning Melber questions the circumstances surrounding Geingob’s departure, suggesting that if an inner circle was aware of his imminent passing, his trip to the US wouldn’t have been for medical treatment.",
"Melber speculates that Geingob may have realised his life was ending during his time in the US and chose to return home to die.However, Melber questions why this should be a contentious issue.",
"Melber speculates that Geingob may have realised his life was ending during his time in the US and chose to return home to die.",
"However, Melber questions why this should be a contentious issue.",
"“This means that as of his arrival back in Namibia people may have become aware that not much time was left. But why make an issue of it?",
"“If one could provide evidence that it was exploited for dubious arrangements, it would be another matter. But I do not see any indications pointing reliably into such a direction,” he says."
]
Expand Down