Fix bad unicode character issue

[koechkevin]

Description

• Fix an issue with our initial implementation of html2text where meedan returns bad unicode character error.
• Replaces html2text with trafilatura.
• A formatted input from this post would now be processed as
  output.txt

Fixes (issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

Screenshots

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation

[kilemensi]

Before review the code, what is the actual issue here @koechkevin ? Meedan Check doesn't accept links?

[koechkevin]

@kilemensi How html2text handles some href links created some invalid unicode characters causing the error. Of all the options mentioned in the documentation, ignore_links is the only that seemed to work for the specific case. Maybe another edge case will be introduced with other uploads

[kilemensi]

The option may work but I still don't understand the issue @koechkevin: Why would links raise unicode error? May be there is actually unicode error in the original article itself?

1. Does Meedan Check expect unicode or ascii?
2. Is there a maximum length limit?

If Meedan Check accepts unicode,

1. Have we tried the unicode_snob option?
2. If this didn't work, did we try manually encoding e.g. html2text(html).encode("utf-8", errors="replace").decode("utf-8", errors="ignore")?

[koechkevin]

@kilemensi this is the encoded url that causes the issues after parse by html2text https://www.reuters.com/world/middle-east/egypt-steps-up-security-border-israeli-offensive-gaza-nears-2024-02-09/#:~:text=CAIRO%2C%20Feb%209%20\(Reuters\),two%20Egyptian%20security%20sources%20said.

[kilemensi]

@kilemensi this is the encoded url that causes the issues after parse by html2text https://www.reuters.com/world/middle-east/egypt-steps-up-security-border-israeli-offensive-gaza-nears-2024-02-09/#:~:text=CAIRO%2C%20Feb%209%20(Reuters),two%20Egyptian%20security%20sources%20said.

But this doesn't seem to have any unicode issues @koechkevin ... as a matter of fact, it's actually just ascii. How does it look after being parsed by html2text?

[koechkevin]

@kilemensi this is the encoded url that causes the issues after parse by html2text https://www.reuters.com/world/middle-east/egypt-steps-up-security-border-israeli-offensive-gaza-nears-2024-02-09/#:~:text=CAIRO%2C%20Feb%209%20(Reuters),two%20Egyptian%20security%20sources%20said.

But this doesn't seem to have any unicode issues @koechkevin ... as a matter of fact, it's actually just ascii. How does it look after being parsed by html2text?
Yes, no unicode issue @kilemensi but html2text adds escape characters around brackets on(Reuters) which is rejected by meedan.

[koechkevin]

Egypt

[koechkevin]

@kilemensi based on caiosba's response, I suggest we go with this implementation because html2text implements links as markdown

[kilemensi]

@kilemensi based on caiosba's response, I suggest we go with this implementation because html2text implements links as markdown

I think caiosba's saying links are important @koechkevin :

for the URLs, since those can be sent to end users on the tiplines, I suggest you simplify them and remove the fragments, in order to guarantee that WhatsApp linkify them correctly

If by current implementation you mean removing links then doesn't that mean URLs won't be sent to the end users?

[koechkevin]

@kilemensi based on caiosba's response, I suggest we go with this implementation because html2text implements links as markdown

I think caiosba's saying links are important @koechkevin :

for the URLs, since those can be sent to end users on the tiplines, I suggest you simplify them and remove the fragments, in order to guarantee that WhatsApp linkify them correctly

If by current implementation you mean removing links then doesn't that mean URLs won't be sent to the end users?

@kilemensi yes

[koechkevin]

@kilemensi I have updated this PR.

As per this,

Links are useful and content should be plaintext (No markdown). However, since html2text only converts to markdown, I have used Beautiful Soup to convert this to text appended links(href & img src) so we won't be having bad unicode errors anymore.

[kilemensi]

Links are useful and content should be plaintext (No markdown). However, since html2text only converts to markdown, I have used Beautiful Soup to convert this to text appended links(href & img src) so we won't be having bad unicode errors anymore.

beatufulsoup output isn't very good @koechkevin... lets just pull the big gun out and use trafilatura then.

[koechkevin]

Links are useful and content should be plaintext (No markdown). However, since html2text only converts to markdown, I have used Beautiful Soup to convert this to text appended links(href & img src) so we won't be having bad unicode errors anymore.

beatufulsoup output isn't very good @koechkevin... lets just pull the big gun out and use trafilatura then.

@kilemensi this is updated

[kilemensi]

@kilemensi this is updated

Can you include an output sample in the PR description @koechkevin ?

[kilemensi]

Is output.txt the current output @koechkevin? Asking because I see links so not sure if this is better than html2text or not.

[koechkevin]

Is output.txt the current output @koechkevin? Asking because I see links so not sure if this is better than html2text or not.

@kilemensi Yes, output.txt is the current implementation. It is better than html2text because, it does not process the whole html as a markdown but only parts with the links are formatted like markdown and no bad unicode characters included.