Exploring Historical Web Content / Internet Censorship Data

[NullPxl]

Hi! I recently used BERTopic as a large part of an approach to discover themes in multilingual web content. More specifically the project looks at millions of tests/URLs submitted to an internet censorship measurement platform across 11 years.

You can explore the data and see more details here: https://whatscensored.peterwhiting.me/

It's not perfect-- stopwords still sometimes appear in the labels, there are overlapping topics, and many outliers. That said, it still shows many interesting themes and I believe it's an interesting example of how BERTopic can be applied. I thought anyone looking at the bertopic discussions page might find it cool!

[MaartenGr]

This is an awesome project, thank you for sharing!! As a quick tip, you can easily remove stopwords by using MMR, KeyBERTInspired or use the reduce_frequent_words params in the ClassTfidfTransformer

[NullPxl]

Thanks Maarten :) I'm currently using KeyBERTInspired + MMR for representations, but I'll definitely try out the reduce_frequent_words param. I checked my code and it looks like I tried it at one point and wrote in a comment to myself that it worked well-- but then proceeded to not use it, whoops!