Deduplication of text chunks with frequency count, training and encoding 5x speedup

[Majdoddin]

In RegexTokenizer, the training text is initially split into chunks, and further processing is performed on individual chunks. This PR optimizes the process by retaining only unique chunks and their corresponding frequency counts. Practically this cuts the number of chunks to 1/7th, resulting in a training speedup of at least 5x.

Similar optimization for encode_ordinary(), where the tokenization of each string is cached. Also 5x speedup.

[ae99]

Independently arrived at this same change myself!

Took training of a 8192 size tokenizer on ~3m words from 8 hours, down to 20mins. I'll likely now re-train on ~1b words given this makes training almost entirely independent of dataset size after the regex is complete. Makes this repo production viable!

[ae99]

Faster to do the counting ahead of converting to unicode

Suggested change

	text_chunks = re.findall(self.compiled_pattern, text)
	text_chunks = re.findall(self.compiled_pattern, text)
	chunks_counted = collections.Counter(text_chunks)
	text_chunks = [chunk for chunk, count in chunks_counted.items()]
	global_counts = [count for chunk, count in chunks_counted.items()]

Then further down we can just go:

            for chunk_ids, global_count in zip(ids, global_counts):
                # passing in stats will update it in place, adding up counts
                get_stats_n(chunk_ids, stats, global_count)