Improve testing and try to fix MySQL hanging

[wxiaoguang]

It's strange that TestLinguist hangs a lot with MySQL (and only MySQL). Update: due to MySQL tests have different indexer&queue behaviors.

Now, by some CI fine tunes (run tests), SQLite & MSSQL could complete in about 12~13 minutes (before > 14), MySQL could complete in 18 minutes (before: about 23 or even > 30)

Major changes:

1. use tmpfs for MySQL storage
2. run make test-mysql instead of make integration-test-coverage because the code coverage is not really used at the moment.
3. refactor testlogger to make it more reliable and be able to report stuck stacktrace
4. do not requeue failed items when a queue is being flushed (failed items would keep failing and make flush uncompleted)
5. reduce the file sizes for testing
6. use math ChaCha8 random data instead of crypot/rand (for testing purpose only)
7. no need to DeleteRepository in TestLinguist
8. other related refactoring to make code easier to maintain

[wxiaoguang]

By some CI fine tunes (run tests), SQLite & MSSQL could complete in about 12~13 minutes (before > 14), MySQL could complete in 18 minutes (before: about 23 or even > 30)

[bohde]

Am I right that the major changes are as follows?

• Use ChaCha8 for generating random test files instead of a cryptographic generator, saving 2 minutes per integration test suite
• Adjust MySQL to run on a tmpfs, saving 4 minutes per mysql integration test suite
• Don't re-enqueue failed jobs when flushing, which makes tests more reliable

[wxiaoguang]

Am I right that the major changes are as follows?

* Use ChaCha8 for generating random test files instead of a cryptographic generator, saving 2 minutes per integration test suite

* Adjust MySQL to run on a tmpfs, saving 4 minutes per mysql integration test suite

* Don't re-enqueue failed jobs when flushing, which makes tests more reliable

Exactly, these are the major changes, and by the way remove the "deleting repo" code in TestLinguist, it also saves a little time.

And fine-tune the testlogger to make it output correct durations, and report the stack if a test gets stuck.

[bohde]

Exactly, these are the major changes, and by the way remove the "deleting repo" code in TestLinguist, it also saves a little time.

And fine-tune the testlogger to make it output correct durations, and report the stack if a test gets stuck.

Thanks for the confirmation!

[wxiaoguang]

Thanks for the confirmation!

Yup, a lot of things are mixed in one PR .... (I haven't thought that it would become more and more complex .....)

Updated the description of the PR, hopefully everything is listed there.