[RFC] Refactor integration testing

[chshersh]

The way tests are implemented currently, it requires lots of manual changes to the CI config. Because of that, the CI configuration is getting bigger, noisy, harder to maintain and error-prone. It also contains lots of boilerplate to support cross-platform tests:

Just look at these 100 lines of YAML 😨

tool-sync/.github/workflows/ci.yml

Lines 43 to 141 in 2bb99cf

	- if: matrix.os != 'windows-latest'
	name: "Integration test: [unix] [install ripgrep]"
	env:
	SYNC_DIR: "install-ripgrep"
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	run: |
	mkdir $SYNC_DIR
	cargo run -- --config=tests/$SYNC_DIR.toml install ripgrep
	ls -l $SYNC_DIR
	if [[ ! -x $SYNC_DIR/rg ]]; then echo "error on: rg"; false; fi
	- if: matrix.os != 'windows-latest'
	name: "Integration test: [unix] [only-ripgrep]"
	env:
	SYNC_DIR: "only-ripgrep"
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	run: |
	mkdir $SYNC_DIR
	cargo run -- --config=tests/$SYNC_DIR.toml sync
	ls -l $SYNC_DIR
	if [[ ! -x $SYNC_DIR/rg ]]; then echo "error on: rg"; false; fi
	- if: matrix.os == 'windows-latest'
	name: "Integration test: [windows] [only-ripgrep]"
	env:
	SYNC_DIR: "only-ripgrep"
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	run: |
	mkdir $env:SYNC_DIR
	cargo run -- --config="tests\$env:SYNC_DIR.toml" sync
	ls -l $env:SYNC_DIR
	if (!(Test-Path $env:SYNC_DIR\rg.exe)) {
	throw 'error on rg.exe'
	}
	- if: matrix.os != 'windows-latest'
	name: "Integration test: [unix] [full-database]"
	env:
	SYNC_DIR: "full-database"
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	run: |
	mkdir $SYNC_DIR
	cargo run -- --config=tests/$SYNC_DIR.toml sync
	ls -l $SYNC_DIR
	if [[ ! -x $SYNC_DIR/bat ]]; then echo "error on: bat"; false; fi
	if [[ ! -x $SYNC_DIR/difft ]]; then echo "error on: difft"; false; fi
	if [[ ! -x $SYNC_DIR/exa ]]; then echo "error on: exa"; false; fi
	if [[ ! -x $SYNC_DIR/fd ]]; then echo "error on: fd"; false; fi
	if [[ ! -x $SYNC_DIR/rg ]]; then echo "error on: rg"; false; fi
	if [[ ! -x $SYNC_DIR/tool ]]; then echo "error on: tool"; false; fi
	# disabled because of: https://github.com/alexcrichton/tar-rs/issues/295
	# if [[ ! -x $SYNC_DIR/tokei ]]; then echo "error on: tokei"; false; fi
	- if: matrix.os != 'windows-latest'
	name: "Characterization test: [unix] [default-config]"
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	run: |
	cargo run -- default-config > tools.toml
	diff tools.toml tests/default-config.toml
	if [[ $? == 1 ]]; then diff tools.toml tests/default-config.toml; false; fi
	- if: matrix.os == 'windows-latest'
	name: "Integration test: [windows] [full-database]"
	env:
	SYNC_DIR: "full-database"
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	run: |
	mkdir $env:SYNC_DIR
	cargo run -- --config="tests\$env:SYNC_DIR.toml" sync
	ls -l $env:SYNC_DIR
	if (!(Test-Path $env:SYNC_DIR\bat.exe)) {
	throw 'error on bat.exe'
	}
	if (!(Test-Path $env:SYNC_DIR\difft.exe)) {
	throw 'error on difft.exe'
	}
	if (!(Test-Path $env:SYNC_DIR\fd.exe)) {
	throw 'error on fd.exe'
	}
	if (!(Test-Path $env:SYNC_DIR\rg.exe)) {
	throw 'error on rg.exe'
	}
	if (!(Test-Path $env:SYNC_DIR\tool.exe)) {
	throw 'error on tool.exe'
	}

I propose

Option 1: Move to bash scripts

A simple option would be to move those commands into Shell scripts (Bash and PowerShell). This doesn't solve the problem of having lots of boilerplate but at least it makes the CI config less noisy (although at the cost of having more indirections).

Pros

✅ Less noise in the CI config
✅ Pretty simple shell scripting and not so much of the code if you're familiar with shell
✅ Testing CLI parsing as well

Cons

❌ Requires more polyglot knowledge
❌ Requires lots of manual updates
❌ Boilerplate
❌ Requires separate static analysis like shellcheck

Option 2: Rust integration tests

Alternatively, all these tests can be written as rust integration tests. They can be put into the tests/ directory and run as cargo test (which we already do)

Pros

✅ No polyglot env, only Rust for testing
✅ No boilerplate, only one test suite, reusing lib utils
✅ Automatically cross-platform
✅ All Rust tooling works for tests automatically

Cons

❌ More code
❌ No CLI parsing testing (maybe it can be done separately with clap?)