Add integration testing support that uses the sample repositories

[gkapfham]

Issue #192 explains that we want to create new repositories that contain examples that are organized in a new fashion and that adhere to a new naming convention.

Once all of these repositories exist, then it is possible to implement a new approach to integration testing. The idea would be that we can use a tool to clone all of the example repositories when we are testing new features of GatorGrader that are currently in a PR.

We would know that a repository like this one:

https://github.com/GatorEducator/java-assignment-starter-100-01

is a starter repository and thus, by design, the checks for this repo should fail, producing a non-zero exit code.

Alternatively, if we were running gradle grade in this repository:

https://github.com/GatorEducator/java-assignment-solution-100-01

then since it is a solution repository, all of the checks should pass, producing a zero exit code.

What I would like to have is a new approach to integration testing that makes it possible to run these checks automatically for all of the existing sample repositories that follow the new naming convention.

[simojo]

I'm working on a bash script to get a proof of concept for this. It will clone down each repo in GatorEducator and run gradle grade on it. I'm starting with the GitHub API v3.

[simojo]

There is a command line tool called jq for parsing JSON. I was using it for data returned from curl but was wondering what our approach is on external tools. @gkapfham Should I stay away from using command line tools for testing on travis? If so, how should I go about parsing JSON via bash/batch?

[gkapfham]

Hi @sjones-gh, I think that it is fine if you want to use jq to parse the JSON files when you are running a program in Travis CI. Will you be calling the jq program from a Python program that you implement? Or, will you simply be calling it from a script in Travis CI? Overall, the main challenge that you may run into when it comes to using external programs like jq is that they may not be platform independent and thus there will not be an easy way to run that program through AppVeyor or on a Windows computer.

[simojo]

@gkapfham the original plan was to call it using a shell script, but if I find that it has a tangled mess of dependencies, I’ll see if there’s some python lib for this. I would like to avoid leaning on yet another python library, though.

[gkapfham]

I'm sure that there are multiple Python libraries that would solve this issue. You would need to add it as a dev dependency in pipenv.