mode to create standalone test output automatically [WIP]

[agroce]

See #208

[agroce]

It sort of works now.

Given:

#include <deepstate/DeepState.hpp>

using namespace deepstate;

#include <assert.h>

int foo(int a, int b, char* c) {
  if (a == b) {
    return c[0] == c[1];
  } else {
    return (a > b);
  }
}

TEST(UnitTests, Foo) {
  int a = DeepState_Int();
  int b = DeepState_Int();
  char* c = DeepState_CStrUpToLen(3);
  int v = foo(a, b, c);
  assert(v);
}

and the config file foo.config:

foo

doing:

deepstate-generate-standalone-output SimpleStandalone.cpp NewSimple.cpp foo.config

outputs:

ADDING STANDALONE TEST GENERATION CODE FOR foo
ADDING TEST GENERATION CODE FOR:   int v = foo(a, b, c);
  LOG(TRACE) << "/* START CODE: */  int v = foo(" << DeepState_Standalone_Wrap(a) << "," << DeepState_Standalone_Wrap( b) << "," << DeepState_Standalone_Wrap( c) << "); // STANDALONE UNIT TEST CODE";

ADDING TEST GENERATION CODE FOR:   assert(v);
  LOG(TRACE) << "/* START CODE: */  assert(" << "v" << "); // STANDALONE UNIT TEST CODE";

and produces the harness:

#include <deepstate/DeepState.hpp>

using namespace deepstate;

#include <assert.h>

int foo(int a, int b, char* c) {
  if (a	== b) {
    return c[0] == c[1];
  } else {
    return (a > b);
  }
}

TEST(UnitTests, Foo) {
  int a = DeepState_Int();
  int b = DeepState_Int();
  char* c = DeepState_CStrUpToLen(3);
  LOG(TRACE) << "/* START CODE: */  int v = foo(" << DeepState_Standalone_Wrap(a) << "," << DeepState_Standalone_Wrap( b) << "," << DeepState_Standalone_Wrap( c) << "); // STANDALONE UNIT TEST CODE";
  int v = foo(a, b, c);
  LOG(TRACE) << "/* START CODE: */  assert(" << "v" << "); // STANDALONE UNIT TEST CODE";
  assert(v);
}

which compiles.

[agroce]

Things this still needs:

• new LOG type that doesn't print all the junk in:

TRACE: NewSimple.cpp(19): /* START CODE: */  int v = foo(0,0,""); // STANDALONE UNIT TEST CODE
TRACE: NewSimple.cpp(21): /* START CODE: */  assert(v); // STANDALONE UNIT TEST CODE

Right now you can just grap what's between /* START CODE: */ and // STANDALONE UNIT TEST CODE but it's ugly.

Also the only wrappers provided are for int, unsigned int, and char*, but that's easy to add to, except for things like pointers and structs (right now, you'll generate that won't compile, so you'll know it doesn't work, if not in the nicest way).

[agroce]

Ok, now you can just scan through test execution and grab everything between /* START STANDALONE CODE */ and /* END STANDALONE CODE */

[agroce]

The approach taken here is quick'n'dirty: it would be ideal to have a tool that just took the harness code and the input file and somehow made a standalone test! This solution is usable for some people, but is not long-term the right thing. Getting it working could help with understanding how to do it The Right Way, though.

[pgoodman]

@agroce I fixed up the merge failure I think, can we merge this?

[agroce]

@pgoodman whoa -- does this work? The students did a big, complex implementation that seems to sort of work, this is just mostly a hacky kluge I'd sort of forgotten about

Did you try it out?

[pgoodman]

Did not try it. Should we prioritize the student work?

[agroce]

The student work was MUCH more powerful and thorough, handled a lot of things. The reason I haven't integrated it is that it is also big and complex, and needed a decent code review/survey of how it changes build and installation, etc., and I never found the time, since it is thorough (uses antlr parsing to do stuff). If we think standalone generation is important, it's probably a large (if heavyweight) step in that direction. There is definitely something attractive in the workflow they showed, where you write a deepstate harness, and run AFL on it, and then process the AFL queue directory and basically have AFL writing a bunch of plain old GoogleTest unit tests for you.