diff --git a/README.md b/README.md index 851976e74..9603be062 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ per developer. CI usage for licensed developers is included. Opensource projects have to acquire their free license per repository. -The license distribution happens through the `mutant-license` gem in mutants dependencies. +The license distribution happens through the `mutant-license` gem in mutant’s dependencies. This gem is dynamically generated per licensee and comes with a unique license gem source URL. @@ -99,8 +99,7 @@ end The mutant license gem contains metadata that allows mutant to verify licensed use. -For commercial licenses mutant checks the git commit author or the configured git email -to be in the set of licensed developers. +For commercial licenses mutant checks the git commit author or the configured git email to be in the set of licensed developers. For opensource licenses mutant checks the git remotes against the licensed git repositories. This allows the project maintainer to sign up and not bother collaborators with the details. @@ -177,8 +176,7 @@ of their private time. Additionally, the following features where sponsored by organizations: * The `mutant-minitest` integration was sponsored by [Arkency](https://arkency.com/) -* Mutant's initial concurrency support was sponsored by an undisclosed company that does - currently not wish to be listed here. +* Mutant's initial concurrency support was sponsored by an undisclosed company that does currently not wish to be listed here. ### Legal diff --git a/docs/commercial-support.md b/docs/commercial-support.md index 851e17344..aaf5a8d7e 100644 --- a/docs/commercial-support.md +++ b/docs/commercial-support.md @@ -10,5 +10,5 @@ Covers 1 incident per quarter, with a max response time of 7 days. Scope is limited to mutant not the application or infrastructure. For support email [Markus Schirp](mailto:mbj@schirp-dso.com?subject=Mutant%20Support). -Please email using the same domain as the roiginal license email or explain +Please email using the same domain as the original license email or explain your connection to the license. diff --git a/docs/limitations.md b/docs/limitations.md index 181177061..42bcc48c7 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -6,8 +6,8 @@ Subject Mutant cannot emit mutations for some subjects. -* methods defined within a closure. For example, methods defined using `module_eval`, `class_eval`, - `define_method`, or `define_singleton_method`: +* methods defined within a closure. For example, methods defined using + `module_eval`, `class_eval`, `define_method`, or `define_singleton_method`: ```ruby class Example @@ -29,7 +29,7 @@ Mutant cannot emit mutations for some subjects. end ``` -* singleton methods not defined on a constant or `self` +* singleton methods not defined on a constant or on `self` ```ruby class Foo @@ -38,6 +38,15 @@ Mutant cannot emit mutations for some subjects. myself = self def myself.qux; end # cannot mutate + + class << self + def corge; end # ok + end + end + + foo = "Hello" + class << foo + def greet!; end # cannot mutate end ``` diff --git a/docs/mutant-rspec.md b/docs/mutant-rspec.md index db0b41063..482ca8aca 100644 --- a/docs/mutant-rspec.md +++ b/docs/mutant-rspec.md @@ -21,7 +21,7 @@ To add mutant to your rspec code base you need to: ## Run through example -This uses [mbj/auom](https://github.com/mbj/auom) a small library that +This uses [mbj/auom](https://github.com/mbj/auom), a small library that has 100% mutation coverage. Its tests execute very fast and do not have any IO so its a good playground example to interact with. @@ -128,3 +128,5 @@ these example groups *must* kill the mutation. ```sh RAILS_ENV=test bundle exec mutant run -r ./config/environment --use rspec User ``` + +TODO something about using `mutant_expression` metadata here. \ No newline at end of file diff --git a/docs/nomenclature.md b/docs/nomenclature.md index a92393269..550680c87 100644 --- a/docs/nomenclature.md +++ b/docs/nomenclature.md @@ -1,13 +1,33 @@ Nomenclature ============ -The following explains several nouns you may experience in mutant's documentation. -It's a good idea to familiarize yourself before moving on. +This document explains several nouns you may experience in mutant's +documentation. It's a good idea to familiarize yourself before moving on. + +## Mutation Testing +The practice of systematically applying small changes one at a time to a +codebase then re-running the (relevant) tests for each change. If the tests +fail, then they cover the semantics of those changes. If the tests continue to +pass with the changes in place, the tests do not cover the complete semantics of +the changed parts of the codebase. + +Each type of change (for example, `a && b` to `a`) is known as a +[Mutation](#mutation), and is applied to the pristine codebase - that is, the +changes are not stacked up, but applied in isolation. The places in the code +where the changes can be made are called [Subjects](#subject). The changed +Subject is then referred to (potentially confusingly) as a Mutant, or sometimes +a Mutation. + +Mutation testing can be useful for suggesting semantic simplifications to your +code, as well as highlighting gaps in your tests as you are writing them. ## AST -Acronym for [Abstract Syntax Tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree) -and the level of abstraction mutant operates on. +Acronym for [Abstract Syntax Tree][AST] and the level of abstraction mutant +operates on. In short, this is a representation of the structure and content of +your code stored in memory which mutant alters. + +[AST]: https://en.wikipedia.org/wiki/Abstract_syntax_tree ## Subject @@ -18,38 +38,172 @@ Mutant currently supports the following subjects: * Instance methods * Singleton (class) methods -Other subjects (constants, class bodies for DSLs, ...) are possible but aren't -implemented in the OSS version. +Other subjects are possible (even project-specific subjects) but aren't +implemented in the OSS version. Some examples are: +* Constants +* Class bodies for DSLs -## Mutation operator +The more subjects that mutant can alter in your project, the more mutations it +can create, and so the higher the confidence you can have that your tests cover +the semantics of your application. Please get in touch if you require subjects +beyond those implemented in Mutant already - support may be available in the +commercial version. -A transformation applied to the AST of a subject. Mutant knows the following high level operator -classes: +## Mutation -* Semantic Reduction -* Orthogonal Replacement -* [Noop](#neutral-noop-tests) +An alteration to your codebase. Each mutation represents a hypothesis that +ideally gets falsified by the tests. Some examples of mutations and the +hypotheses they represent: -An exhaustive list can be found in the [mutant-meta](https://github.com/mbj/mutant/tree/master/meta) -subdirectory of the source. +```ruby +# before application of mutation operator +def equal_to_5?(number) + number == 5 +end -## Mutation +# after application of `#==` -> `#eql?` +# hypothesis: the conversion semantics of #== are not necessary for this method +def equal_to_5? + number.eql?(5) +end + +# after removal of `number == 5` +# hypothesis: that line of the method is dead code +def equal_to_5?; end + +# after replacement of `number == 5` with `true` +# hypothesis: the tests only check the happy-path +def equal_to_5?(number) + true +end +``` + +### Categories of mutation +Mutations broadly fall into a couple of categories. There is no reason to +deliberately seek to learn these categories (there’s no quiz at the end of this +semester!) - we’re just providing them to give you some ideas of what high-level +classes of changes mutant makes to your code. For a full list of all the types +of mutation that mutant can perform, see the code in the [meta directory][meta] + +**Evil** mutations are mutations which should cause the test suite to fail, +while **neutral** mutations are expected not to break the test suite. +[No-ops](#no-op) are the only form of neutral mutation in mutant - all others +are evil. + +[meta]: https://github.com/mbj/mutant/tree/master/meta + +#### Semantic Reduction + +This type of transformation replaces a piece of code which has somewhat complex +semantics with one that has simpler semantics. To aid understanding, here are a +couple of different sub-categories you could put them into. + +* **Method call replacement** + + Example: + + ```ruby + num == 1 + # gets replaced by + num.eql?(1) + ``` + + `#==` commonly performs conversion between types in addition to checking + equality, while `#eql?` tends to check only that the class and instance + variables are equal. (sticking with numbers, `1 == 1.0`, but `!1.eql?(1.0)`) + Therefore, `#== -> #eql?` is a semantic reduction. + + You could also think of a semantic reduction in these cases as an increase + in “strictness” of the code. `#equal?` (object identity) is a stricter + equality test than `#eql?`, which is stricter than `#==`. + +* **Code removal** + + Example: + + ```ruby + def my_method + if some_condition? + do_something + end + end + + # replaced by -The result of applying a mutation operator to the AST of a subject. A mutation represents a -hypothesis that ideally gets falsified by the tests. + def my_method + end + ``` + + It's arguable whether or not this is its own category - we include it only + to show that entire chunks of code are removed in some mutations performed + by mutant. These mutations make mutant the ultimate ruby dead code finder! + +* **Interface reduction** + + Example: + + ```ruby + def join2(one, two, options) + one + two + end + + # replaced by + + def join2(one two) + one + two + end + ``` + +#### Orthogonal Replacement +Example: + +```ruby +def true? + a.equal?(true) +end + +def true? + a.equal?(false) +end +``` + +Unlike semantic reduction, where the result is a simpler or stricter version of +the input, an orthogonal replacement changes code with a given function into +code which does something quite different (usually the opposite, or a different +value of the same type). This category is probably better understood by +examples: + * `true` -> `false` + * `#>` -> `#<` + +These mutations are less frequent, owing to their relative lack of use as +suggestions for dead code removal or code simplification. The category really +only exists to include constant replacements such as `true` -> `false` - which +is needed to ensure that you have tested that `true` is indeed the required +value in that circumstance. + +#### **No-Op** + +This type of mutation makes no changes to the code, but performs the test +execution in exactly the same way as an "evil" mutation It is needed in order +to ensure that mutant’s presence, and prior mutations’ side effects, do not +cause the test suite to fail. + +A no-op mutation is added at the beginning of the mutation test run (to ensure +that using mutant alone doesn't cause the test suite to fail) ## Insertion The process of inserting a mutation into the runtime environment. Mutant currently supports insertion via dynamically created monkeypatches. -Other insertion strategies (such as "boot time") are possible but aren't implemented -in the OSS version. +Other insertion strategies (such as "boot time") are possible but aren't +implemented in the OSS version. ## Isolation -The attempt to isolate the (side) effects of killing a mutation via an integration -to prevent a mutation leaking into adjacent concurrent, or future mutations. +The attempt to isolate the (side) effects of killing a mutation via an +integration to prevent a mutation leaking into adjacent concurrent, or future +mutations. Examples of sources for leaks are @@ -58,18 +212,23 @@ Examples of sources for leaks are * DB State * File system -Natively, mutant offers fork isolation. This works for any state within the executing -Ruby process. For all state reachable via IO, it's the test author's responsibility to -provide proper isolation. +Natively, mutant offers fork isolation. This works for any state within the +executing Ruby process. For all state reachable via IO, it's the test author's +responsibility to provide proper isolation (for example, by wrapping tests which +touch the database in a transaction) ## Integration -The method used to determine if a specific inserted mutation is covered by tests. +The method used to determine if a specific inserted mutation is covered by +tests. Currently mutant supports integrations for: -* [mutant-rspec](/docs/mutant-rspec.md) for [rspec](https://rspec.info) -* [mutant-minitest](/docs/mutant-minitest.md) for [minitest](https://github.com/seattlerb/minitest) +* [mutant-rspec](/docs/mutant-rspec.md) for [rspec][rspec] +* [mutant-minitest](/docs/mutant-minitest.md) for [minitest][minitest] + +[rspec]: https://rspec.info +[minitest]: https://github.com/seattlerb/minitest ## Report @@ -80,3 +239,6 @@ Mutant currently provides two different reporters: A reporter producing a machine readable report does not exist in the OSS version at the time of writing this documentation. + +See the [reading-reports.md](./reading-reports.md) file for documentation on the +information provided by those reports. diff --git a/meta/README.md b/meta/README.md new file mode 100644 index 000000000..40c188858 --- /dev/null +++ b/meta/README.md @@ -0,0 +1,50 @@ +Mutant’s Mutation Operators +=== + +This directory contains all the [Mutation Operators](../docs/nomenclature.md#mutation-operator)[^mutation-operator] that Mutant can apply. + +The filenames mostly correspond to the names of code elements as used by the [parser](https://github.com/whitequark/parser) and [unparser](https://github.com/mbj/unparser) libraries. For the most part they are the same as keywords within Ruby, but here are some explanations of the less obvious ones: + +* `str` and `sym` are string and symbol literals that do not contain dynamic elements (`"I'm a dstring #{i_am_the_dynamic_element}"`) + * `dst` and `dsym` are the equivalents for strings and symbols which do +* `regexp` is a regular expression without any options (`/like this/`) + * `regopt` is a regular expression with options (`/like this/i`) +* a few come in regular and `-asgn` variants - the latter are the assignment versions. E.g., `||=` is the assignment version (`or_asgn.rb`) of `||` (`or.rb`) +* `gvar`, `cvar`, `ivar` and `lvar` refer to global (`$VERBOSE`), class (`@@debug`), instance (`@amount`) and local (`response`) variables, respectively. +* `csend` is the conditional send operator (`&.`) +* `cbase` is a top-level constant (`::Kernel` or `::Errno::ENOENT`), whereas `const` is a "normal" constant (`Kernel` or `Errno::ENOENT`) +* `casgn` is an assignment to a constant (`VERSION = "0.1.0"`) + +All the files in this directory use a fairly simple DSL. To explain it, let's run through (a slightly modified version of) `return.rb`: + +```ruby +# Adds an example matching expressions like `return foo`, where foo +# can be anything. In parser's s-expression DSL, this would match +# `s(:return, s(...))`, but not the simpler `s(:return)`. +Mutant::Meta::Example.add :return do + # Each Example can only have one source, which provides a pattern to match + source 'return foo' + + # multiple `mutation`s are provided. Each is simply a ruby + # source-code representation of the code after the change, using + # the same placeholders as the source + mutation 'foo' # replaces `return foo` with `foo` + mutation 'return nil' # replaces `return nil` with `nil` + mutation 'return self' # replaces `return foo` with `return self` +end + +# This Example matches the simpler `s(:return)` case (or simply, +# `return` expressions) +Mutant::Meta::Example.add :return do + source 'return' + + # singleton_mutations is a convenience method for the following: + #  mutation 'nil' + #  mutation 'self' + # This just helps ensure consistency throughout - a very large + # number of expressions can be usefully mutated to `nil` or `self` + singleton_mutations +end +``` + +[^mutation-operator]: Remember, Mutation Operator just means “a change that can be applied to the code” \ No newline at end of file