Cake
-
|
We need to be able to assert that the script given a set of parameters executes the correct set of Tasks. We are trying to write a set of integration tests to do that and an end-to-end ci/cd pipeline after the tests pass. |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 1 reply
-
|
@supernik there are several ways to do this, the example below is one way, it's just a quick n' dirty script I did now, so room for improvement, but hopefully you get the gist of it, happy to elaborate if anythings unclear. Example script ( var target = Argument("target", "Default");
Task("TaskA")
.Does(()=>Information("Hello from TaskA"));
Task("TaskB")
.IsDependentOn("TaskA")
.Does(()=>Information("Hello from TaskB"));
Task("TaskC")
.IsDependentOn("TaskB")
.Does(()=>Information("Hello from TaskC"));
Task("Default")
.IsDependentOn("TaskC");
RunTarget(target);This will for and and One way to from the outside observe if right set of tasks are executed is to capture the console output of the executing script. So let's create an integration test script called var testCases = new Dictionary<string, string[]> {
{
"TaskA",
new[] {
"Task",
"--------------------------------------------------",
"TaskA",
"--------------------------------------------------",
"Total:"
}
},
{
"TaskB",
new[] {
"Task",
"--------------------------------------------------",
"TaskA",
"TaskB",
"--------------------------------------------------",
"Total:"
}
},
{
"TaskC",
new[] {
"Task",
"--------------------------------------------------",
"TaskA",
"TaskB",
"TaskC",
"--------------------------------------------------",
"Total:"
}
}
};I've intentionally omitted durations from expected output, as those are expected to vary between executions. Integration test task could then look something like Task("Integration-Tests")
.DoesForEach(
testCases,
testCase => {
Information("Executing test {0}", testCase.Key);
IEnumerable<string> redirectedStandardOutput;
var exitCode = StartProcess(
Context.Tools.Resolve("dotnet-cake.exe") ?? Context.Tools.Resolve("dotnet-cake"),
new ProcessSettings {
Arguments = new ProcessArgumentBuilder()
.AppendQuoted("build.cake")
.AppendSwitchQuoted("--target", "=", testCase.Key),
RedirectStandardOutput = true
},
out redirectedStandardOutput
);
if (exitCode != 0)
{
throw new Exception($"Executing {testCase.Key} unexpected exit code {exitCode}.");
}
testCase.AssertEqual(redirectedStandardOutput);
Information("Successfully executed test {0}", testCase.Key);
})
.DeferOnError();
RunTarget(Argument("target", "Integration-Tests"));This will for each
public static void AssertEqual(this KeyValuePair<string, string[]> expect, IEnumerable<string> value) =>
value
.Reverse()
.Take(expect.Value.Length)
.Reverse()
.Aggregate(
(index: 0, lines: new List<(int index, string expect, string actual, bool sucess)>()),
(ack, line) => {
var lineExpect = expect.Value[ack.index];
ack.lines.Add(
(
ack.index + 1,
lineExpect,
line,
line.StartsWith(lineExpect)
)
);
return (ack.index + 1, ack.lines);
},
result => {
var errors = result
.lines
.Where(line => !line.sucess)
.Select(line=>$"{System.Environment.NewLine}\tLine: {line.index} Expected: {line.expect} Got: {line.actual}")
.ToList();
if (errors.Count == 0)
{
return true;
}
var errorLines = string.Concat(errors);
throw new Exception($"Test case {expect.Key} {errorLines}");
}
);this could be written in many ways, but essentially just did a quick Linq aggregate, that ensures all lines start with the expected value. Complete var testCases = new Dictionary<string, string[]> {
{
"TaskA",
new[] {
"Task",
"--------------------------------------------------",
"TaskA",
"--------------------------------------------------",
"Total:"
}
},
{
"TaskB",
new[] {
"Task",
"--------------------------------------------------",
"TaskA",
"TaskB",
"--------------------------------------------------",
"Total:"
}
},
{
"TaskC",
new[] {
"Task",
"--------------------------------------------------",
"TaskA",
"TaskB",
"TaskC",
"--------------------------------------------------",
"Total:"
}
}
};
Task("Integration-Tests")
.DoesForEach(
testCases,
testCase => {
Information("Executing test {0}", testCase.Key);
IEnumerable<string> redirectedStandardOutput;
var exitCode = StartProcess(
Context.Tools.Resolve("dotnet-cake.exe") ?? Context.Tools.Resolve("dotnet-cake"),
new ProcessSettings {
Arguments = new ProcessArgumentBuilder()
.AppendQuoted("build.cake")
.AppendSwitchQuoted("--target", "=", testCase.Key),
RedirectStandardOutput = true
},
out redirectedStandardOutput
);
if (exitCode != 0)
{
throw new Exception($"Executing {testCase.Key} unexpected exit code {exitCode}.");
}
testCase.AssertEqual(redirectedStandardOutput);
Information("Successfully executed test {0}", testCase.Key);
})
.DeferOnError();
RunTarget(Argument("target", "Integration-Tests"));
public static void AssertEqual(this KeyValuePair<string, string[]> expect, IEnumerable<string> value) =>
value
.Reverse()
.Take(expect.Value.Length)
.Reverse()
.Aggregate(
(index: 0, lines: new List<(int index, string expect, string actual, bool sucess)>()),
(ack, line) => {
var lineExpect = expect.Value[ack.index];
ack.lines.Add(
(
ack.index + 1,
lineExpect,
line,
line.StartsWith(lineExpect)
)
);
return (ack.index + 1, ack.lines);
},
result => {
var errors = result
.lines
.Where(line => !line.sucess)
.Select(line=>$"{System.Environment.NewLine}\tLine: {line.index} Expected: {line.expect} Got: {line.actual}")
.ToList();
if (errors.Count == 0)
{
return true;
}
var errorLines = string.Concat(errors);
throw new Exception($"Test case {expect.Key} {errorLines}");
}
);and its output if made a small change to test cases i.e. {
"TaskC",
new[] {
"Task",
"--------------------------------------------------",
"TaskA",
- "TaskB",
"TaskC",
"--------------------------------------------------",
"Total:"
}
}then during the execution of ========================================
Integration-Tests
========================================
Executing test TaskA
Successfully executed test TaskA
Executing test TaskB
Successfully executed test TaskB
Executing test TaskC
- An error occurred when executing task 'Integration-Tests'.
- Error: One or more errors occurred. (Test case TaskC
- Line: 1 Expected: Task Got: --------------------------------------------------
- Line: 2 Expected: -------------------------------------------------- Got: TaskA 00:00:00.0104040
- Line: 3 Expected: TaskA Got: TaskB 00:00:00.0004915 ) |
Beta Was this translation helpful? Give feedback.
-
|
Another alternative would be to use a wrapper script and inject some code before For same #addin "nuget:?package=xunit.assert&version=2.4.1"
using Xunit;
public class TestCase
{
private List<string> result;
public string Name { get; }
public ICollection<string> Expect { get; }
public ICollection<string> Actual => result;
public void Add(string name) => result.Add(name);
public TestCase(string name, ICollection<string> expect)
{
Name = name;
Expect = expect;
result = new List<string>();
}
}
Setup(setupContext => new TestCase(
setupContext.TargetTask.Name,
setupContext.TargetTask.Name switch {
"TaskA" => new [] { "TaskA" },
"TaskB" => new [] { "TaskA", "TaskB" },
"TaskC" => new [] { "TaskA", "TaskB", "TaskC" },
"Default" => new [] { "TaskA", "TaskB", "TaskC", "Default"},
_=> throw new NotSupportedException($"Unknown test case target {setupContext.TargetTask.Name}")
}
));
TaskTeardown<TestCase>((taskTeardownContext, testCase) => testCase.Add(taskTeardownContext.Task.Name));
Teardown<TestCase>((teardownContext, testCase) => {
Assert.Equal(testCase.Expect, testCase.Actual);
teardownContext.Information($"Integration tests for target {testCase.Name} executed successfully");
});
#load "build.cake"then executed tests for each target ( dotnet cake inject_integration_tests.cake
dotnet cake inject_integration_tests.cake --target=TaskA
dotnet cake inject_integration_tests.cake --target=TaskB
dotnet cake inject_integration_tests.cake --target=TaskCand output for i.e. and if we changed expect for ----------------------------------------
Setup
----------------------------------------
========================================
TaskA
========================================
Hello from TaskA
========================================
TaskB
========================================
Hello from TaskB
----------------------------------------
Teardown
----------------------------------------
- An error occurred in a custom teardown action.
- Error: One or more errors occurred. (Assert.Equal() Failure
- Expected: String[] ["TaskA", "TaskB", "TaskC"]
- Actual: List<String> ["TaskA", "TaskB"]) |
Beta Was this translation helpful? Give feedback.
-
|
Imho asserting on strings as a test is not very robust; something we wouldn't normally do in normal code and therefore not particularly in keeping with ci-as-code spirit then. I have been mulling the same probalem: i.e. wite unit-tests on cake cicd orchestrator that asserts the state passed to the tools & the right sequence of tools executed given some state (env vars, config, etc). What do you think? |
Beta Was this translation helpful? Give feedback.
-
|
For integration tests it's hard to exactly reproduce environment, code and tools being executed. So for some scenarios testing whole thing in an isolated environment i.e. a throw away container is easiest path. But others it's great to have abstractions, i.e. with Cake.Kudu.Client I mock the httpclient used for all operations, so tests can run in memory with known request/responses https://github.com/cake-contrib/Cake.Kudu.Client/tree/develop/tests/integration |
Beta Was this translation helpful? Give feedback.
-
|
@devlead that is an amazing response, thanks. Originally I was thinking if there was a way to have a more structured text output, so I've been looking into building a module to do that. Testing wise I was planning to use Pester, primarily because we use the bootstrapper to run cake. I'll run through your examples, I appreciate you taking time to respond in this much detail. |
Beta Was this translation helpful? Give feedback.
@supernik there are several ways to do this, the example below is one way, it's just a quick n' dirty script I did now, so room for improvement, but hopefully you get the gist of it, happy to elaborate if anythings unclear.
Example script (
build.cake) to execute in integration tests.This will for
TaskAoutput something like