-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
148 lines (148 loc) · 9.79 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
---
layout: default
title: Home
keywords: tlb, test-load-balancer
description: Home page of 'Test Load Balancer(TLB)' project(s).
---
<div>
<h3>What is Test Load Balancer (TLB)?</h3>
<div class="scope section">
Test Load Balancer(TLB) is a tool that can automatically partition tests into multiple subsets each one of which can be executed in parallel. The execution can happen on different physical/virtual machines or on the same machine as different processes or threads.
More the partitions, lesser are the number of tests executed on each one, and since all of the partitions start at the same time(and finish almost at the same time) overall test-execution time gets divided by the number of partitions you make. Test-running is by far the longest step in most(if not all) builds, and cutting down test-running time speeds up the build, hence feedback loop.
TLB can be used for any kind of test-suite, it can be unit, integration or functional tests.
<p class="clob_end"/>
In addition to balancing, TLB does other interesting things like re-order tests within a subset(set of tests that run on a partition) before they are executed. For instance, it re-arranges tests to execute failing ones(that failed in previous build) first, hence ensuring early feedback.
<p class="clob_end"/>
<a href="http://dl.dropbox.com/u/16810295/TLB%20-%20rocket%20booster%20for%20your%20build.pdf">Here is the slide-deck</a> that we use for introduction talk in conferences.
</div>
</div>
<hr/>
<h2 class="big_button_container">3.. 2.. 1.. {% include abs_tlb_getting_started %} </h2>
<hr/>
<div>
<h3>LICENSE</h3>
<div class="scope section">
<p>TLB is released under the BSD (2-clause version) license. Check out the <a href="https://github.com/test-load-balancer/tlb/blob/master/LICENSE" target="blank">License</a></p>
</div>
</div>
<hr/>
<div>
<h3>What Language(s)/Platform(s) does TLB support?</h3>
<div class="scope section">
<p>TLB is written in Java, which means, it can be used on pretty much all platforms that Java can run on.</p>
<p>However, TLB was written from ground up to support both JVM and non-JVM based languages and runtime(s). The Balancer is capable of running as a standalone process, in alien-environment(as we call it). The build/test framework and programming language that tests are written in does not matter.</p>
<p>{% include tlb_rb_link %} for instance, supports MRI(CRuby) and using standalone balancer. However, since TLB-core is written in java, you will need to have java installed on the box running tests(so balancer process can be launched).</p>
<p><a href="#supported_tools_and_environments">Supported tools and environments</a> section below has the list of all frameworks/environments TLB supports as of now.</p>
</div>
</div>
<hr/>
<div>
<h3>What testing-tools/build-tools does TLB support?</h3>
<div class="scope section">
<div class="notice warning">
This list is updated as upstream evolves. Please check version specific documentation to find out what a particular version supports.
</div>
<p/>
<h5>TLB supports:</h5>
{% include upstream_supported_frameworks %}
<p class="clob_end"/>
<div class="with_desc">
<h5>Work in progress:</h5>
<span class="heading_desc">TLB team is working on adding support for the following tool combinations:</span>
</div>
<table>
<thead>
<tr>
<th>Testing Tool</th>
<th>Build Tool(s)</th>
<th>Programming Language(s) or Platform(s)</th>
</tr>
</thead>
<tbody>
<tr>
<td>{% include junit_link %}</td>
<td>{% include maven_link %}</td>
<td>{% include java_link %}</td>
</tr>
<tr>
<td>{% include nunit_link %}</td>
<td>{% include nant_link %}</td>
<td>{% include dot_net_link %}</td>
</tr>
</tbody>
</table>
<p class="clob_end"/>
<div class="with_desc">
<h5>We plan to add support for:</h5>
<span class="heading_desc">We have not yet started work on adding support for tools mentioned in this section.</span>
</div>
MSTest(.Net), MSBuild(.Net), Maven(Java), unittest(Python), FiveAM(CommonLisp), cppunit(C++) etc.
<p class="clob_end"/>
And of course, anything else you can mail us patches for... :-)
</div>
</div>
<hr/>
<div>
<h3>Dedicated support</h3>
<div class="scope section">
If you need help getting up and running with TLB or require help tuning your build, do write to us at <a href="mailto:[email protected]">[email protected]</a> or <a href="mailto:[email protected]">[email protected]</a>. You can also drop us a note on the <a href="mailto:[email protected]">project mailing list</a>.
</div>
</div>
<hr/>
<div>
<h3>What problem does TLB solve?</h3>
<div class="scope section">
Most build servers(like Hudson, Go, TeamCity, Bamboo etc.) provide parallel execution capability(capability to execute command(s) on different machines at the same time).
However, parallelization of tests needs a tool that can decide what tests need to be run in each such parallely running process across machines. This is where TLB comes in.
<p class="clob_end"/>
Given that you have a way to invoke the test command(project's test target) on multiple machines at the same time(which can be using a CI server's agent-grid environment, or can even be actually invoked by a tiny utility script from your terminal), you can use TLB to ensure each one of these invocations execute only a few tests(and not all).
TLB ensures these few tests for each partition are selected in a way, such that:
<p class="clob_end"/>
<ul>
<li>No test is run on more than one partition: <i>Mutual Exclusion</i></li>
<li>No test is missed out(every test is selected by at-least one partition): <i>Collective Exhaustion</i></li>
</ul>
<p class="clob_end"/>
The problem TLB solves for you is that of slicing and dicing your test suites in the most optimal way, allowing you to get the best of parallel execution(and not that of launching parallel processes on single or multiple machines).
TLB engages once your test target is launched, it remains agnostic to what triggers it.
</div>
</div>
<hr/>
<div>
<h3>How does TLB work?</h3>
<div class="scope section">
<p/>
{% include abs_components_of_tlb %}
Balancer hooks-up with your build-framework and testing-framework to do the actual work, whereas Server is primarily a data-repository that balancer talks to. Data from historical run(s) is used by the balancer to partition/re-order the current run, and in-turn, data from the current run is posted back so it can be used as historical data for future runs.
<p class="clob_end"/>
It actually forms a cycle where historical data is used to partition accurately, and run data is captured to seed future runs so accuracy is maintained(as data is updated with every new run).
{% include interaction_while_tlbing %}
</div>
</div>
<hr/>
<div>
<h3>Deep dive: Understanding and Configuring TLB</h3>
<div class="scope section">
While {% include abs_tlb_getting_started %} is a good place to start off, sooner or later you will want to understand the basic concepts involved, and find out more about configurable parameters TLB exposes to allow you to tune it best for your project and environment. Getting a good grasp of TLB concepts will enable you to leverage a lot of very useful features TLB exposes.
<p class="clob_end"/>
{% include documentation_concepts_link %} covers TLB concepts. {% include abs_documentation_config_vars_link %} section for configuration related documentation which covers configuration parameters in detail.
</div>
</div>
<hr/>
<div>
<h3>Philosophy and Inspiration</h3>
<div class="scope section">
Running tests is by far the biggest time slice in any project's build, unless the project has very little or no tests at all(which if the case, is obviously a bigger problem).
<p/>
Over a period of time, functionality and features grow, and so do tests. Eventually because of these huge number of tests, the time taken by end-to-end build starts eating into the productivity pie. Every upstream change-set goes through a build process which is several minutes(or worse, hours) long, which makes cost of a fixing a failing test really high.
This is particularly true for automated functional tests or integration tests just because they generally take more time to execute than unit-tests.
<p/>
When things fail in a slow build, fixing em, getting a green build and having development/testing teams back on track can be an extremely frustrating and time consuming process. It takes awful lot of time to attempt fixes(especially so, when dealing with an indirect or complicated issue, which takes multiple attempts) and having slow build often forces the developer(s) to wait for hours together to get a single good build.
<p/>
Given that hardware is so cheep now a days and that assuming tests are independent of each other and are order insensitive, most teams can cut their build time by parallelizing test execution. No matter how many tests the project has, you can maintain really low build time just by throwing more hardware at it. Parallelization and scaling-out(distribution of computing to several commodity machines) is a terrific strategy to conquer independent computing problems that when executed serially, take long time and running of tests is one such problem.
<p/>
Before coming up with TLB, we faced the slow-serial builds problem on every single project we worked on, and couldn't find anything out there capable of solving it satisfactorily. TLB has been written from ground-up to solve this problem for every language and every testing-framework.
<p/>
TLB is our final answer to the slow builds problem.
</div>
</div>