It often happens when I find an error in a new or changed feature that my “spider sense” starts tingling. The test scenario, test data and intuition/experience make me start thinking that this error is just the tip of the iceberg … The literature on software testing confirms that errors rarely come alone, but how far should we go along the path of searching for more errors – using less and less typical data / scenarios on the way?

What should a tester do in this situation? Ignore the probability that there are more errors in this feature and therefore risk producing incidents in the productive environment? Or should we continue to test in this area, therefore reducing the time spent on other features?

If we take the first option, and the error is found by the customer, the question is likely to come up: “why didn’t the tests find this?” If we take the second choice, then we run the risk of missing errors in more central parts of the software (the question will be the same, though!). Is testing condemned to contradictions? Regardless of which test cases we exclude, there is always the chance that they were the ones we should have been doing.

I came to the conclusion that the reason for the apparent contradiction comes from the idea of “searching for errors”. The aim of tests is to create a sense of trust in the software, like a safety net. To be efficient, we have to prioritise. The priority should be to cover the most important scenarios, and not to check for the most likely errors. So the aim of testing isn’t to go “looking for trouble“, but to “discover” any errors in the most important scenarios. If no errors are found, then the trust in the software is reinforced.

So when we have good reason to think that one error may have a couple of friends hanging around in the same feature, should the test for these errors be classified as “searching”? I don’t think so. These errors aren’t floating around in the ether, far out of our grasp; rather they’re lurking just below the surface, waiting to be found. So if a tester reports an error from a less typical scenario, then it doesn’t mean that s/he went on a random quest to search for errors. Some errors make themselves known as soon as we get close to them.

The full program is available on the Eclipse Testing Day Wiki Page, and we’re pleased to be able to offer a selection of talks about different areas of testing, including:

• Acceptance and GUI testing
• Embedded testing in OSGi
• Scope and coverage of tests
• Unit testing
• Test generation

As mentioned in the previous post, registration is now open on the Eventbrite Site. Member tickets (Eclipse and OSGi members) are priced at 40€ net, non-members at 50€ net. The ticket cost covers the event organization and includes catering during the day and the evening reception.

We’re looking forward to a successful day – start inviting colleagues, contact us if you’d like the logo and we’ll see you on the 8th September in Darmstadt!

We’ve created an Eventbrite site to ease the registration process. Details of prices and payment are both on the Eclipse wiki and directly on the Eventbrite site, but just get in touch if you have questions.

We’ve also extended the call for papers for another week – you have until the 10th July to submit your abstract (again, details on the Eclipse wiki). Any Eclipse and/or testing enthusiasts are invited to tell us about how they test, and of course, to register for the day. Tickets are limited to 80 participants, so we recommend registering early!

The first half of the evening saw Brox’s demonstration of SMILA followed by a demonstration of testing Eclipse applications with GUIdancer by Bredex. After a short break the camp continued with the two final demos: Erhard Weinell of Yatta Solutions showing their UML Lab and Sven Efftinge demonstrating XText. If there is a prize for how many Eclipse Award winners are present at a Demo Camp, then this one is a good contender: Both XText and GUIdancer were winners of awards this year.

After the demos there was enough time for chatting, discussions and introductions – Ralph Müller from Eclipse and Michaela Kraft from Microsoft were both on hand to provide excellent networking opportunities.

With even more new faces at this demo camp than the previous one in November, our hope of creating a strong Eclipse community in the Hannover/Braunschweig region seems to be becoming reality. Stay tuned for more Eclipse events!

The slides from the demo camp are available on the Eclipse Wiki.

Previously we’d found ourselves wasting time because a story wasn’t thought through well enough, either from the development/implementation perspective (we went the wrong way technology-wise) or in terms of the features (incomplete or inconsistent concepts, for example). We were also ending up at the end of a sprint with nothing deliverable because our chunks were often too big.

After a good session with Lisa Crispin last October, we put some ideas into motion about making our stories better described, with a focus on deliverable, testable requirements.

1. We take the time as a team (sometimes all of us, sometimes just a few, depending on the meeting and story) to really talk our way through stories for an iteration. This often means spending more time discussing things than we previously did – we even invented a new word for the process “excrucialating”. A combination of excruciating and crucial. Especially in terms of epics, this has really paid off for us though. Now even stories that are connected and continue over many sprints end up being deliverable and we find ourselves on the wrong path much less often.
2. We make a huge effort to make story points as “thin” as possible. We focus on value – what *must* be achieved to make this story useful for the customer? Nice-to-haves and other features either get put into the product backlog, or we write cards in case we have time at the end to smooth things out.
3. Acceptance test criteria are a part of the story and we write them on the card. Sometimes a story has to be written in a more technical way, but wherever possible, we describe (and often automate) the test that will prove that the story is done.
4. We introduced a story board in the room where we have our stand up meetings. We have four columns: planned, under construction, to test and done. The “to test” column came later to make us really aware that “done” is also tested. Ideally, cards shouldn’t be in the “to test” column for very long though. An automated test should tell us the next day if the feature is done, and a manual test should be done as quickly as possible once the feature is committed.

These few steps have really helped us over the past few sprints. We’re coming along well with our stories and are getting much better at identifying (and sticking to) what we *need* to implement to release a new feature.

So far, so advantageous for the team. But we’ve run into the question a few times now of “where is the time to refactor”? The focus on thin stories that implement the minimum amount of code possible to gain new features doesn’t leave much room for making the internal code better, clearer or more maintainable on the way. After all, no-one can release a new version of the software that proclaims to do everything it did before, just with nicer internal code and updated libraries.

There’s a few ways we could try to deal with this:

1. Adjust the amount of stories we try to fit into a sprint to leave us the time to look at such areas.
2. Work on a case-by-case basis. If we have to extend the time a story will take to gain much better code, then we can make this decision for certain stories.
3. Introduce regular sprints (perhaps shorter than our usual 4-weekers) to work on internal knots we’ve identified recently.
4. Add a certain amount to each story for “possible refactoring”.

We haven’t decided which of these options we’re going to try (first) yet, but I can imagine that it’s not a question that just affects our team. We’re pleased that we’re focusing on features, but we are noticing that the time-pressure has increased somewhat by doing this. We’d be interested to hear suggestions of what other people have done to combat this, and I’ll keep you up to date on our process and results.

Eclipse Demo Camp
On the 21st June, we’ll be organizing an Eclipse Demo Camp in Hannover with Brox IT Solutions GmbH. We already have three confirmed demos and are waiting on another two, so we’re looking forward to another successful camp. The Eventbrite page is up and running for you to register, and details about the Demo Camp are on the Eclipse Wiki.

Eclipse Testing Day
On September 8th, we’ll be in Darmstadt with MicroDoc GmbH for the Eclipse Testing Day. The call for papers is now open, and we look forward to hearing your suggestions for talks about testing Eclipse as a platform, testing OSGi applications and testing tools based on Eclipse. We’re also still looking for sponsors – more details are on the wiki page.

These two events should be a great chance to meet people from the Braunschweig and Hannover region, as well as test enthusiasts within the Eclipse Community – so get registering, submit a talk and come and meet us at one (or both) of the events!

Lately, I was confronted with a problem concerning formula evaluation. I had a pre-existing Excel template file that had some cells with placeholder texts. These texts are replaced with number values available at runtime. The result is saved as a new file. This works great with POI until a cell with a placeholder text is used within a formula of another cell. When looking at the template, Excel then obviously complains about the data type of the values used within the formula.

This behaviour is alright and expected at this point, since the placeholder texts are indeed of the wrong data type and Excel cannot evaluate the formula correctly. The error should go away once the template has been modified as described above.

However, this is not how it works! Excel caches the result of each formula evaluation within the spreadsheet file. When the file is programmatically modified with POI, the cached result remains unaltered and is displayed again when the file is opened.

Excel uses the caching to avoid performing redundant and possibly long-running calculations when opening existing spreadsheet files. If the files are not tampered with, this strategy works fine, because Excel knows when a cell value changes and when a recalculation is necessary.

In order to deal with the problem of out-dated cached values, I tried out four different solutions, where each solution somehow evolved from the previous one by improving its shortcomings.

For the first attempt, there is the key combination CTRL-ALT-F9, which tells Excel to recalculate all formulas of the entire workbook. Although this is already better than having errors on the sheet, it is still poor to require user interaction before the data on the sheet becomes valid.

So, the next idea was to automate the process by adding a little Visual Basic for Applications (VBA) script to the file that triggers the recalculation when the workbook is loaded. Again a little bit better, but requiring a script is still not a preferable solution.

This is where POI comes back into play. POI provides a formula evaluator that can calculate the value for a formula cell and update the cached value on the cell. Updating all the cached values of the workbook before saving it produces the result that was already expected in the very beginning.

Yet there is another pitfall that may cause the calculation to fail: the formula evaluator cannot handle decimal numbers that do not use the dot as decimal character. For example, the German locale has the comma as decimal character. A formula that references such a value once again displays the invalid data type error.

Finally, I found the fourth solution that passes the responsibility of calculating the formula result back to Excel. The idea is to simply remove the cached results from the file. Without a cached value, Excel is forced to evaluate the formula just-in-time when the cell is first displayed. In order to clear the cached value from a cell, you just have to temporarily set the cell type to something different than Cell.CELL_TYPE_FORMULA and then back to the original cell type.

    String formula = cell.getCellFormula();
    cell.setCellType(Cell.CELL_TYPE_BLANK);
    cell.setCellType(Cell.CELL_TYPE_FORMULA);
    cell.setCellFormula(formula);

A disadvantage of removing the cached calculation results may be a decreased performance when opening the workbook. Nevertheless, a noticeable slowdown should not happen unless the file has at least a few thousand (uncached) formulas. Also, it is not necessary to remove all the cached calculation results. It is perfectly OK to start with the POI formula evaluator and only use the trick for clearing the cell cache, if the evaluator reports an error.

Now we have a continuous integration process; the latest version of the software is built and tested every night. This has brought us many benefits: we have quicker feedback on developments and the team can work with the current version each day. However, despite having functional software every morning, we don’t have a version any more that we can call a beta. Daily builds are feature-incomplete and there are occasionally areas that shouldn’t be touched or worked with because they are still somewhat under construction. If we’ve been working on epics, then we often have new developments that are leading up to a new big feature, but not the actual feature itself. It’s hard to make a cut and say “this is the beta”.

We’re still unsure about whether this bothers us, though. Do agile projects need specific beta versions, or should customers be able to work with the daily build, having been informed if any areas are unfinished? If betas are desired, how can they be planned for and constructed in the agile paradigm?

Maybe it’s an obvious point, but I’d like to suggest that agility isn’t just beneficial for the customer; it’s also beneficial for the team.

The combination of both obfuscation and JAR signing nearly drove me crazy last week – none of my executable JARs were working any more. All the information which had been stored in the MANIFEST.MF during the JAR creating process via ANT (e.g. the classpath entries and main class attribute) disappeared during the obfuscating and signing process. A good few hours  later I finally found out why:  the code obfuscation tool we use rewrites all manifest files with a minor but significant difference:  instead of using CR+LF characters at the end of each line it only uses LF.  This small difference means that the JDK jarsigner, which is implicitly used by the ANT SignJar-task,  is not able to parse the manifest attributes correctly and so only rewrites the class fingerprints to the manifest – silently leaving out all other information.

Next time I’ll bear this in mind when I’m signing JARs and obfuscating – and hopefully save myself a great deal of confusion.