BUG - VS Code Coverage automatically closes testrunconfig Dialog

This refers to a solution which has a project that is a custom project template, such as BizTalk or Reporting Services.

When the testrunconfig Dialog box is opened, either by double clicking it in solutionitems, or through the test menu, 'Edit Test Run Configurations', when the Code Coverage item is selected, the dialog box disappears.

The workaround is to unload the offending projects allows that Code Coverage item to be changed, but be aware that you might need to reload the project to allow the tests to run after making the code coverage changes.

I have this logged as a bug with Microsoft, but as yet is not officially confirmed.

Unit Testing - Watch out!

I just thought that I'd write a little article to discuss some of the things to watch out for when unit testing. People tend to talk more about why it is or isn't good, and that you should use it, I'm proposing to tell you what to do when you do!

The first thing, the level of confidence in the test of the application is only as good as the tests that you write. Seems obvious, right? Using unit testing techniques does not make your code any more manageable, or correct if the tests that you write are of a poor quality. so all that you can say about unit tests, is that the code passes for the tests, and never that it 100% works, unless of course you are 100% certain that you have catered for all eventualities in the code.

Using code coverage as a tool to target your tests will not guarantee anything about the quality of confidence you should have in those unit tests. Here, I mean that having 100% coverage means that all the code is tested; it doesn't. What it means is that all the code has been exercise, but not that the code has been exercised with every possible value. We tend to use boundary conditions and the like to use in our tests reasoning that these values will best test our code. They don't, they're only representations.
Code coverage, you might not be able to exercise 100% of the code, if your code uses some .NET object that you can't exercise all the CLI for. You should find out whether that is the case though. It can often be an indication that the boundary condition by itself is woefully inadequate.

Integration tests. Until you do integration tests, you can't only speculate as to how well the code will work in the real environment. What happens if you get data that you didn't expect? What it means is that you write more tests, so that it doesn't happen again, even if the data was in error. The further upstream you go in terms of integration tests, toward a full system test, the better the tests are, and the more likely it is to fail. You're using real data, in real situations, and so can have a certain amount of confidence that the components at least work in that real world scenario. Look at exercising as much of the code using the coverage tools again, obviously some parts, such as testing exception handling belong only in the unit tests, but anything that can have data driven through it, to get as much coverage as possible is good.

If we do things right, then when we find errors, we write more tests, so that the tests become more and more complete, and our experience in where the tests could fail with the components increases, and thus gives us a better indication and knowledge of the risks involved with deploying the software.

So long as we bear in mind that we're never going to be able to say with 100% certainty, that a component is 100% bug free, we're probably safe. What we do with unit testing techniques is mitigate the risk, improve our knowledge of the system and its inner workings thus limiting risk and document part of the system.

These techniques are very useful, if used to their full potential, but never make a sweeping statement that the software is 100% bug free because all the tests pass, since that proves nothing.

Happy coding!

Software patterns - useful or over-used?

Recently, on the MSDN forums I had someone comment that design patterns are largely over-used. The answer I gave to a question involved the use of a design pattern for a developer who was clearly inexperienced.

The comment was that the developer should learn proper OO techniques rather than simply saying that they "would use that pattern". I would tend to agree that understanding proper OO principles is a pre-requisite to most jobs in the software industry these days. That got me thinking, does using design patterns give rise to a set of developers who understand that they need to use a pattern rather than understanding the intent behind it?

I would say no. If said developer is using a pattern because someone said so rather than because they know that it fits their requirements, perhaps they need some career counselling.

That brings me on to patterns. A pattern is a well define solution to a particular programming problem, and that solution may in fact consist of a number of patterns to make up the full solution.

It is very true that you could use TDD, and evolve such a design, or pattern, and that might be the acceptible way to approach the situation, it certainly would be if you're using pure TDD. It just seems that having an idea of possible approaches, a solution space, if you like is a great idea. It gives a set of scenarios to consider, or a set of implementations that you must consider, which is a good thing.

Then, if you're not doing pure TDD, you might want to adopt a pattern, which is a well understood way forward to implementation. If you're an experienced developer, you'll probably have implemented these patterns before without calling them patterns, so in that case, it's a descriptive label to put on an "experience catalogue". It means other developers who have also developed using those same algorithms can have a common understanding of what you're talking about.

Primarily, I see this as the major force behind patterns. The ability to communicate a shared understanding of a design, in a short and succinct manner.

People have often asked me what they need to do to make the code a such and such pattern, or instead, which falvour of the pattern is this, I want to use this pattern, how can I covert it from a similar pattern to that pattern? These sorts of questions seem strange to me. The pattern is there to help you think about the problem in isolation, decoupling, and good things like that. Whether it has one name or another name is really irrelevant, it's all about whether the software does what it is supposed to, and does it in an efficient manner.

I have also come across situations where people try to follow the pattern to the letter. To me, a pattern is a template, but there is no reason why that template can not be extended to fit the problem space, so long as that design is well considered.

So in commenting that software patterns are over-used, that really is kind of like saying that experience is over-used and overrated. Patterns, whilst they're not a new idea should be used more, if nothing else so that it is an enabler for communication.

Making a mockery of Testing

How much testing is enough testing?

If for example, you have to deliver a piece of software, some functionality and you have tight deadlines to meet, is 50% of code tested sufficient enough? How do you know that you have tested the most likely to fail 50% of the codebase?

That question probably sounds quite silly, but whenever I mention to people that they should be using Mock objects to test, they say that it is overkill, and that unit tests are enough. Further probing often reveals that these unit tests are really integration tests, tests such as ask for a particular record, or match filter to return records, and if records are returned, the test correctly passes.
That sort of a test is as brittle, weak and unknown a code fragment as the code fragment that it actually tests.

The database goes does, is that a code failure? Bet there's no test for the server actually going down, and if there was, how would you be able to descriminate between an infrastructure problem, and a coding error?

There are two approaches, one is to write tests, using MSTest, NUnit or whatever else you like to use, you pass a value, and expect a given response. This situation makes it incredibly difficult to test all of the code, since you have no way to control flow through a particular code branch. An example would be testing the exception handling block of a routine, given a unit test, there is no way to inject the exception into the code, so you're stuck with compiler switches to throw exceptions. Compiler switches is changing the code from what it will be in production to something else, which carries its own set of risks.

The alternative is Mocking. Now the concept behind mocking is that you create objects that are effectively proxies to the interfaces that a particular method or class uses. You then set expectations on those objects, and tell them to return data, which effectively drives the code through the desired execution path.
In that instance, if the database goes down, the tests will still pass, you are just driving the interface that is implemented at the data layer. You can't be touched, nobody can take that code down.

As a shortcut, to achieve that deadline, how much time would it take up to consistently and continually test and retest the code with such rigour?
Another question you will be asking yourself, is, is 50% tested code enough now, and will it actually take me less time to resolve all the bugs in the code using the debugger continually, instead of having repeatable and isolateable test targets to check instead?

Keep an eye out for a further article on how to do some mocking to enable the test coverage statistics to get up into the 90% area with nearly all classes.

Real world test driven development (TDD) - Perhaps "Requirements Driven Design and Development"?

Test driven development seems to divide programmers into three different camps; those that think that it is the best thing since sliced bread, those that think it is a good idea but impractical due to the time it takes to use, and those that are completely ignorant of what exactly it is.

It seems, that if you start to try using TDD for any projects, it will very quickly become clear why you might want to implement using this technique. Simply, it isn't so much about writing tests, but more about evolving a solid, loosely coupled solution that meets the needs of the requirements and nothing more. From the point of view of that statement, surely nobody would argue that up front design is a good idea, and in so agreeing to that statement, you also agree that you agree with the principles of TDD.

But there's more! As a by-product of evolving this wonderful software design, that is neither over-engineered nor highly coupled, we also end up with tests, and code!
It probably seems strange that I have mentioned that we end up with tests; very quickly a practitioner of TDD will notice that tests are manifestations of the requirements.

So we're really looking at test driven development from the point of view of what we want to do with the solution, which is the software's requirements. This technique doesn't in any way imply that we don't gather requirements, but what is does do is allow us a framework by which to evolve yet more requirements as they become apparent. What I mean by that is that we have a bunch of tests, that will always test for functionality, and should that functionality cease to work correctly, the test will fail, and the requirement will not be met. This means that we are then able to add, or change functionality, and know that a particular requirement is or is not met.

Well, I've worked on a few projects in my time, and in every one, we are asked how far from completion we are. How can we tell? Well, if we're sensible, we have some unit tests. The number of not yet implemented tests, and the number of failing tests should give us an idea of how far we are away from a successful build of the solution. We now have a technique that allows us to design our software according to requirement, and produces tests to give us an idea of how we are progressing.

Upon implementing the code to allow a particular test to pass, we also develop our code solution. Now, I'm wondering how we could possibly not want to develop good mature designs, unit tests and a solution using a technique that is directly aimed at that process?

One major barrier to the use of TDD are things like patterns. Don't misquote me here, design patterns are awesome, but what they often do is give a developer a design possibly before they fully understand the problem space. What that means is that as a developer, we're often tempted to jump straight in and start coding, without full consideration of the requirements. TDD forces you to implement only what is needed of the solution and nothing more. Design patterns lead developers to thinking that TDD is slow because they already know all the answers. What they should really be asking is, does that design pattern come with code and unit tests already constructed for their requirements. Almost certainly it does not.

Instead of rushing to start with a preconceived idea of implementation, a developer should instead use TDD to evolve the design. When it becomes apparent that the solution has evolved to a point at which the use of a design pattern is appropriate, to solve a requirement, then it should be used. In using this technique, a developer has documented evidence that such a pattern is a requirement of the solution, through the tests that are already in place.

Test driven development really requires the use of mock objects, which are strange to say the least, until you understand how they work, and then they become invaluable. When the design pattern requirement becomes apparent, a new set of tests should probably be constructed to test the pattern separately. In the original test project where the use of a pattern was discovered, mock objects should be used to 'pretend' that the pattern is being called.

We use mock objects so that a particular test class only tests one class directly, and everything else is mocked, and so has been assumed to be working unless the mocked code's unit tests fail.

A big part of TDD is that we end up with a very loosely coupled solution. This is largely due to the mock objects that I previously mentioned. Mock objects should really use interfaces to access code, and not concrete classes. Using a class mock would make the implementation be coupled to that class and so would be easily broken, or brittle in construction.

In working through code attempting to use interfaces, a developer will often come across resistance in one way or another. The problems may be due to the fact that the .NET framework does not offer an interface to a particular object directly. In that instance, what that is actually telling you is that you need to create your own wrapper class and interface. This is actually good news, if you stop and think about it, you are in effect abstracting functionality away from the implementation in the framework. This process would allow refactoring to take place in a much simpler way, and would also allow that implementation to be swapped out should the there be the need to do so. It is starting to sound like the beginnings of a Strategy, or Provider pattern now isn't it?! Shhh! Don't tell the naysayers!
One thing of note is that when the developer is trying to mock and test a particular aspect of the system, and it becomes hard work, that is often a good point at which to reconsider the implementation of those requirements. In the case of an object that doesn't have an interface, that leads us to wrapping up something that essentially provides functionality for us. Implementing tests on such a design should be once again very straight forward.

Generics are somewhat of an exception to the rule. They're great for giving type to classes, but unless you can perform all your tests on the functionality with only interfaces, you should consider refactoring the design, or hiding the generic interfaces behind non-generic interfaces. If this becomes too hard still, you may well find yourself wondering whether the generic class is really appropriate. This process is good, but don't do as many people do and blame TDD because you can't test the code; or at least don't keep the generic class that you can't test unless you're happy to have all aspects of the system that the generic class touches also be untested.

Red - green - refactor. What a shame that somebody came up with that as a mantra for TDD. Write the test, it fails, implement the code to make it pass. Seems pretty straight forward in concept, but as you can see from the discussion above, there is so much more to it.

It is strange that so many people like the idea of TDD, but so few have actually found a use for it. It is also a shame that there is so little information on the use of TDD and its practical implementations. I have seen many a simplistic example, and have given then myself in times, but when the technique is used commercially we want to be able to use it to its full potential, and solve every problem we come across with it, otherwise we simply will not use it.

If anybody has questions regarding TDD, please leave comments, and I will do my best to find answers, or offer possible solutions for you.

Agile - Should we all?

In the world of software it seems that it should be expected, no, encouraged that people come up with new ideas to make software more useful. Computing systems are no longer simple or can be conceived of in one effort.

With that idea in mind, why doesn't every project run in an agile manner? Well, I think it's down to the fact that people want to be able to budget, and constrain their costs. That's also of high priority, and completely understandable as an approach.

What happens then? Well, it seems that projects often have the higher levels of management, and those in charge of the purse strings winning the standoff. What happens then, is that the project becomes one that has all requirements defined up front, so that they can be detailed, and costs attached to them.

What happens now? Well, unfortunately, the prject is now forced into a waterfall type, up front design project. What that means is that either the client can't change their mind, and so can't evolve their ideas, or the project has to do work for new features without a budget. Is that good? From the point of view of the project is probably is, but the end product is going to suffer as a consequence.

Can we run a project an constrain it to a budget, and a project end date? You could do, but you would probably have to put limits on a budget for each iteration, which will in turn put a limit on the features. That limitation is imposed by the budget, so there's little we can do to limit that situation. However, what we do by running the project in this way is to allow the project to evolve and improve. Now, that's a good thing isn't it?

Why is it then, that most projects are not agile projects?

Risk, Who needs it?

Hello,

You're a software developer, you love to live by the seat of your pants, right? You can't function without loads of pressure, and the vast amounts of coffee that goes with it right? You'll probably want to stop reading this then!

So many projects and features for a piece of software get into a project plan without consideration to the risks involved. The client wants it, therefore it must go in. Is that a realistic proposition? Well, it might be, depending upon what the feature it is, and whether the inclusion of the feature overrides the risk involved in its implementation.

Something to consider, if late inclusion of a feature into an existing project is going to take that software back to a beta version, and the risk is unknown, would you do it? I wouldn't. You need to know the risks, and you need to be able to predict the worst case scenario, and describe what aspects of the software will be affected.

Given these assessments of impact and risk, you can attach a dollar cost to the cost of implementing the additional cost, versus the cost of not doing so, and always look at the worst case.

Imagine the number of features that started life as a throw away comment, and ended up being a major feature in a piece of software simply would not be there if decision makers knew the true cost of doing someting versus the cost of not doing something.