Unit Testing in VS

Can't believe it, every time I look at it, how comical it is - generate unit tests from code.

Yeah, it allows you to get tests up and running quickly, but makes one big supposition - that the code that you're generating a test for actually works.

Make TDD easier, and remove the feature, it's silly.

Martin.

The best software to do "x"....

Hi,

Been pulling my team together for a new project, and whilst doing so, have been looking out for software that allows us to do things properly.
Now, before I start, I have to admit, that I absolutely love TFS, and VSTS, and that really makes it difficult to look for something else out there. Being realistic, to keep costs down in the current economic climate, I have been on the look out for cheaper or free alternatives. It seems that they're few and far between these days.

So first of all - non-VSTS unit testing? Well, clearly we're going to want a mocking framework there - that one is easy - Rhino Mocks. I've used NMock, which was great, but I find that once you're used to it, Rhino is the way to go. I tried Moq but found that it was less powerful than Rhino.

Okay, second on the list - code coverage. This one is difficult, Microsoft used to have a product, Coverage Eye which I used on a few occasions, and worked really well. There's NCover - that's quite expensive now, it seems, almost to the point where I should be considering VSTS which seems a lot better. The only alternative I found was partcover - it works, but the experience isn't at all integrated. And that leads me on to another point...

Unit testing framework... Okay, so the integrated MSTest framework is pretty good, and I get why you'd move the test output to a completely different location to where you would normally run from - to run the tests in isolation. No complaints from me there. However, trying to hook code coverage in to that build location from a coverage component that wasn't IDE integrated, and thus could easily capture the test folder location, is a pain. So much so that I'd suggest that using NUnit is unfortunately a necessity. The trouble is that then we're working development from our IDE, and tests from NUnit, which fires up code coverage. Not too bad a solution, but it can be a pain.

Personally, I'd really like to see something more in the professional versions of VS, what I'm saying is that every organisation is not so big it can afford, or wants to afford the all singing all dancing solution, much as I'm sure most would if it was cheaper. I'd love to see either the integration through VS to be brought a little more to the surface to open up the free version use a little better.

As previously mentioned, I love what Microsoft has done with Team System, and TFS, really I do, and I think for corporations that are able to afford it, they will go with that model. I would given the money, no doubts. Though, the alternative at the moment is really not the most compelling - I know, you say, "Save up and buy the proper version then", but at the moment, simply that's not going to happen, which cheapens the experience.

I think that the opening up of the IDE should be a little like love, you don't know that you've got it until you set it free!

And Microsoft, keep up the good work with these tools - for the first time in a long time, they really are impressive!

Unit Testing - Watch out!

I just thought that I'd write a little article to discuss some of the things to watch out for when unit testing. People tend to talk more about why it is or isn't good, and that you should use it, I'm proposing to tell you what to do when you do!

The first thing, the level of confidence in the test of the application is only as good as the tests that you write. Seems obvious, right? Using unit testing techniques does not make your code any more manageable, or correct if the tests that you write are of a poor quality. so all that you can say about unit tests, is that the code passes for the tests, and never that it 100% works, unless of course you are 100% certain that you have catered for all eventualities in the code.

Using code coverage as a tool to target your tests will not guarantee anything about the quality of confidence you should have in those unit tests. Here, I mean that having 100% coverage means that all the code is tested; it doesn't. What it means is that all the code has been exercise, but not that the code has been exercised with every possible value. We tend to use boundary conditions and the like to use in our tests reasoning that these values will best test our code. They don't, they're only representations.
Code coverage, you might not be able to exercise 100% of the code, if your code uses some .NET object that you can't exercise all the CLI for. You should find out whether that is the case though. It can often be an indication that the boundary condition by itself is woefully inadequate.

Integration tests. Until you do integration tests, you can't only speculate as to how well the code will work in the real environment. What happens if you get data that you didn't expect? What it means is that you write more tests, so that it doesn't happen again, even if the data was in error. The further upstream you go in terms of integration tests, toward a full system test, the better the tests are, and the more likely it is to fail. You're using real data, in real situations, and so can have a certain amount of confidence that the components at least work in that real world scenario. Look at exercising as much of the code using the coverage tools again, obviously some parts, such as testing exception handling belong only in the unit tests, but anything that can have data driven through it, to get as much coverage as possible is good.

If we do things right, then when we find errors, we write more tests, so that the tests become more and more complete, and our experience in where the tests could fail with the components increases, and thus gives us a better indication and knowledge of the risks involved with deploying the software.

So long as we bear in mind that we're never going to be able to say with 100% certainty, that a component is 100% bug free, we're probably safe. What we do with unit testing techniques is mitigate the risk, improve our knowledge of the system and its inner workings thus limiting risk and document part of the system.

These techniques are very useful, if used to their full potential, but never make a sweeping statement that the software is 100% bug free because all the tests pass, since that proves nothing.

Happy coding!

Real world test driven development (TDD) - Perhaps "Requirements Driven Design and Development"?

Test driven development seems to divide programmers into three different camps; those that think that it is the best thing since sliced bread, those that think it is a good idea but impractical due to the time it takes to use, and those that are completely ignorant of what exactly it is.

It seems, that if you start to try using TDD for any projects, it will very quickly become clear why you might want to implement using this technique. Simply, it isn't so much about writing tests, but more about evolving a solid, loosely coupled solution that meets the needs of the requirements and nothing more. From the point of view of that statement, surely nobody would argue that up front design is a good idea, and in so agreeing to that statement, you also agree that you agree with the principles of TDD.

But there's more! As a by-product of evolving this wonderful software design, that is neither over-engineered nor highly coupled, we also end up with tests, and code!
It probably seems strange that I have mentioned that we end up with tests; very quickly a practitioner of TDD will notice that tests are manifestations of the requirements.

So we're really looking at test driven development from the point of view of what we want to do with the solution, which is the software's requirements. This technique doesn't in any way imply that we don't gather requirements, but what is does do is allow us a framework by which to evolve yet more requirements as they become apparent. What I mean by that is that we have a bunch of tests, that will always test for functionality, and should that functionality cease to work correctly, the test will fail, and the requirement will not be met. This means that we are then able to add, or change functionality, and know that a particular requirement is or is not met.

Well, I've worked on a few projects in my time, and in every one, we are asked how far from completion we are. How can we tell? Well, if we're sensible, we have some unit tests. The number of not yet implemented tests, and the number of failing tests should give us an idea of how far we are away from a successful build of the solution. We now have a technique that allows us to design our software according to requirement, and produces tests to give us an idea of how we are progressing.

Upon implementing the code to allow a particular test to pass, we also develop our code solution. Now, I'm wondering how we could possibly not want to develop good mature designs, unit tests and a solution using a technique that is directly aimed at that process?

One major barrier to the use of TDD are things like patterns. Don't misquote me here, design patterns are awesome, but what they often do is give a developer a design possibly before they fully understand the problem space. What that means is that as a developer, we're often tempted to jump straight in and start coding, without full consideration of the requirements. TDD forces you to implement only what is needed of the solution and nothing more. Design patterns lead developers to thinking that TDD is slow because they already know all the answers. What they should really be asking is, does that design pattern come with code and unit tests already constructed for their requirements. Almost certainly it does not.

Instead of rushing to start with a preconceived idea of implementation, a developer should instead use TDD to evolve the design. When it becomes apparent that the solution has evolved to a point at which the use of a design pattern is appropriate, to solve a requirement, then it should be used. In using this technique, a developer has documented evidence that such a pattern is a requirement of the solution, through the tests that are already in place.

Test driven development really requires the use of mock objects, which are strange to say the least, until you understand how they work, and then they become invaluable. When the design pattern requirement becomes apparent, a new set of tests should probably be constructed to test the pattern separately. In the original test project where the use of a pattern was discovered, mock objects should be used to 'pretend' that the pattern is being called.

We use mock objects so that a particular test class only tests one class directly, and everything else is mocked, and so has been assumed to be working unless the mocked code's unit tests fail.

A big part of TDD is that we end up with a very loosely coupled solution. This is largely due to the mock objects that I previously mentioned. Mock objects should really use interfaces to access code, and not concrete classes. Using a class mock would make the implementation be coupled to that class and so would be easily broken, or brittle in construction.

In working through code attempting to use interfaces, a developer will often come across resistance in one way or another. The problems may be due to the fact that the .NET framework does not offer an interface to a particular object directly. In that instance, what that is actually telling you is that you need to create your own wrapper class and interface. This is actually good news, if you stop and think about it, you are in effect abstracting functionality away from the implementation in the framework. This process would allow refactoring to take place in a much simpler way, and would also allow that implementation to be swapped out should the there be the need to do so. It is starting to sound like the beginnings of a Strategy, or Provider pattern now isn't it?! Shhh! Don't tell the naysayers!
One thing of note is that when the developer is trying to mock and test a particular aspect of the system, and it becomes hard work, that is often a good point at which to reconsider the implementation of those requirements. In the case of an object that doesn't have an interface, that leads us to wrapping up something that essentially provides functionality for us. Implementing tests on such a design should be once again very straight forward.

Generics are somewhat of an exception to the rule. They're great for giving type to classes, but unless you can perform all your tests on the functionality with only interfaces, you should consider refactoring the design, or hiding the generic interfaces behind non-generic interfaces. If this becomes too hard still, you may well find yourself wondering whether the generic class is really appropriate. This process is good, but don't do as many people do and blame TDD because you can't test the code; or at least don't keep the generic class that you can't test unless you're happy to have all aspects of the system that the generic class touches also be untested.

Red - green - refactor. What a shame that somebody came up with that as a mantra for TDD. Write the test, it fails, implement the code to make it pass. Seems pretty straight forward in concept, but as you can see from the discussion above, there is so much more to it.

It is strange that so many people like the idea of TDD, but so few have actually found a use for it. It is also a shame that there is so little information on the use of TDD and its practical implementations. I have seen many a simplistic example, and have given then myself in times, but when the technique is used commercially we want to be able to use it to its full potential, and solve every problem we come across with it, otherwise we simply will not use it.

If anybody has questions regarding TDD, please leave comments, and I will do my best to find answers, or offer possible solutions for you.

Unit testing, Test Driven Development (TDD)

It still amazes me that there are so many projects out and about that have lots of lines of code, interoperating, and communicating systems, and no unit tests.

I don't know how projects get out in time, or even slightly slipping without having any confidence as to how much of the solution actually works.

I would have expected all greenfield projects, and some new additions to software to be able to be built using TDD, and be able to be refactored, and for delivery of the project to be able to be done with confidence.

I know that it often scares people to be using MVC and MVP patterns and dependency injection, but I know that it would scare me more to be in charge of a project that didn't have any unit testing.

Another thing, of projects that I have worked on that have had unit tests, they're often not unit tests, and are instead system or integration tests. Who cares what they are, right? If you run an integration test, and the server goes does, does that mean that your code has failed to pass its requirements? Will it take you an age to figure out if the problem is environmental including lots of debugging, and so on? Quite possibly, as there's so much involved in the real system, so many dependencies. Ultimately you need those tests, but to be able to write solid code, unit tests are what you need.

What else intrigues me is that you hear so many people talking Agile, and yet so few actually show any evidence of being capable of doing it. It seems implausible to run an agile project without unit tests to constrain your planned and unplanned changes to code.

I heard often of people complaning that test driven development takes a lot of overhead, well yes it does, but in the long term, that will mean less time to finish the project, unless it is the most simple project you've ever seen, and even then, I think you could argue a good case for TDD.

I know there are people out there who use Agile and TDD type methodologies, please show yourselves, and make yourselves known to me, and to the world at large!