Member Discussions

Has anyone attempted to implement test driven development while developing a Mud?

I'm kind of interested in how one would approach emulating the client without doing lots of string matching (which would lead to fragile tests).

If everything is properly decoupled, I wouldn't have thought you would need to "emulate the client" very much. You would have a set of unit tests to verify that a representative set of text inputs (including all standard verbs) are parsed correctly. Everything after that should function in a way that doesn't depend on specific text input to make it happen.

So, you'd have a test to check what happens when a sword is picked up; the fact that this tends to happen in practice as a result of someone typing "get sword" into a telent window is irrelevant; what matters is whether the action of picking up the sword executes successfully or not.

Has anyone attempted to implement test driven development while developing a Mud?

In theory test driven development is writing the tests before the code. So for me yes and no, as I wrote the bulk of the TeensyMud test suite after coding the classes. Only for a few classes did I code the tests first.

I'm kind of interested in how one would approach emulating the client without doing lots of string matching (which would lead to fragile tests).

I used FlexMock to create "mock" objects. For instance I created a MockSocket object which responded to a subset of commands like recv() and would load the buffer with test data. In my case part of the problem with this is that actually testing how a given client responds to Telnet and VTxxx commands pretty much has to be done manually for each client.

Hi,
I am developing a C++ based codebased fully by the Agile XP method. It currently has over 200 unit tests and supports a lot of stuff in just 8500 lines of code. I'm still looking for a name, but once i have that, I'll Open Source it, so people can help improve it.
My conclusion is that Test Driven keeps the design very simple and elegant. The unit tests allow me to change stuff without worrying too much about breaking something.

200 unit tests dosnt sound anywhere near enough for 8000 lines of code. What coverage are you getting?


I'm still struggling with the concept of writing tests for business logic like the following:
When an object is dropped by a player, assure that all other players in the room recieve a message to say it was dropped.

I guess I could do a check and see that a message was received by the other players and see that the players name was in the message, that the item name was in the message and that the message contains the word drop...
At this point, its begining to smell of fragility a bit.

I am using Test Driven Development.

I find that, overall, it takes roughly the same amount of time to develop a feature in either case, but I am so very much happier with my code when using TDD, because I really do know it works exactly as I expect it to.

Also, because I use mock objects, it has improved my code because each piece of functionality needs to be usable in isolation from the rest of the system. Low coupling, and all.

I will mention that it does take some discipline not to go code something "small, so I know it'll work" without writing the test first, though.

I am using CppUnit and MockPP, and my "test LOC/source LOC" ratio is roughly 3/2.

Skarsnik:
When an object is dropped by a player, assure that all other players in the room recieve a message to say it was dropped.

I guess I could do a check and see that a message was received by the other players and see that the players name was in the message, that the item name was in the message and that the message contains the word drop...

The reason this is complicated is because you're trying to test too much at once: this is actually several separate unit tests. For example:

1) Does dropping the object trigger a "dropped" event successfully?

2) When a player enters the room, is he correctly added to the list of objects that are to be notified about a "dropped" event?

3) Is he correctly removed from the list when he exits?

4) When a "dropped" event happens, does the player object correctly pick it up?

5) When a "dropped" event is detected, does the text correctly match what it should do?

At the very least, the actual contents of the text should be calculated and tested separately from the triggering and detection of the "dropped" event. Any modifications you subsequently make to the way "dropped" events behave then cannot possibly affect the actual text displayed; they can only affect whether the text is actually displayed at all, not what it consists of.

Direct text-matching is a perfectly valid test so long as the only thing you're testing is the actual contents of the text message; other tests deal with whether or not there is a message.


(Comment added by shasarak on Mon Feb 25 7:19:02 2008)

For the record: I hate test-driven development. :-)

I'm thoroughly in favour of extensive, automated unit-testing: that's a necessity for making sure that any change you make doesn't break any existing code. But I've never been comfortable with the idea of writing the test before I write the code, for the simple reason that I'm usually three quarters of the way through coding something before I have the faintest idea how it's going to work.

I'm a very good programmer, and most of the processing that goes on in my brain happens at the subconscious level. I identify a solution as a good candidate not just because I can logically recognise it to be valid but because it feels right; and I've been doing this for long enough that my instincts are usually correct.

Initial analysis and design may be good enough to work out the outline correctly, but when it comes to the fine details (which is the level at which unit tests apply) if it turns out that a particular approach won't work, this usually doesn't become obvious until I'm well into the coding process. There's also usually some extensive refactoring that happens as I go along; the realisation that a few lines of code need to be a method in their own right rather than embedded in a larger method, or that, rather than A being a subclass of B, A and B should actually be subclasses of a not-yet-written, common, abstract ancestor - these are things that (again) tend not to become obvious until some way into the coding process.

So I prefer to hold off writing my unit tests until the code is in a semi-operational state; prior to that and I have to do so much rewriting of the tests that it just generates unnecessary work.

I program for fun, not as a job. For *me*, using
a super rigid development standard like what you
guys are talking about would just totally suck
the fun out of it.

When I was a kid I used to like writing essays
and such, but I could never do it like the teacher
wanted. They wanted outlines and stuff *first*, and
I just don't work that way. To me it was a creative
process that was squelched by using the dull tools
that the "normals" needed.

I've never used this "test-driven" stuff, but it
sounds like exactly the sort of thing that would
make me stop working on whatever it is.

Maybe it makes me a crappy developer not to
appreciate the beauty of such things, but I tend to
dislike focusing on process rather than the
joy of rolling around in code, getting it all
over myself and in my hair and laughing and gobbling
it all up.

-Crat
http://lpmuds.net

> I'm thoroughly in favour of extensive, automated
> unit-testing: that's a necessity for making sure that any
> change you make doesn't break any existing code. But I've
> never been comfortable with the idea of writing the test
> before I write the code, for the simple reason that I'm
> usually three quarters of the way through coding something
> before I have the faintest idea how it's going to work.

As someone who has only been doing TDD for about six months, I understand where you're coming from on that. My revelation was that you should test *small* -- which is exactly the same way you're coding already. You think a bit, write a bit. Think a bit more, write a bit more. After several iterations, you have a "big picture".

For example, I'm writing a virtual machine, and each instruction in the VM must have a number of tests. Each instruction relies only on the VM for its interaction ... which is mocked up, so I have complete control over it.

Let's take the SPUTCRRR instruction. This takes three register arguments (I call them accumulators in code to work around the register keyword) and, at index arg2, puts the character in arg3 in the string in arg1.

So I think: basic case: we read our arguments, and the string pointer is nil. What do I do? I chose nothing.

So, my test case looks exactly like this:

void sputcrrr_test::test_sputcrrr_nil()
{
    vision::mock_virtual_machine_chain vm("sputcrrr");
    
    // First, the instruction should read the three accumulator indices.
    vision::mock_get_argument8(vm, program_counter + 0, accumulator_string);
    vision::mock_get_argument8(vm, program_counter + 1, accumulator_index);
    vision::mock_get_argument8(vm, program_counter + 2, accumulator_character);
    
    // Next, it should read from the first accumulator to find out the address
    // of the string in question.
    vm.chain_get_accumulator
        .expects(once())
        .with(eq(accumulator_string))
        .will(return_(string_address))
        .id("get string address");

    // Then the instruction should check the address for nilness.  We're going
    // to respond true.
    vm.chain_is_nil
        .expects(once())
        .with(eq(string_address))
        .after("get string address")
        .will(return_(true));
    
    // Finding that the string is nil, nothing further should happen.  
        
    vision::sputcrrr sputcrrr;
    sputcrrr(vm);
    
    vm.verify();
}

To explain: MockPP has a featured called "chained" mocks, which essentially use a declarative syntax to describe what can and/or should happen. So

vm.chain_get_argument8.expects(once())

means that

vm.get_argument8

should be called exactly one time.

The first thing that happens is that the arguments to the instructions are read from the virtual machine. This happens with every instruction, so naturally I have dealt with that in a common function.

Next, we say that get_accumulator() must be called, with the accumulator the string pointer is contained in (as returned from the arguments to the instruction). It will contain a pointer value. Then, the instruction will ask if it's a nil pointer. We say yes, and verify that nothing else happens.

After writing that, I go and develop the instruction to a point that passes that test. It looks like this:

void sputcrrr::execute(virtual_machine &vm)
{
    using namespace memory_layout::string;
    
    u8  accumulator_dest  = get_argument8(vm);
    u8  accumulator_index = get_argument8(vm);
    u8  accumulator_char  = get_argument8(vm);
    
    u64 address = vm.get_accumulator(accumulator_dest);
    vm.is_nil(address);
}

At this point, I have one use-case and a little bit of code. Next, I asked: what would happen if the string were not a valid string? Each type in my system has a "magic number" that can be used to validate it. I implemented a test that did that (using C+P from my "nil" test, but changing the return for is_nil to false...). Then I amended the code to take the string's magic number into account.

The next test was: what if the index of the string was out of bounds? The final test was: what if everything's right?

And at each stage I knew that, if I broke the functionality I'd already written (and it does and did happen as I moved if-statements around), then I know instantly. I've been saved quite the number of "doh!" moments.

It may sounds quite onerous, but it's mostly what you do when you're writing on the fly anyway. Write a bit, test a bit. Build up the big picture over time.

Let's say you're implementing a "drop" command.
Basic case: what if no argument is specified?
Next: what if there is an argument, but it's not found?
Next: what if there is an argument, it is found, but it's cursed?
Finally: what if there is an argument, and it is found, and it's not cursed?

Over time, you might find more reasons not to drop something, and you can add them secure in the knowledge that you're not breaking any of the other reasons not to drop something.

It's really quite a useful tool.

(Comment added by Kastagaar on Mon Feb 25 9:12:04 2008)

Ignore the using declaration. That's spurious.

Kastagaar:
Let's say you're implementing a "drop" command.
Basic case: what if no argument is specified?
Next: what if there is an argument, but it's not found?
Next: what if there is an argument, it is found, but it's cursed?
Finally: what if there is an argument, and it is found, and it's not cursed?

The trouble is that the way my brain works, even that much forward planning is more than I'm usually able to do. What I would tend to do is just to say "right, let's implement a drop command." Then, a few lines in I would think "ah, hang on, what if the named object isn't actually here?" and go off and code that bit, then come back to the main function, then (after another few lines) think "oh, hold on, what if it's cursed?" then go away and write that bit, then come back to the main function again, and so on. My brain is only capable of coming up with the necessary exceptional cases by doing a lot of processing below the threshold of consciousness, and it only gets started doing that if I'm actually already in the middle of a related piece of code.

(Obviously this is a fairly trivial example; the chances are that if I were really coding a "drop" command I would actually be able to think of the necessary exceptions up front! But in the more general case of "thinking of all the exceptions that it is necessary to deal with" I generally don't find it helpful to try and make a list of them before I start coding; they only come into my head as part of the coding process.)

This is not to say I don't do design work up front; indeed, one of the things I'm quite good at in my day-to-day job is domain model design - that is, translating the real-life problem into entities and decisions simple enough for a computer to be able to make sense of them. But the process of converting that into actual code is far more iterative: I start coding, then feel my way through to the correct solution with the code continuously evolving under my hands.

There are two other issues I have with automated testing which make me less keen on it than some people are. The first is that (by definition) it's not possible to test for a condition you haven't yet thought of. Most of the errors in my code happen as a result of some combination of circumstances I hadn't anticipated might happen. If it's a combination I did anticipate then the odds are that I will have written the code to be able to deal with it and the test will pass. To be really useful, testing has to test for things I haven't thought of yet, which is impossible if I write the tests myself.

The other issue is the fact that it's very hard to do useful automated integration testing, as distinct from unit testing. I get a lot of errors that happen because an invalid parameter is passed into a function, but this is essentially untrappable by means of unit testing: you can test what happens to a function if an invalid parameter is passed, but most of the time there simply isn't any behaviour at all that is actually useful when operating on invalid parameters: raising an exception is as good a way of any of responding to it. What you need to test for is not "what happens if the parameter is invalid?" but "are there actually any ways for an invalid parameter to be passed in here?"; unit-testing that piece of code won't determine that, instead you need to test every possible output value in all of the places in the code where your current function is being called from (which, again, is impossible because you can't test anything against all possible values).

> The trouble is that the way my brain works, even that much
> forward planning is more than I'm usually able to do. What I
> would tend to do is just to say "right, let's implement a
> drop command." Then, a few lines in I would think "ah, hang
> on, what if the named object isn't actually here?"

You can do it that way too: write a test for the "Happy Path", and implement code that passes it. Find an exception. Write a new test that triggers that exception and modify the happy path so that it doesn't trigger that exception. Write code until both tests pass. *shrug* it's the same either way.

> Obviously this is a fairly trivial example; the chances are
> that if I were really coding a "drop" command I would
> actually be able to think of the necessary exceptions up
> front!

And this is where I find beauty in the system, because it requires that you design your code as small, decoupled, easy to understand bite-sized pieces.

You know, the way all the experts say you should.

(note, btw: I'm not saying that everyone must go use TDD now now now! There are probably many projects where it's a bad choice, and I also find it troublesome to "back-date" it to an existing project (which is why I only have TDD for the parts of my codebase that I've written within the last six months. Which, incidentally, are the parts I'm happiest with). What I'm saying is that I have used it, and a brief summary including information that it's been an overwhelmingly positive experience).

> The other issue is the fact that it's very hard to do useful
> automated integration testing, as distinct from unit
> testing. I get a lot of errors that happen because an
> invalid parameter is passed into a function, but this is
> essentially untrappable by means of unit testing:

I agree. Unit testing tests that a module works as you expect it to. It doesn't test that it works as other modules expect it to. It's just another piece in the Quality jigsaw.

For all the people arguing the pros and cons of test driven development. If you step back and think, as a concept it is really only 6-7 years old. Object oriented development, which was concieved in 62? and really only became mainstreem in the late 80s...

And still today I see alot of 'expert developers' with 10+ years experience behind them (that had bad mentors) fail to understand the basic principles of OO design and development.

TDD is still in its infancy.

/endrant

(Comment added by Skarsnik on Tue Feb 26 8:51:00 2008)

And yes, test driven is only one piece in the puzzle, if your not doing TDD, the chances are your not writing automated performance tests, or integration tests... as both of these are the logical extension of TDD...

Skarsnik:
And yes, test driven is only one piece in the puzzle, if your not doing TDD, the chances are your not writing automated performance tests, or integration tests... as both of these are the logical extension of TDD...

I prefer to think of TDD as an illogical extension of automated testing. :-)