Testing maps

20 March 2004

The results from the first roll-over of Net Mike are available now. Until Monday, teams will only know the results in their own world. I wonder what lessons lessons are taken away from this first roll-over. How many people will consider their assumptions confirmed? What will happen in those firms that did not perform according to the managers expectations?

All over the know Mike’s Bikes universe maps will be confirmed or disconfirmed. Maps will be considered reliable or will be discarded and new maps will be sought for.

On one level, I’m a big fan of the scientific method especially in its guise of falsificationism. As an aside, I occasionally teach a course called Managing Science & Technology. At the start of the course we (it is taught by the faculties of Science and Business & Economics) often ask What is the scientific method?, and usually this is followed by along silence in the class. Anyway, the rough idea of the scientific method is that we develop theories about how ‘things’ work and test those theories by look for proof that the theory is broken. For example, if we believe that all swans are white, one approach to proving that would be to look at swans. Every time we saw a white swan, that would mean our theory was working. If we found a black swan, we’d have to discard that theory and come up with a better one. Of course, it would be nice to get all the swans together and check them all, but sometimes that luxury is not available.

As yet another aside, the expression It is the exception that proves the rule is often misused. The use of the word prove in this case is similar to test; in other words, the expression should be It is the exception that tests if the rule [is right or wrong]. Thus, if there is an exception, then the rule is wrong (or incomplete).

Anyway, for a more complete explanation of falsificationsim read about Karl Popper and here.

However, what happens in reality is different. When asked to prove something, people will seek confirmation rather than disconfirmation. So, rather than looking for a swan that is not white, people take every sighting of a white swan as further evidence that they were right. So, if you want to know if the map you are using is a good map, look for the ways in which it is wrong, rather than the ways in which it is right – then you’ll know the limits of the map (because we already know maps are wrong in some ways, and right in others).

So, for example, if you believe that increasing marketing has a positive impact, how can you prove it?