Validity of Computational Simulation

As a computational scientist, I believe it is important that I question the validity and reliability of my work. And that is precisely what I will be talking about today.

Computational simulation is a strange game. The entire process of simulation can usually be laid out as follows:

(1) Development of a mathematical model which approximates reality.
(2) Development of a computational model which approximates the mathematical model.
(3) Extraction of useful information from the computational model.

Item (3) has little room for error. Item (2) is more prone to error. However, we have a good understanding of how to estimate or control this error and thus verify our computational model. In many cases, we can use hard mathematical analysis to quantify how different our computational model is from our mathematical model. We can use adaptive methods to control this error and even drive the error to zero. Item (1), on the other hand, can be a bit of a mystery.

The process of determining how well a mathematical model approximates reality is known as validation and is currently an extremely hot topic in computational science. The most common validation techniques involve comparing the results of a mathematical model to those of reality for certain, often simple, explicit scenarios. These validation experiments are often combined with parameter estimation in order to arrive at "better" models. However, these validation experiments by no means quantify how close our model is to reality in every scenario. In fact, it is often impossible to validate a complex system for even simple scenarios. The data just simply isn't there. How can we begin to trust our model when validation isn't even an option? Furthermore, we run into more awkward situations when we have an abundance of data. In these situations, one often finds it is impossible to tune parameters in order to eliminate error from every validation experiment. One either has to either (i) introduce more parameters and hence introduce a more complex model or (ii) choose a subset of validation experiments. Option (ii) immediately seems like a cop-out. However, option (i) often leads to models that aren't even computationally tractable. Of course, one can point to the rapid increase in computational resources and say, "Just wait a few years. Then we can simulate whatever we want." However, as our computational resources have become larger, so have our problems, often at a faster rate. Will this trend continue? I feel that these are important questions that we as computational scientists often ignore.

One last thought: if we can, theoretically, pinpoint the "modeling" error associated with a mathematical model down to say 50%, is there a reason to adaptively reduce the "mathematical" error associated with a computational model to, say, below 5%? Or, more simply put, shouldn't we establish a direct relationship between validation and verification?

With all this in mind, I pose the same question that one of my mentors, Professor Ivo Babuska, often does: "Will you sign the blueprints?"