How to plan an experiment?
DoE (Design of Experiments), in our experience, is the most effective way to collect data and increase understanding of cause-effect relationships. What we learn from a DoE depends primarily on how we plan it. No level of analysis advancement will be helpful if the data we collect lacks knowledge – and how much new knowledge we gain and what kind depends primarily on the planning stage. Analytical skills are, of course, necessary, but it’s a relatively simple procedure to learn and is reproducible in every DoE. Planning, however, although it can be systematized, will be different in each experiment.
Establishing the Goal of the Experiment
To avoid focusing on technical aspects, we’ll use an example of a process everyone is familiar with: hard-boiling an egg. Some people may have different requirements, but for us, the ideal hard-boiled egg is one in which the yolk and white are completely set, without any signs of rawness, but also without any signs of dryness. The white should be set, firm, and elastic, matte white without porosity. The yolk, also set, should have a creamy consistency and be golden brown, without a green/gray rim. The egg’s flavor should be neutral, without a metallic aftertaste, which indicates overcooking. The shell should be free of cracks, easy to peel, and the egg’s surface should be smooth after peeling.
The goal of the experiment will be to test how the process factors at our disposal affect meeting our requirements.
Importantly, the goal of the doe is to create variation. This variation should be greater than what we observe on a daily basis. If we only create perfect eggs during the doe, we won’t really learn anything. The ideal situation is one in which, during a single doe, we obtain eggs ranging from those with a slightly runny yolk, through perfectly creamy but solid, to completely overcooked eggs with a green rim. The same applies to peeling. Ideally, it would be best to create a variety of situations. From difficult situations, where the egg is difficult to peel and some of the egg white remains attached to the shell, to eggs that can be peeled quickly and without protein loss.
Depending on the strategy adopted, you could set the first stage as the goal of learning how to achieve a perfect yolk, and only then, in the second, set the goal of learning how to peel the eggshell. This is especially true when we have a large number of factors, and current knowledge tells us that yolk quality is determined by factors other than ease of peeling.
In a single experiment, we will create variation in both of these egg characteristics.
Choosing Testing Factors
Keeping in mind that the goal of the experiment is to create variation in both the structure of the yolk after cooking and the ease of peeling, we define the factors that will create this variation.
In our opinion, the factors that influence both characteristics of a hard-boiled egg are:
- Temperature of the egg before cooking
- Water temperature at the start
- Burner power
- Cooking time
- Cooling method after cooking
- Salt addition to the water
How should you choose factors? Primarily, based on your current knowledge of the process and product. Current theories, questions, ideas, and observations. You can, of course, use a Process Map, invite others knowledgeable about the process to participate in the planning, or consult the literature – but expert knowledge is key here. If there are many factors, it’s worth conducting a Component of Variation Study (CoV) to determine the dominant sources of variation – but that’s a completely separate topic.
Although each of you may have ideas for additional factors at this point or believe that some of the above are irrelevant (which reflects the situations you’ll encounter when planning an experiment ), for the purposes of this example, let’s assume that the above six factors exhaust our expert knowledge of the hard-boiled egg process.
Choosing Testing Levels
Choosing factors is just the beginning. In fact, choosing the testing levels will define the variation we create. The testing levels define the experimental space. Theoretically, there are three requirements for testing levels:
- They cannot be too narrow
- They cannot be too wide
- They should be evenly distributed across all factors
Take the factor selected above as an example: cooking time. By testing level, we mean defining the cooking times we want to test. For now, assume that within a single doe, the best strategy is testing at two levels. There are several reasons for this, but more on that another time.
We need to establish two cooking times that will create variation in at least one of the interesting characteristics of a hard-boiled egg. Cooking time will certainly affect the structure of the yolk. For example, levels that are too narrow: 12 minutes vs. 13 minutes. These times are too close together, which will result in very similar yolks, and the doe analysis will show no effect of cooking time—which is not true, but rather a consequence of the levels we defined.
- Example of levels that are too wide: 5 minutes vs. 15 minutes. Here, we’ll create unrealistic variation, and the entire cause-and-effect relationship lies somewhere between these levels.
- Evenly distributing the levels means that we should plan the levels so that the tested factors have a similar impact on the yolk structure.
It sounds scary, but believe me, this is just a theory. It’s important to remember that if we knew exactly which factors to define at which levels to create the entire range of variation we planned, we most likely wouldn’t need DOE on this topic. We experiment in areas where we have problems and challenges, and therefore we are aware that we don’t know everything.
Based on how to define the levels:
- Current knowledge of the process/product. Often, one level is planned as having a negative impact and the other as a positive impact.
- Current theories, ideas, and questions we want to answer.
- Historical data. Depending on the experiment’s goal, we go beyond the current variation of a given factor or test its current extreme settings.
- Instructions from machinery or raw material suppliers as one level, and the other as our predicted better level for a given factor.
- Comparison with competitors, comparison with similar processes.
- Results from the previous DOE (this option is not available the first time around). The DOE analysis will tell us how we defined the levels and whether they need to be expanded, for example, or moved to a different level entirely.
We set the testing levels for our example as follows:
- Egg temperature before cooking: refrigerator vs. room temperature
- Water temperature at start: cold (tap) vs. boiling
- Burner power: 7 vs. 9
- Cooking time: 6 vs. 12 minutes (time counted from the water boiling)
- Cooling method after cooking: none vs. cold water
- Salt addition to water: none vs. a tablespoon
Selecting factors and testing levels is just the beginning. In the next section, we’ll cover the next steps in experiment planning, such as setting inference constraints and determining the necessary measurements during the experiment.
