Exploratory Data Analysis
Do vary all relevant factors
Don’t change two things at once
- Ascribed effects of heuristic to the algorithm
- changed heuristic and algorithm at the same time
- didn’t perform factorial experiment
- But it’s not always easy/possible to do the “right” experiments if there are many factors
Do measure CPU time
- In exploratory code, CPU time often misleading
- but can also be very informative
- e.g. heuristic needed more search but was faster