Predicting project performance: Evaluating the forecasting accuracy

Submitted by Mario Vanhoucke on Tue, 01/10/2012 - 10:44

Controlling a project is key to the success or failure of the project. Earned Value Management (EVM) is a well-known technique to control the time and cost performance of a project and to predict the final project duration and cost. Predicting the final project duration and/or cost of a project in progress, given the current project performance, is a crucial step during project control. In an EVM analysis, quite a number of time and cost forecasting techniques are available, but it is however a cumbersome task to select the right technique for the project under study.

In this article, a simple simulation approach is presented to guide the project manager in his/her selection of the most accurate time and cost forecasting technique of the project in progress. The technique evaluates the accuracy of all EVM predictions of a project by means of a computerized simulation model. An overview of the EVM metrics is given in “Earned Value Management: An overview” and the formulas are summarized in “Earned Value Management: The EVM formulary”. The approach to evaluate the accuracy of the forecasting techniques available in EVM consists of a three step procedure, as summarized along the following lines:

Step 1. Define uncertainty
Step 2. Simulate project progress
Step 3. Evaluate the forecast accuracy

Define uncertainty

Uncertainty on the activity durations or costs, resource use, the presence of precedence relations or even on the existence of an activity in the project network is what typifies projects. Therefore, single point estimates should be better replaced by probability distributions to incorporate this uncertainty. In doing so, project parameters that are considered to be deterministic are then modeled as random variables that enable the project manager to develop a computerized simulation model that imitates project progress. More information can be found in “Project risk: statistical distributions or single point estimates?”.

Simulate project progress

Random numbers are generated from the probability distributions defined earlier to imitate a fictitious project progress run. This process is repeated a (huge) number of times such that each simulated project run is different. The general idea is that each simulation run reflects a realistic project run that can possibly occur when the project is in real progress. More information on the use of simulation runs in project scheduling is given in “Monte-Carlo simulations: How to imitate a project’s progress”. A simple simulation approach consisting of 8 special simulation scenarios is proposed in “Monte-Carlo simulations: Linking critical path schedules to project control”.

Evaluate the forecast accuracy

During each project simulation run, the periodic time and cost performance of the project is measured using Earned Value Management data which are then used to predict the final duration and total real cost of the project. At the end of each project run, the simulated project duration and cost is known and is compared to the planned duration and budgeted cost in order to evaluate the accuracy of these periodic predictions. More information on the general concept of making predictions using EVM data is given in “Earned Value Management: Forecasting project outcome”.

The experienced project manager who is knowledgeable about the use of EVM metrics and predictions can choose among a set of time and cost prediction techniques, as discussed in “Earned Value Management: Forecasting time” and “Earned Value Management: Forecasting cost”. However, despite the rich availability, no guidelines are given on which techniques to use for a specific project.

This is where the Monte-Carlo simulation runs come into play. The periodic forecasts can be compared with the known project durations and costs and the accuracy can be evaluated by means of the two following measures:

Mean Absolute Percentage Error (MAPE): The average of the absolute values of the relative deviations between the periodic time/cost predictions and the final project duration/cost, as follows:
- MAPE (time) = 1 / #Periods * SumAllPeriods {|EAC(t) - RD| / RD}
- MAPE (cost) = 1 / #Periods * SumAllPeriods {|EAC - RC| / RC}
Mean Percentage Error (MPE): The average of the relative deviations between the periodic time/cost predictions and the final project duration/cost, as follows:
- MPE (time) = 1 / #Periods * SumAllPeriods {(EAC(t) - RD) / RD}
- MPE (cost) = 1 / #Periods * SumAllPeriods {(EAC - RC) / RC

with

EAC(t): Time forecast (at each periodic review period)

EAC: Cost forecast (at each periodic review period)

RD: Real duration (known upon completion of each simulation run)

RC: Real cost (known upon completion of each simulation run)

#Periods: Number of periodic reviews in EVM (= number of EAC and EAC(t) values)

Consequently, the MAPE and MPE are calculated in a similar way. While the MAPE gives an indication about the average deviations as a non-negative value, the MPE calculations can result in both positive and negative values and measure over- or underestimations of the final project duration or cost.

The MAPE and MPE calculations can be easily illustrated on the nine time forecasts of the example given in “Earned Value Management: Forecasting time” and the eight cost forecasts given in “Earned Value Management: Forecasting cost”. For the sake of simplicity, these forecasts are copied in table 1. The planned duration of the project is equal to 9 weeks while the real duration equals 11 weeks (2 weeks delay). The planned cost is equal to € 150 while the real cost equals € 210. In the last two columns, the MPE and MAPE values are calculated for all methods. These calculations are described in detail below for the EAC(t)_PV1 time forecasting method and the EAC_V1 cost forecasting method.

MPE (time) = {(9.10 - 11) + (9.20 - 11) + (9.90 - 11) + (10.95 - 11) + (9.90 - 11) + (11.25 - 11) + (11.40 - 11) + (10.20 - 11) + (10.20 - 11) + (9.60 - 11) + (9.00 - 11)} / (11 * 11) = -0.09
MAPE (time) = {|9.10 - 11| + |9.20 - 11| + |9.90 - 11| + |10.95 - 11| + |9.90 - 11| + |11.25 - 11| + |11.40 - 11| + |10.20 - 11| + |10.20 - 11| + |9.60 - 11| + |9.00 - 11|} / (11 * 11) = 0.10
MPE (cost) = {(156.67 - 210.00) + (163.33 - 210.00) + (170.00 - 210.00) + (202.50 - 210.00) + (200.00 - 210.00) + (222.50 - 210.00) + (230.00 - 210.00) + (225.00 - 210.00) + (220.00 - 210.00) + (215.00 - 210.00) + (210.00 - 210.00)} / (11 * 210.00) = -0.04
MAPE (cost) = {|156.67 - 210.00| + |163.33 - 210.00| + |170.00 - 210.00| + |202.50 - 210.00| + |200.00 - 210.00| + |222.50 - 210.00| + |230.00 - 210.00| + |225.00 - 210.00| + |220.00 - 210.00| + |215.00 - 210.00| + |210.00 - 210.00|} / (11 * 210.00) = 0.10

These calculations show that the time forecast has an absolute deviation of 10% with an average underestimation (MPE = -9%). Similarly, the cost forecast is on average slightly underestimated (MPE = -0.04) with an absolute deviation of 10%. The MAPE and MPE values in table 1 show that the best time forecasting method is EAC(t)_ES1 and best performing cost forecasting method is EAC_V3-SPI(t), which is obviously not a general conclusion but only holds for the data in the table.

It should be noted that this illustration is somewhat biased and can certainly not be generalized since it does not rely on a simulation with multiple project progress runs, but instead the values in the table are given for only one project progress run. In order to evaluate the forecasting accuracy in a more clever way, multiple project progress runs should be performed, such that the averages of all MAPE and MPE values can lead to a much more reliable decision to select the technique that fits best for the project.

Table 1. Nine periodic time forecasts and eight periodic cost forecasts

	W1	W2	W3	W4	W5	W6	W7	W8	W9	W10	W11	MPE	MAPE
	9 time forecasting methods
EAC(t)_PV1	9.10	9.20	9.90	10.95	9.90	11.25	11.40	10.20	10.20	9.60	9.00	-0.09	0.10
EAC(t)_PV2	13.50	13.50	22.50	22.00	10.93	13.09	13.00	10.50	10.38	9.64	9.00	0.22	0.30
EAC(t)_PV3	40.50	40.50	67.50	73.33	18.73	24.60	24.56	17.06	15.98	14.12	12.60	1.89	1.89
EAC(t)_ED1	9.33	9.67	10.80	11.36	9.88	10.88	11.15	10.14	10.20	10.67	11.00	-0.05	0.06
EAC(t)_ED2	13.50	13.50	22.50	22.00	10.93	13.09	13.00	10.50	10.38	10.71	11.00	0.25	0.27
EAC(t)_ED3	38.50	36.50	61.50	64.00	15.16	19.32	18.33	12.06	11.13	11.05	11.00	1.47	1.47
EAC(t)_ES1	9.33	9.67	10.00	10.17	9.50	10.08	10.86	11.00	11.00	11.00	11.00	-0.06	0.06
EAC(t)_ES2	13.50	13.50	13.50	12.71	10.00	10.98	12.25	12.00	11.57	11.25	11.00	0.09	0.11
EAC(t)_ES3	38.50	36.50	34.50	33.02	13.57	15.36	16.92	14.50	12.96	11.83	11.00	0.97	0.97
	8 cost forecasting methods
EAC_V1	156.67	163.33	170.00	202.50	200.00	222.50	230.00	225.00	220.00	215.00	210.00	-0.04	0.10
EAC_V2	450.00	450.00	450.00	500.00	257.14	281.82	283.33	243.75	230.77	219.64	210.00	0.55	0.55
EAC_V3-SPI	230.00	235.00	380.00	386.67	217.14	253.18	256.67	230.00	223.08	215.71	210.00	0.23	0.23
EAC_V3-SPI(t)	230.00	235.00	240.00	255.00	208.89	237.37	251.67	235.00	225.71	217.50	210.00	0.10	0.10
EAC_V4-SCI	670.00	665.00	1080.00	1113.89	286.53	339.46	333.70	251.88	235.50	220.69	210.00	1.34	1.34
EAC_V4-SCI(t)	670.00	665.00	660.00	675.00	272.38	309.76	324.26	260.00	239.56	223.30	210.00	0.95	0.95
EAC_V4’-SPI	376.67	378.33	433.85	471.19	246.71	274.83	276.76	240.20	228.85	218.64	210.00	0.45	0.45
EAC_V4’-SPI(t)	376.67	378.33	380.00	409.06	243.71	269.47	275.18	241.71	229.61	219.16	210.00	0.40	0.40

PM Knowledge Center

Predicting project performance: Evaluating the forecasting accuracy