Chapter 5.

0
5.1 Evaluation Criteria

System Evaluation

The latter half of the previous chapter defines and describes the system requirements and functional requirements essential for the real-time advanced warning and traffic control system to be deployed on I-81. The system should be designed and developed to meet the system and functional requirements specified. Once the system is designed and developed in compliance with the predefined requirements, the next step would be to carry out an evaluation of the system to ensure its effectiveness in resolving work zone safety and traffic control issues. The main objective of carrying out this evaluation is to verify how well the system fulfills the intended objectives. In order to perform a comprehensive evaluation of the system, appropriate evaluation criteria that address system performance issues need to be developed. This chapter identifies and describes the potential criteria that can be used for the evaluation of a real-time advanced warning and traffic control system. It also identifies measures of effectiveness (MOEs) required to quantify the performance benefits during or after system usage. Finally, this chapter examines the issues related to evaluation, explaining how to quantify each of the MOEs and highlighting the various issues that need to be considered during the evaluation of the system.

The criteria that may be used for the evaluation of the real-time advanced warning and traffic control system include:

1. Functionality 2. System reliability 3. Performance Criteria - Measurement Accuracy - Algorithmic Accuracy 4. Installation Ease

5. Operation and Maintenance Ease 6. Technology Maturity 7. Portability 8. Adaptability, and 9. Cost

Following is a description of each of the evaluation criteria:

System Evaluation
67

1. Functionality: This criterion refers to evaluating how well the system meets the predefined functional requirements. As per the functional requirements, the system should be capable of detecting congestion, incidents, and queues within the work zone. In the event of congestion, incident occurrence, or queue formation, the system should be capable of providing warning and advisory messages to prevent the situation from worsening and to return traffic flow conditions back to normal. This criterion also ensures that the system is functionally sound and successfully performs the required calculations to detect anomalies in traffic flow. When evaluating the functionality of the system, two important features should be verified. They are, how many of the defined functional requirements does the system meet and secondly, how well does the system meet these requirements. The functionality of the entire system as a whole, in which the functions of each component complement the functions of the others, is another essential criterion that needs to be evaluated. The entire system should function efficiently as a single unit and display sensible outputs. For example, if the system does not sense any problems with traffic flow within the work zone, conflicting messages should not be displayed to motorists. In general, this criterion ensures that the system performs its functions as intended in fulfilling its objectives.

2. System Reliability: The temporary nature of work zones requires that the real-time advanced warning and traffic control system be moved from one work zone to the other under varying climatic conditions and at different geographic locations. The system should thus be rugged enough to sustain the frequent dismantling and moving activities associated with work zone applications. Additionally, the designed system should function reliably at all times and under different conditions (climatic and geographic), with no or minimal breakdowns. This criterion, therefore, ensures the reliability of the system in performing efficiently and accurately under varying conditions. The reliability of the system depends on the reliability of the individual components and the efficient integration of these components to work as one cohesive unit. The reliability of the system can be measured by the number of system failures and false alarms that occur.

System Evaluation

68

3. Performance Criteria: The system employs various algorithms, measures several traffic flow parameters, and calculates important traffic parameters to determine the status of traffic within the work zone. It is extremely important that the system does not fail in performing any of these functions accurately. The performance of the system in carrying out these tasks can be evaluated using the following two criteria:

- Measurement Accuracy: The system will need to measure many traffic flow variables (speed, occupancy, etc.) to monitor traffic conditions and to make advisory and control decisions. There is a need to evaluate the accuracy of these measurements to ensure that the system is able to make the correct decisions. The measurement accuracy of the system can be evaluated by comparing known values of traffic variables with systemderived values.

- Algorithmic Accuracy: The system will employ algorithms to translate raw traffic data into useful information on traffic conditions, occurrences of incidents, congestion, and queues, and to display appropriate messages to motorists. The accuracy of these algorithms needs to be evaluated to ensure that the system is correctly interpreting the actual traffic conditions within and around the work zone, and relaying accurate information/messages to motorists. Again, this may be verified by observing if the system correctly interprets current traffic flow conditions and displays appropriate messages.

4. Installation Ease: The system should be easy to install in all types of work zone configurations and should be ready for use in a short period of time. The installation should require minimum manpower. The easier the installation, the better it is for deployment at various work zone sites. As an estimate, the system should take no more than two experienced personnel no longer than one hour to set up.

System Evaluation

69

5. Operation and Maintenance Ease: The operation and maintenance of the system is an important factor that needs to be considered during an evaluation. The system, once installed at a work zone, should be easy to operate and maintain. All operations that need to be performed should be simple and obvious. Basically, the system should be userfriendly, uncomplicated, and designed such that it is hard to make a mistake. In the initial stages of deployment, the system may be operated by trained personnel on a regular basis, but later it should have the capability of operating autonomously. Ideally, the system should be maintenance free, but this is not practically possible. The software employed by the system should function as expected at all times, while the hardware should have minimal maintenance requirements. Other important considerations include availability of spare parts and the lifecycle of components.

6. Technology Maturity: The system needs to be evaluated for maturity of the technologies/components being used. Mature and well-tested technologies/components can greatly enhance the reliability and performance of the system. A point that may be noted is that several technologies may be proven to work efficiently but may not be used effectively in the transportation environment.

7. Portability: Portability of the entire system is another important issue that needs to be evaluated. The temporary nature of work zones requires that the system be portable, easy to dismantle, and easy to put together. If the system is portable, it can be shifted to various work zone sites and re-installed with minimal effort. Portable subsystems make it easy to shift the position of some devices as required by changing traffic conditions within the work zones.

8. Adaptability: Another desirable feature of the system is adaptability. Since work zones are of different types and configurations, the system must be adaptable to these changing conditions. Modularity of components/subsystems is another desirable feature.

System Evaluation

70

9. Cost: The cost of the system as a whole, as well as its operation and maintenance costs, will be a very important factor in adopting the system for use. The use of lowpowered system components such as sensors and variable message signs and solar power for operation of these components may help reduce operation costs of the system.

5.2

Measures of Effectiveness (MOEs)

All systems are developed to satisfy certain functions. These functions may be easy to state but at the same time can be difficult to measure. Measures of effectiveness (MOEs) provide a quantitative basis for determining the effectiveness of the system in fulfilling the functional requirements. The level to which the system meets the required functions can be effectively determined by comparing before and after measurements of the MOEs. The real-time advanced warning and traffic control system should be evaluated using some type of performance measures to ensure the effectiveness of the system in meeting its objectives and goals. Given below is a list of potential measures of effectiveness (MOEs) that may be used for system evaluation. These MOEs can be categorized under two types, namely MOEs for the entire system and MOEs for system performance (Table 9).

Table 9. Categories of Potential Measures of Effectiveness

Measures of Effectiveness (MOEs)

MOEs for the Entire System Reduction in the number of occurrences of congestion and queues Reduction in speeds Reduction in mean queue length Reduction in delays Reduction in the number of accidents/severity of accidents Speed and advisory compliance MOEs for System Performance Accuracy in measurement of traffic flow variables Incident detection rate Mean detection time False alarm rate

Each of these MOEs are geared towards the four basic functional requirements of the realtime advanced warning and traffic control system, namely surveillance, advanced warning,

System Evaluation

71

advisory, and control functions. Additionally, these MOEs are formulated to quantify the effects of the system on accuracy, congestion, queues, delays, incidents, and motorist compliance. Following is a description of each of these MOEs.

MOEs for System Performance:

1. Accuracy in measurement of traffic flow variables: The real-time advanced warning and traffic control system employs traffic sensors to gather information on various traffic flow variables. Some of the important flow variables that may be measured include speed, traffic count, occupancy, and flow. The variables measured are then analyzed by the system. Based on the analysis, decisions regarding the status of traffic within the work zone are made. This logic behind the working of the system clearly depicts the reliance of the entire system operation on the accurate measurement of traffic flow variables. It is thus necessary to evaluate and confirm the accuracy of the system in measuring the required traffic flow variables. This MOE verifies the accuracy of the measured variables to be used by the system. This is done by comparing system-calculated values to previously known values of the traffic flow variables.

2. Incident detection rate: Incidents are the main cause for the occurrence of congestion and queuing. Incidents vary from a vehicle stranded on the shoulder to an accident involving one or more vehicles. Any form of incident reduces roadway capacity, in some cases causing congestion and queuing. When the amount of congestion and queuing builds up, it leads to unwanted delays, loss of time, fuel, and motorist frustrations. Thus, incident detection is an important subset of the real-time advanced warning and traffic control system. The incident detection rate is a measure that verifies the accuracy of the incident detection algorithm. This MOE may be represented as a percentage and may be defined as the percentage of detected incidents among the total number of incidents. A high incident detection rate is desirable for the system. Ideally, a 100 percent incident detection rate is desirable, but may not always be possible. Thus, the system must work efficiently and have highest possible incident detection rates. It is essential that the system have a high incident

System Evaluation

72

detection rate, as this plays an important role in the effectiveness of the system to adequately warn motorists and facilitate appropriate traffic control.

3. Mean detection time: This MOE refers to the mean detection time for an incident, congestion, or queue. The mean detection time may be defined as the mean time interval between the start of an incident/congestion/queue and its detection. The system should be capable of detecting an incident/congestion/queue as soon as possible. The sooner these problems are detected, the easier it is to contain their effects and return traffic flow conditions to normalcy. Thus, it is desirable to have the mean detection time as small as possible.

4.

False alarm rate: This MOE is pertinent to the system capabilities of detecting

anomalies in traffic flow within the work zone and may be applied to three cases, namely the detection of incidents, congestion, and queues. The false alarm rate verifies the accuracy of the algorithms involved in detecting incidents, congestion, and queuing. A low false alarm rate is highly desirable. If possible, the system should have no or a minimum number of false alarms. This will help save useful resources. The false alarm rate is represented as a ratio and may be defined as the ratio of the number of false alarms for incidents/congestion/queuing and the total number of alarms for each of these.

MOEs for the Entire System:

1. Reduction in the number of occurrences of congestion and queues: Recurring congestion and queuing is a severe problem faced at all highway work zones. These two features may be considered to be interrelated as increased congestion finally leads to the formation of queues, which may initially be moving but may ultimately end up as stopped queues. These events pose potential hazards for rear-end collisions, which would further contribute to congestion and queuing problems. Thus, there is a need to reduce the occurrences of such hazardous situations within the work zone. The system when in use should prevent the occurrence of congestion and queues by effectively regulating the flow

System Evaluation

73

of traffic. The effectiveness of the system in preventing the frequent occurrence of congestion and queues may be gauged by measuring the difference in the number of occurrences of congestion and queues with and without system usage. The reduction in the occurrences of congestion and queuing may be calculated by keeping a record of the number of queue and congestion formations experienced within the work zone for a certain period of time. The record should be kept for both with and without system implementation scenarios. A comparison between the two scenarios may be used as a basis to establish the reductions in the occurrences of congestion and queues. Issues such as time frame for taking observations, definitions for congestion and queuing, and sample size need to be decided upon by the evaluator in advance of carrying out the actual evaluation.

2. Reduction in speeds: Speeding within work zones is one of the major issues that needs to be addressed in terms of work zone safety and traffic control. Several studies have shown that exceeding safe speed within the work zone contributes to a significant portion of the work zone-related accidents. Innovative devices such as unmanned drone radar have been developed and evaluated for their effectiveness in reducing speeds. The realtime advanced warning and traffic control systems have the advantage of being dynamic in nature and therefore may help reduce excessive speeding within the work zone. Using advisory and control measures and dynamically warning motorists of possible dangers within the work zone, the system can be designed to encourage reductions in traffic speeds and speed variance. There are several methods available for measuring speed reductions within work zones. Data collection may be done using either the individual vehicle selection method or the all-vehicle sampling method. In the context of this study, the allvehicle sampling method would be a more appropriate method for data collection. Speeds may be collected and recorded using either a direct (radar gun) or indirect (speed trap) measurement technique. Using either the direct or indirect measurement technique, speeds are observed for before and during system operation scenarios. These observations can then be compared to determine the effective reductions in speed due to system implementation.

System Evaluation

74

3. Reduction in mean queue lengths: Queue formation is a common and hazardous condition experienced at several work zones. A small incident can cause queues to build up rapidly, especially on high speed highways and freeways, deteriorating traffic flow considerably in just a few minutes. The real-time advanced warning and traffic control system is expected to reduce the time needed to dissipate queues and bring traffic flow conditions back to normal as soon as possible. Reduction in mean queue lengths is a useful measure to quantify the benefits of using the system for minimizing queue formation via advanced and effective warning, signage, and traffic control. To measure this MOE, a record of the average queue length experienced over a specified period of time needs to be recorded. These values may be averaged to obtain the mean queue length experienced for the time period of the study. The mean queue length should be acquired for both with system implementation and without system implementation scenarios. A comparison of these values will give the reductions in mean queue lengths due to system implementation.

4. Reduction in delays: Delays due to congestion and queuing are a recurrent feature at many highway work zones. One of the objectives of the real-time advanced warning and traffic control system is to reduce such congestion and queuing related delays at work zones via dynamic signage and traffic control. Using before and after measurements of delays, the reduction (or % reduction) in delays at work zones due to system usage may be determined. Significant reduction in delays at work zones indicates that the system is effective in mitigating traffic delays and maintaining a continuous flow of traffic at the work zone. As specified earlier, the delays may be caused by congestion or queuing, resulting in stopped delays and delays due to slow-moving traffic. Both types of delays should be quantified for with and without system implementation scenarios and compared to ensure effectiveness of the system in reducing traffic delays at work zones. Issues such as time period for study and sample size are discussed later in this chapter.

5. Reduction in the number of accidents/severity of accidents: The main objective of deploying the real-time advanced warning and traffic control system is to reduce the

System Evaluation

75

number of accidents experienced at highway work zones. Statistics on work zone-related accidents play a major role in depicting the need for such dynamic/real-time work zone systems. Thus, reduction in the number of accidents/severity of accidents occurring within the work zone is the most important measure of effectiveness to demonstrate the systems effectiveness in fulfilling its objectives and goals, and to justify the deployment of such a complex and expensive system. The quantification of this MOE may take a long period of time. Due to the temporary nature of work zones, maintaining similar conditions for with and without system implementation scenarios may pose a problem. The reduction in the number of accidents can be derived from observations of the number of accidents occurring at highway work zones before and during system implementation scenarios. Until now, traffic exposure data has not been accounted for in statistics depicting work zone accidents. To accurately fathom the work zone accident problem and ensure the effectiveness of the system in reducing work zone accidents, exposure data must be taken into consideration. Exposure data provides for a common base for accident comparison. Thus, to quantify this MOE, accident rates based on vehicle exposure data for with and without system implementation scenarios must be used. A comparison of these accident rates will give the reduction in the accident rate due to system usage. In addition to exposure, there are several other points, such as accounting for accident severity and method of accident type classification, that need to be considered when carrying out an evaluation of a traffic control device in terms of reduction in number of accidents. These are described in detail in the following section, which identifies and examines the various issues to be considered during system evaluation.

6. Speed and advisory compliance: Motorist compliance to posted speed limit signs and advisory signs in work zones is a very important factor in determining the success of the system. A system, however complex, will not succeed in fulfilling its objectives unless and until the users/motorists recognize the system and follow its advisories. Credibility is a key issue in promoting motorist compliance of advisories. Other factors influencing compliance would include location and readability of messages. Motorist compliance to

System Evaluation

76

signs regarding advisory speeds, lane changes, and alternate routing may be verified indirectly by measuring traffic flow parameters or directly through surveys.

5.3

Issues in Evaluation

The evaluation of the real-time advanced warning and traffic control system for its effectiveness is an important part of this project. The evaluation process will assess the performance levels of the system in terms of its effects in mitigating existing work zone problems. It will be based on the functional requirements, evaluation criteria, and measures of effectiveness (MOEs) identified earlier in this chapter and in the previous chapter. Specifically, the evaluation will quantify the identified MOEs for before and after, or before and during, scenarios of system usage. Comparison between before and after observations will be used as an estimate of the system benefits and effectiveness in fulfilling its objectives. The evaluation considers each of the MOEs, explaining in detail how they are to be measured. It also identifies the key constraints that could affect the evaluation process. Figure 13 shows the basic framework for the evaluation and identifies the issues to be considered for each of the MOEs.

The before and after measurement of MOEs is the most common approach used for evaluation of improvements in traffic conditions due to a new or improved system implementation. In this method, the before measurements are treated as a baseline condition and the after or during measurements represent the effect of system implementation. Although some MOEs may be best measured through long-term evaluations, this approach can prove to be susceptible to errors. In such cases, the factors that may influence the before and after measurements include population growth, economic fluctuations, completion of major traffic generators, and other changes. In this case it may also be beneficial to obtain a third scenario where data is collected on the same section of roadway without any work zone in place. When compared to the after or system implemented scenario, this information will provide an estimate as to how effective the system is in returning traffic flow variables to normalcy.

System Evaluation

77

EVALUATION

FUNCTIONAL REQUIREMENTS

EVALUATION CRITERIA

MEASURES OF EFFECTIVENESS (MOEs) Definition of Baseline Conditions Identification of Variable(s) to be Measured Available Methods of Measurement Selection of Measurement Method Key Constraints/Issues

Figure 13. Framework for Evaluation

Sample size requirements need to be worked out to ensure accurate evaluation results. The sample size will be a function of several factors, such as design variables, the desired precision, size of the population being sampled, time constraints, data collection budget, and other limitations. The remainder of this chapter describes each MOE and its measurement in accordance with the basic framework provided in Figure 13. Relevant issues such as those dependent on individual evaluators, will be brought to light as part of the issues to be addressed for the evaluation of the system.

1. Accuracy in measurement of traffic flow variables:

Traffic flow variables are the primary source of input provided to the system to comprehend the status of traffic within the work zone. The accurate measurement of these
System Evaluation
78

traffic flow variables is thus extremely essential for the effective functioning of the system. Verification of the accuracy of the system-measured traffic flow variables is the evaluation of the accuracy of the sensor being employed for the purpose. The traffic flow variables that may be measured include speed, traffic volume/count, and presence/occupancy. Depending on the functional requirements, the system may be required to measure traffic flow variables other than those mentioned above. The required variables to be measured may be verified for their accuracy by comparing them to known values or values calculated by alternative means. For example, a specific number of vehicles traveling at predetermined speeds may be exposed to the sensor for a certain time period. The values of the sensor/system calculated variables should then be compared to actual field values to determine the accuracy of the measured variables.

There are certain issues that need to be resolved before carrying out the evaluation for the accuracy of the variables measured. These include: Accuracy Requirements: The evaluator needs to decide what measurements are to be considered as accurate and what measurements are to be rejected. To accomplish this, the evaluator needs to set a certain percentage error or deviation from actual values. This may be used to decide upon a measure for accuracy. Sample Size Requirements: A decision also needs to be made on the number of observations to be recorded to evaluate the accuracy of the variables measured. In other words, a decision on the part of the evaluator needs to be made regarding sample size requirements. The sample size may be determined using standard available procedures.

2. Reduction in the number of occurrences of congestion and queues:

One of the objectives of the real-time advanced warning and traffic control system is to prevent the occurrences of congestion and queues. In the event of any congestion and queuing, the system is required to bring traffic flow within the work zone back to normalcy as soon as possible. This MOE provides for a means to quantify the benefits of

System Evaluation

79

implementing the system in terms of reduction in the occurrences of congestion and queuing. To quantify this MOE, it is essential to compare before and after system deployment records for occurrences of congestion and queues. The baseline conditions to quantify this MOE will be a work zone scenario employing current work zone signage and strategies for safety and traffic control. For a predetermined period of time, the work zone should be observed and a record maintained for each occurrence of congestion and queue. For the after implementation scenario, occurrences of congestion or queuing is recorded with the system in operation. For both scenarios similar conditions are to be maintained and congestion and queuing must be identified using similar criteria. It is evident that the main traffic variable that needs to be measured for the quantification of this MOE is the mean speed of traffic. There are several methods of calculating mean speeds of traffic for a particular section. Any one of the available methods may be used for this MOE. Several readings of speeds may be taken and averaged to find the mean speed of traffic. This may be done at different locations within the work zone. For the system implementation scenario, speeds may be measured at different locations within the work zone using the traffic sensor that the system employs. A comparison between the number of occurrences of congestion and queuing with and without system operation may be used to quantify the benefits of system deployment. The MOE may be expressed as a percentage reduction in the number of occurrences of congestion and queuing.

There are several issues that need to be considered for the measurement of this MOE. These include: Threshold Values of Speed: This MOE is based on the definition of congestion and queuing. An event may be pronounced as a state of congestion if the speed of traffic within the work zone falls below a certain threshold value. Alternatively, if there is stopped traffic building up due to excessive congestion, it may be considered a queue. The evaluator must decide what threshold speeds are to be used to define a condition of congestion or queuing.

System Evaluation

80

Sample Size Requirements: The sample size requirements need to be worked out for the accurate evaluation of the system. The evaluator needs to decide on the desired level of accuracy to compute sample size requirements.

3. Reduction in speeds:

As discussed earlier, a large percentage of work zone accidents are attributed to excessive speeding within the work zone. Hence, the system when deployed, should address this problem of excessive speeding within work zones. This MOE is designed to quantify the benefits of system implementation in terms of reductions in traffic speeds within the work zone. The baseline conditions for the measurement of this MOE would include a typical work zone layout without system implementation. The existing methods of work zone safety and control may be used during measurement of speeds for the before system implementation scenario. Mean speed of traffic may be used as the variable of measurement. The readings may be taken at critical locations near the work area. Similar conditions must be applied after system implementation. For the system implementation scenario speeds may be measured using the traffic sensor employed by the system for the measurement of traffic flow variables. The reduction in speeds may be calculated by comparing the values obtained for the before and after system implementation cases. The reductions in speeds can be expressed as a percentage to give a perspective of the level of speed reduction due to system implementation.

The issues involved in the measurement of speeds within work zones include the following: Type of Data Collection Method: There are two types of data collection methods: the individual vehicle selection method and the all-vehicle sampling method. The individual vehicle selection method uses small sample sizes and can be done over a relatively short period of time. This method is typically used for measuring the effectiveness of a traffic control device, checking the effect of speed enforcement at a particular location, or establishing the location for a traffic sign. If this method is used for the evaluation

System Evaluation

81

of the real-time advanced warning and traffic control system, speed data of individual vehicles must collected. The data collection method should involve a random vehicle selection method to obtain unbiased results. The all-vehicle sampling method is generally used when the purpose of the study requires or can accommodate the measurement of spot speeds of all vehicles passing a certain point for a sample of time periods. Applications of this method include monitoring speed trends, assessing highway safety, or establishing speed limits (Robertson et al., 1994). In the context of this study, the all-vehicle sampling method may be used to collect speeds of all vehicles at a certain location during a number of time periods, such as morning peak or evening peak hours. The evaluator should make a decision regarding which approach is to be adopted. For the evaluation of the real-time advanced warning and traffic control system, the all-vehicle sampling method would be the more appropriate data collection technique. The data collection may be done at critical spots near the work zone. Location of Speed Measurement: Critical areas where speed measurements are important need to be identified. Past studies show that the taper, buffer, and advance zones of a work area are accident prone areas within a work zone. This combined with the fact that a high percentage of work zone accidents are caused due to excessive speeding may lead us to believe that the advance, taper, and buffer zones are potential locations within a work area where excessive speeding cases are observed. Thus, speed measurements made for each of these zones will be helpful in assessing the meaningful reduction in speeds due to system implementation. Time period: An important issue that needs to be addressed for speed studies is the time period for the data collected to be considered as representative of the desired study conditions. A decision on what time period to consider for data collection must be made by the evaluator.

System Evaluation

82

4. Reduction in Mean Queue Lengths:

A queue is a hazardous phenomenon and can lead to congestion, delays, and accidents. The system must be designed to reduce the occurrence of queues and facilitate their quick dissipation. This MOE is a means of quantifying the systems capabilities of reducing queue lengths. For the baseline conditions, queue lengths must be measured without the system in place. This may be done either manually, by having a count of the number of stopped vehicles or slow-moving vehicles for each occurrence of queues or by employing a queue detector. The product of the count of vehicles and average vehicle lengths will give the total queue length. The average queue length may then be calculated for a number of queue occurrences. For the system in place scenario, the queue lengths may be calculated by employing system capabilities of queue and queue length detection or through other means. The average queue length may then be calculated. A comparison between the before and after system implementation cases (may be expressed as a percentage) will give an estimate of the reductions in mean queue lengths.

The main issues of concern and key constraints for determination of mean queue lengths include: Threshold Speed Values: As mentioned earlier, the system detects queues based on threshold speed values defined within a queue detection algorithm. The desired threshold value for speed must be defined as per the judgment of the evaluator. A factor that may affect the threshold speed decision is the type of roadway facility and the possibility of quick build-up of queues. Based on experience, the evaluator may decide upon a suitable threshold speed value to define a queue. As an example, the algorithm may consider traffic speeds below 10 mph as queues, and consequently anything between 10 mph and the free-flow speed will be considered congestion. Time Period: The mean queue lengths should be measured for a certain time period. This time period has to be identical for both before and after system implementation scenarios. The time period for which the system will be evaluated for mean queue length must be determined in advance by the evaluator.

System Evaluation

83

Detection Zone: An unresolved issue and concern for automatic queue detection is what happens if the queue exceeds the detection zone? A decision should be made regarding the location of the last sensor upstream of the work zone. When making this decision, factors such as history of queuing and maximum anticipated queue length should be considered.

5. Reduction in Delays:

Billions of dollars are lost annually due the great number of delays experienced by motorists. Work zone delays may constitute a significant percentage of those lost dollars. The real-time advanced warning and traffic control system is expected to reduce delays caused by congestion and queuing at work zones, thus reducing the amount of resources lost due to excessive delays. The delays expected at a work zone can be caused either due to stopped traffic or slow moving traffic. The delays caused by stopped traffic can be attributed to the formation of queues, and those due to slow moving traffic to congestion at work zones. The system should work effectively to reduce delays caused by both congestion and queues. The reduction in delays due to congestion and queuing at the work zone can be measured by comparing the delays observed for two scenarios, namely with and without system implementation. The baseline conditions for the measurement of delays should be the regular work zone layout as per the MUTCD. Currently employed safety and control devices may be used for work zone safety and traffic control. For the before scenario, delays experienced should be computed without the implementation of the system. The computation of delays may be done using standard available methods and procedures. For the after scenario the system capabilities of delay computation may be used. In order to obtain consistent and meaningful results, the delay measurement for the before and after system implementation scenarios should be made for the same time period, at the same work zone and work zone configuration, and during similar traffic conditions.

System Evaluation

84

The key constraints or issues that require attention during the evaluation of the system for its effectiveness in reducing delays include: Time of Study: For effective results, delays are to be measured during peak hours in the direction of the heaviest flow of traffic. Large benefits may also be obtained if system can effectively reduce unexpected non-peak hour delays. The evaluator should make sure that similar conditions are maintained for both before and after system implementation scenarios. Sample Size Requirements: The sample size requirements need to be worked out for the accurate evaluation of the system. The evaluator needs to decide on the desired level of accuracy to compute sample size requirements. The Manual of Transportation Engineering Studies (Robertson et al., 1994) provides guidelines on the minimum sample size requirements for a 95% confidence interval for delay computations. The evaluator may refer to these values to reach a decision on the sample size requirements for the study.

6. Reduction in the Number of Accidents/Severity of Accidents: The increasing trend of accidents and fatalities at highway work zones warrant the investigation of real-time systems for improved safety and traffic control. Reductions in the number of accidents/severity of accidents after deployment of the system will be an important and key measure to evaluate the system for its effectiveness. Measuring the reduction in the number of accidents can be done by simply keeping a log of the number of accidents for both before and after system implementation scenarios at the work zone, but there are several difficulties in procuring this information. Some of the difficulties encountered include having a work zone which is operational long enough to get readings for both cases. In the event of non-availability of such long-term work zones, ones that have similar geometric characteristics and similar traffic characteristics may be used. To accurately determine the reduction in the number of accidents due to system usage, unbiased accident data should be available for before and after system implementation scenarios for the same work zone. Once this is achieved, a comparison between the accidents will give us the reduction in the number of accidents/severity of accidents. The

System Evaluation

85

accident data should be recorded in a detailed format to check for reductions in particular types of accidents such as rear-end collisions, which form a major percentage of work zone-related accidents.

The major issues that need to be considered when performing the evaluation of the system in meeting one of its main objectives, namely accident reduction, are: Choosing a Time Frame: The Manual of Transportation Engineering suggests that a period of three years is the most common choice for a detailed accident analysis. Three years represents a compromise between the desire to have large samples and a desire to have a time frame during which traffic conditions are unlikely to change significantly. Non-availability of such long-term work zones may constrain the time frames to be smaller, thereby affecting the scope of the study due to smaller sample size. Data Collection Technique: Traditionally, accident data recorded the number of accidents occurring during a certain time period. This gives us a very inaccurate picture of the actual situation because the number of accidents occurring within a work zone also depends on how much traffic was exposed to the work zone and for how long. For example, the number of accident occurring at a work zone that is operational for two months cannot be compared to those at a work zone operational for a week. It is essential to consider several other factors such as length of the work zone, number of vehicles exposed to the work zone, and time frame to set up a common base for comparison of accident rates. This may be done by considering traffic exposure data. Thus, it is extremely important to consider exposure data and represent accident rates in terms of vehicle miles of travel (VMT) when evaluating the system. The accident rates for a section of highway can be computed using the following formula, which accounts for exposure by representing accident rates in terms of 100 million VMT:

R=

100 ,000 ,000 * A 365 * T * V * L

System Evaluation

86

where

R = accident rate for a section (per 100 million VMT) A = number of reported accidents T = time frame of analysis in years V = average annual daily traffic (AADT) volume for the section L = length of the section in miles.

Analyzing Accident Data: Accident data should be recorded in detail, giving information on accident type and location with respect to work zone, and also providing information on probable cause for the accident. This enables the evaluator to evaluate system effectiveness for decreases in certain types and causes of accidents specific to work zones, such as rear-end collisions, fixed-object off road collisions, sideswipe, driver inattention, exceeding safe speed, and following too close. The location of the accident within the work zone also gives the evaluator insight into the number of accidents occurring at hazardous locations, such as at the taper zone and at the buffer area.

Accident Severity: Accident rates may be adjusted to reflect the greater costs of injury and fatal accidents. A common method of taking accident severity into account is to compute equivalent property damage only (PDO) accidents. The number of PDO accidents that are equivalent to a fatal or an injury accident is used to form an equivalent factor to fatal accidents. This factor is used to account for PDO-type accidents into accident rates. For example, Kentucky uses a factor of 9.5 PDO accidents per fatal or A-injury accident, and 3.5 PDO accidents per B-injury or Cinjury accident (Zeeger, 1982).

7. Incident Detection Rate: The incident detection rate is defined as the percentage of detected incidents among the total number of incidents. The basic variables that need to be measured here are the number of incidents detected by the system and the total number of incidents for the entire period of the evaluation.

System Evaluation

87

There are some issues that need to be considered when evaluating the system in terms of its efficiency for incident detection. Firstly, an incident should be clearly defined by the evaluator. A common definition of an incident is any event that reduces capacity, hindering smooth flow of traffic. This may range from a vehicle changing a flat tire on the shoulder or requiring to be towed, to an accident involving one or more vehicles. Secondly, a suitable time frame to carry out the evaluation of the system for incident detection capabilities needs to be worked out by the evaluator. The evaluation may be carried out at the same time as the evaluation of accident reductions, which would provide a large sample size and include accidents as part of incidents. It is also highly desirable that the system have a very high incident detection rate. The incident detection rate should be as close to 100% as possible.

8. Mean Detection Time: The mean detection time may be defined as the mean time interval between the start of an incident and its detection. To evaluate the effectiveness of the system for the mean detection time, the actual start times of every incident that occurs within and around the work zone should be recorded. These observations will be compared with the time the system records the detection of the incident. The time interval between these observations will provide the detection time for every incident. These values may be averaged to obtain the mean detection time.

There are some key issues that need to be resolved for the evaluation of the system for mean incident detection time. Firstly, a suitable time frame for the computation of mean detection time needs to be decided by the evaluator. Evidently, results obtained with larger sample sizes will be more reliable and acceptable. Thus, it is desirable that the evaluation take place along with the evaluation for reduction in the number of accidents, as it requires the longest time frame for evaluation. It is also desirable that the mean detection time of the system be as low and as close to zero as possible.

System Evaluation

88

9. False Alarm Rate: False alarm rate is defined as the ratio between the number of false alarms for congestion, queuing, or incidents and the total number of alarms for each. The variables to be observed for this case include the total number of alarms recorded by the system and the number of false alarms that were observed. The alarms may be for either congestion, queuing, or incidents.

The major issue of concern regarding false alarm rate is choosing an appropriate time frame for the evaluation. The time frame must be determined in advance by the evaluator. It would be logical to carry out the evaluation for the longest time frame possible. As in the case of incident detection rate, the false alarm rate may also be evaluated during the evaluation period for accident reductions. It is also highly desirable that the system have a low false alarm rate with the ratio of false alarms to total alarms being as close to zero as possible.

10. Speed and Advisory Compliance: Motorist compliance to posted advisory and control signs is an indicator of system effectiveness in reducing work zone-related problems. This study involves several human factors issues, but may be approximated by obtaining indirect measures of motorist compliance. The available methods of carrying out compliance studies include interviews, videotaping, simulation, and carrying out tests with subjects on the road and observing their reactions. Since these are independent studies, for the real-time advanced warning and traffic control system, motorist compliance will be recorded via indirect observations. This can be done by indirectly recording motorists reaction in terms of traffic speeds and maneuvers when they encounter a particular control or advisory sign. The signs displayed by the system requiring motorist compliance may include speed limit signs, advisory speed signs, change lane signs, and alternate routing signs. The compliance to speed limit and speed advisory signs can be measured as a function of the traffic speeds within the work zone where the signs are posted. Compliance may also be a function of the reduction in speeds when a particular advisory speed or speed limit sign is encountered. These speeds may be measured using the traffic sensors employed by the system, or through other means. Compliance to control signs

System Evaluation

89

such as change lane signs may be measured as a function of the number of vehicles occupying the lanes or the number of vehicles changing lanes after encountering the change lanes sign. This can be measured by having a sensor count traffic on the particular lane of interest, or by having a video surveillance camera that records the behavior of motorists when the sign is encountered. Compliance to alternate routing signs can be measured as a function of traffic volumes diverted from the highway to arterials. Traffic sensors can be used to measure the decrease in traffic volumes on the highway, and the corresponding increase in traffic volumes on the arterials. The time and conditions under which the study is carried out can affect the results obtained. Most compliance studies are carried out in good weather and under normal traffic conditions. Samples for the study should be taken for all relevant periods of the day, such as morning and evening peak hours and non-peak hours. Due consideration should also be given to sample size requirements for meaningful results of the evaluation.

System Evaluation

90