Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

Analyzing the Effectiveness of Web Application Firewalls

A benchmark study of eight web application firewalls and intrusion prevention systems

by Larry Suto Application Security Consultant San Francisco November 2011

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

Table Of Contents
Analyzing the Effectiveness of Web Application Firewalls Table Of Contents 1. Executive Summary 2. Key Findings Overall Findings Findings: Baseline Tuned Findings: DAST/Scanner Tuned 3. Evaluation Criteria & Results Evaluation Notes Barracuda Networks: Barracuda 660 Citrix: NetScaler NS9.3 Build 48.6.nc DenyAll: rWeb 4.0 - Core manager v0.3.0.220 F5: ASM 3900 Imperva: SecureSphere 8.0 Trustwave: ModSecurity 2.6.2 SourceFire: SourceFire 3D Sensor 2100 - 4.9.1.6 build 150 Unnamed IPS 4. Conclusions 5. Challenges False Positive Risk WAF Evasion 6. Background 7. Methodology Overview Constraints and Assumptions Targets of Attack (TOA) Testing Tools Experimental Tests Setup TEST BL Baseline Test Vulnerabilities Included Configuration Notes 8. Future Directions 9. References 10. Bibliography 11. Acknowledgements Appendix A: Definition of Terms and Acronyms Appendix B: Test sites and vulnerability data

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

1. Executive Summary
This study evaluated the effectiveness of Web Application Firewalls (WAFs) and Intrusion Prevention System (IPS) devices at protecting web applications against external attacks in comparison to each other. In addition, this study provides general guidelines about the ease of use, and factors affecting the effectiveness of web application protection solutions. The study examined the following cross-section of modern WAFs and IPSs both proprietary and open source. (in alphabetical order): WAFs: Barracuda 660, Citrix NetScaler, DenyAll rWeb, F5 ASM, Imperva SecureSphere, ModSecurity IPSs: Sourcefire 3D, Unnamed IPS Each of the eight systems was evaluated in two separate tests. Each of the tests leveraged the devices in different configurations. The tests were as follows: 1. Baseline Tuned: Tuned configuration requiring 1 day or less of an experienced experts time 2. DAST : Trained solutions by application generated filters from Dynamic Application Security Testing (DAST) software products. DAST tools perform automated web application vulnerability scans. Some of the DAST solutions now generate filters which can be imported into WAF and IPS protection policies.The NTODefend DAST tuning solution was used in this study to generate filters for five of the eight tested platforms. The other three platforms are not yet supported the DAST tool used in this study. There are other DAST vendors that have WAF integration such as Cenzic and Whitehat Security but they were not used due to time and logistical considerations. Future testing should compare these solutions as well. The following graph (Figure 1) shows a high-level summary of the test results - the % vulnerabilities found with baseline tuning versus the % vulnerabilities found when tuned with DAST generated filters.

Figure 1

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

The following table shows the same results of both tests in a table format and includes both the actual numeric results as well as the percentages.

Figure 2

2. Key Findings
Overall Findings
The most significant conclusions of the study are as follows: To maximize effectiveness, WAF solutions must be tuned by a trained professional. The study found that basic tuning takes an average 3.5 hours by a informed user or expert. (see Figure 6) For the solutions supported, there was an average improvement of 36% more vulnerabilities blocked when the DAST generated filters were applied. (see Figure 3 below) IPS solutions are not designed for and are not very effective at blocking web application vulnerabilities with their default settings. However; when tuned by DAST tool generated filters, IPS solutions can be as or more effective than several of the WAFs. (see Figure 3 below)

NTODefend does not yet support for Barracuda, Citrix, or F5

Figure 3

Additional general conclusions from this study are as follows: A web application firewall is a recommended element of the security defense strategy. WAF and IPS solutions are both capable of providing effective protection, but neither solution is a sufficient application defense strategy in isolation. They need to be accompanied by solutions & processes such as DAST and Static Analysis Security Testing (SAST) tools, risk management etc. Organizations should understand and test the differences of their solutions detection and blocking effectiveness before and after training with DAST generated filters. Rule creation requires detailed knowledge of both the application and the WAF or IPS solution. It is not practical for a general user to produce effective filters in a reasonable amount of time. Using an automated solution made the rules generation part of the experiment manageable, otherwise any manual attempt at generation of such rules would require a deep understanding of regular expressions, several target systems such as SQL databases, and a deep consistency across all rules to ensure that they do not violate each other, etc. Training with a DAST tool as only as effective as the tools output. Care should be taken that the tool has a quality ruleset and that false positives have been remediated.

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

DAST tuning solutions can significantly improve the effectiveness of WAFs if the WAFs allow the DAST to create new customer rules (e.g. Sourcefire and ModSecurity). Solutions that only allow DAST solutions to turn on existing proprietary WAF rules (e.g. Imperva) do not benefit from such improvement. The improvement of the WAFs and IPS solutions is, however, dependent on the quality of the DAST tuning system.

Findings: Baseline Tuned

From the first test, it became clear that considerable vendor-specific knowledge and understanding is required to properly configure and tune the WAF and IPS solutions. The key findings from the first test where the solutions were tuned by an experienced expert were as follows: The WAFs are fairly effective at detecting and defending web attacks on their own; the most effective solution found 88% of the vulnerabilities known in the test applications while the average effectiveness across all WAF solutions was 62%. The IPS solutions are not designed to protect web applications and were not very effective at defending them during this test.

Findings: DAST/Scanner Tuned

Test two yielded the following results when WAF and IPS solutions were trained by DAST generated filters. (See Figure 3 below) The IPS solutions improved by an average of 60% bringing up their performance at-par or better than the trained/configured WAFs; with their overall blocking effectiveness averaging 82%. In this study, the supported WAFs that were tuned with DAST solution improved an average of 19% from their baseline tuned state.

3. Evaluation Criteria & Results

Throughout the study, results and information were captured according to the studys methodology. Three criteria were defined by which to score the solutions, then the weighting and specific scoring guidelines were defined for each criteria. The following table details the evaluation criteria and their weightings:

Figure 4

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

The results closely matched the results from the percentage blocked scores

Figure 5

The following table details the summary of the results from both tests:

Figure 6

Evaluation Notes
List of the defensive products used and experiences using them for the testing.

Barracuda Networks: Barracuda 660

Pros: The WAF is fairly full featured and has a responsive and well laid out web GUI. Basic signatures are easy to enable. It offers above average protection against attacks. With some work, advanced signatures can be added. Cons: Advanced tuning and signature enablement is a bit counterintuitive and not easy to enable or execute. The signature strategy seems to be focused mostly on regular expressions which sometimes are difficult to get right. The default signatures/basic tuning processes were fairly effective but the strict mode did not seem to improve blocking accuracy. Learning mode is not recommended out of the box because of performance issues. Support: Barracudas support is very responsive and adequately addressed all operational issues. They assisted in turning on all the default necessary signatures and demonstrated how to enable and apply the stricter signatures as well.

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

Citrix: NetScaler NS9.3 Build 48.6.nc

Pros: The evaluation version of the Netscaler platinum is full featured and includes the web application firewall feature. The setup was straightforward. Accuracy of blocking was decent in the tests. Cons: Management interface could be a little more intuitive. Policy configuration and management could be better. Support: Relatively little support was required to perform the setup.

DenyAll: rWeb 4.0 - Core manager v0.3.0.220

Pros: The web interface is easy to use, and the default blacklist is easy to configure. Cons: While the interface does allow for custom rules, it does not have a method to optimally import a custom blacklist such as from NTODefend. Support: DenyAll support was very helpful in providing instructions to install a custom blacklist using the command line interface.

F5: ASM 3900

Pros: Excellent appliance quality. Great intuitive and snappy web based interface. Once basic configuration is accomplished, there are a lot of options for tuning the protection policies. With basic tuning F5 performed very well in the tests. The F5 sits on a very advance load balancing platform and provides other advanced security features such as DDOS protection. It is supposed to do well against HTTP resource exhaustion attacks. Cons: For users not familiar with the F5 product line, the setup can be quite challenging. Support: F5 was very helpful and gave a comprehensive walk-through of the tool including the basic setup and policy configuration.

Imperva: SecureSphere 8.0

Pros: Imperva has a high quality protection engine with a very robust set of basic policies. It is also one of the easier WAFs to configure and provides the most value straight out of the box. Cons: The Imperva UI while providing a lot of information is a bit difficult to navigate. Modifying existing policies to handle exceptions can be unwieldy. Creating custom policies can be difficult. System can be be chatty with false positives during learning. Support: Imperva was very helpful in assisting with setting up the appliance and ensuring that everything was properly configured.The recommended implementation is to allow the appliance to collect traffic to develop and application profile before stricter rules are applied. Thus there is a bit of a grey area between what a default policy and a stricter applied policy might be. Nevertheless, after turning on a reasonable amount of the policies, it became clear that it was a very effective solution.

Trustwave: ModSecurity 2.6.2

Pros: Free open sourced software makes it lowest acquisition cost. Easy to integrate with Apache. An expert user can modify filters to accomplish nearly any task Cons: Limited only to Apache users, and is installed on the application server, thereby limiting scalability options. There is no user interface. Requires command-line activity and numerous config files. Support: Did not use anything other than the projects forums and mailing lists. Live support is purchased separately from 3rd party consulting companies.

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

SourceFire: SourceFire 3D Sensor 2100 - 4.9.1.6 build 150

Pros: The web interface is very full featured, and importing/enabling custom rules is easy to accomplish. SourceFire supports the SourceFire/NTODefend solution as a WAF for PCI section 6.6 compliance. Cons: Not useful on its own to function as a WAF. Had some troubles with clearing rules and reloading them, but their support team was able to clear up the problem quickly. Support: Sourcefire was very helpful when needed. There is also a very active community of support around the snort project which includes volunteers as well as Sourcefire paid staff.

Unnamed IPS
This IPS vendor did not give authorization to be named in this study. The IPS solution supports Snort 2.8 rules, which was used for this testing. Pros: The UI was very compact, appealing, and visually oriented. Cons: Not useful on its own to function as a WAF. Support: Their support team was very helpful in overcoming the few configuration difficulties encountered.

4. Conclusions
WAFs are a controversial technology. While some security professionals argue that they are easily avoided and the only remediation to web application vulnerabilities is to fix the code [7], others argue the opposite [6]. Manual code reviews and code fixes are essential, however they take considerable amount of time: days, weeks or even months once a vulnerability becomes known to the developers. In case where users do not have access to source code for any number of reasons, (3rd party software, legacy software, external service etc.) there is no manual recourse for fixing bugs. For some organizations, contractual issues and elaborate change control processes means that known vulnerable applications can and do long periods without being remediated. This problem, where for a period of time a vulnerability is known, but not fixed at the source, is addressed by "well-trained" defensive tools like WAFs and IPSs where a vulnerability can be effectively patched while the code is being repaired. This study clearly demonstrated that using WAFs/IPS devices in default or minimal mode to protect web applications are often not useful, and WAFs configured with the recommended policies and tuned can often be bypassed by a moderately skilled hacker or an automated hacking tool. This has been well documented (WAF Evasion section in this document). What is interesting is that training these tools with a DAST filter generator can dramatically increase their effectiveness and make them a far more useful part of an enterprise's application security strategy. Additionally, trained IPS devices can be effective at protecting Web Applications. And finally are we staring at the abyss of a flawed model at detecting attacks? If WAFs could be made more intelligent somehow the future is bright for this technology.

5. Challenges
The WAF vendors have spent a significant amount of time making claims of one sort or another as to why their technology is better, and this data may not speak to all of the various claims and may not be fair to some of the individual strengths of each technology. However, it is the fairest and most impartial model that the study could come up with to test these technologies in a vendor-neutral way. It is also important to note that while the tests are complete as far as they go, that this test is not an exhaustive study and only begins to scratch the surface of the problem. In particular, WAF evasion research was completely left out, due to time constraints and to keep the scope of the study reasonable. Even with this limited level of testing, several conclusions can be made without doubt. For instance, it is almost certain that even with a reasonable amount of learning and tuning most WAFs will miss well-know and well understood vulnerabilities. Suffice it to say that the problem escalates polynomially once we start throwing in complications due to AJAX/JSON type sites or various Java, ActiveX and Flash plugins.

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

False Positive Risk

Although there was not time to complete a comprehensive study of false positives (the blocking of good traffic) A limited amount of effort was used to try an eliminated false positives, but in future tests this area should have more attention. The primary focus in this study was blocking of known vulnerabilities.

WAF Evasion
After a cursory review it can be discovered that there are numerous examples in the field where it becomes obvious that the status-quo solutions to run-time attack detection and containment technology are routinely by-passed. At the time of the writing of this paper, the following examples were retrieved from the Web: Example 1: ModSecurity SQL Injection challenge held by Trustwave's SpiderLabs. In this challenge the ModSecurity WAF was tuned and set up to protect a set of four of the scanner vendor vulnerable websites. Coincidentally, this was a subset of the flawed websites used in the current study. So lets take a look at what happened. In one of the successful bypasses of the ModSecurity inbound CRS SQL Injection rules, an individual by the name of Johannes Dahse used a combination of MySQL comments and new line characters to evade the CRS SQL Injection filters. The problem was not in the actual filter itself, which used regular expressions by the way, but in the transformation function -t:replaceComments (Modsecurity transformation functions are a pipeline of normalization functions that modify the input so that the rule can detect the attack). In this case -treplaceComments altered the input (by ignoring standalone termination */ and simultaneously replacing unterminated /* comments with a space( 0x20)) so that the filter could not recognize it as an attack. In the end it was not possible to create a rule to remove SQL comments reliably so another rule with a complex regular expression was created instead to detect the presence of SQL comments themselves. Example 2: The second example comes of failed context-free matching techniques comes to us from a paper from the recent Defcon 19. The talk discussed the paper titled "Bypassing PHPIDS 0.6.5." One of the attacks presented bypassed "ALL" the PHPIDS rules as of 0.6.5. How could of this happened you ask? The flaw was the incorrect assumption that "attacks can never be repetitive". The heart of the problem was the following bad expression: Line 90 in ./lib/IDS/Converter.php (?:(.{2, }) \1{32,}) The regular expression matches any two characters which are repeated 33 or more times.The first match is a back reference created with the \1 and then 32 additional matches are required to trigger the regex. Even though the attack string <script>alert(1)</script> is matched by 4 filter sets, if it is repeated 33 times it will bypass PHPIDS entirely. Example 3: JSReg issues has had issues as well. The details are here "http://code.google.com/ p/jsreg/wiki/Exploits", Not notice that one of them is a regular expression rewrite error: "the failure to strip the single line comment which then fools the regex rule into thinking that the code is a regex object and not function calls".

The commonality between these examples is that all them are using pattern matchingin particular regular expression type of context free matchingwith a little bit of AI and boundary restriction of parameter values thrown in to detect attacks. We have already seen that pattern matching does not yield optimum results elsewhere in the industry, I refer to Anti-virus applications of signature detection, where the upper bound of successful detection seems unable to cross the 60% mark. I think that a defense

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

mechanism that can only protect against 70-80% of the attacks is like a house thats missing a wall. We have a long way to go, and while the state-of-art against attackers is not where we need it to be, I think that next section discusses some future directions that have promise.

6. Background
When comparing web application security solutions amongst each other for the purpose of assessing which one to choose and why consumers are faced with several difficulties. These difficulties arise from the inability to compare apples to apples when it comes to comparing Web Application Firewalls (WAFs), network Intrusion Prevention Systems (IPSs), and similar defense technologies that purport to safeguard against Web attacks [1]. Each Web application firewall vendor delivers different strengths and weaknesses, and presents varying philosophies of how Web attacks should be defended against [2]. Security certification standards such as Common Criteria, PCI DSS 6.6, DIACAP etc. focus on the theoretical for the most part, and do not provide a reasonable guideline for comparison, despite their scorecard nature [1, 2, 3]. Security pundits are at odds with each other on whether the problem can be solved using fully automatic techniques such as WAFs and IPSs or is it only solvable using partial automated tools such as staticanalysis tools in conjunction with manual code review and remediation strategies. There is a list of dos and donts; with advice like DO consider WAF for performance monitoring; DONT think its set and forget [1]. Non-profit consortia provide an invaluable resource to the consumer, however they are very sensitive to vendor-neutrality, and their core approach of staying that way is to keep out of the comparative-studies field all together. It remains difficult to find standards that provide direct comparative quantitative measures, and especially difficult to find ones that can explain it to average users of this technology. So, as things stand, it falls on to the consumer to fund an internal study on some standards basis. The most promising guide for end-users seems to conduct such a study seems to be The Web Application Firewall Evaluation Criteria (WAFEC) and its corresponding WAFEC Evaluation Response Matrix. However, WAFEC only provides only qualitative comparative analysis between potential candidate solutions. They still require extensive knowledge of each aspect of web technology and this author found that Evaluation Response Matrix has no mention of relative importance of each criterion, nor any method of how to measure evaluate over-all effectiveness of the targets of evaluation [5,6]. Previous research on WAFs have been confined to vendor organizations and customer evaluation engagements. There have been some publications discussing the functionality of WAFs but most have been limited to discussions on defining WAF functionality and providing advice on what keep in mind when conducting private evaluations and deployments. See the appendix for some references. To the best of the authors knowledge this is the first study to conduct a general purpose evaluation of WAF and IPS systems ability to protect web applications using a uniform testbed with a simple and easily reproducible methodology. An interesting thing to keep in mind is that in the literature a WAF is referred to as security mechanism with an intimate understanding of the nature of HTTP traffic and web application vulnerabilities. The consensus is that a WAF is an HTTP focused implementation of general purpose IPS technology. This study is also one of the first to provide clear empirical evidence that without specialized configuration and tuning, a general purpose IPS provides limited protection against web application attacks.

10

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

7. Methodology
Overview
The study tested each solution against the same set of websites and web application prototypes to ensure that the experiments were instantiated against well-known and well understood vulnerabilities. This provided a uniform baseline for comparison. This experimental study provides a quantitative comparison between various web application security solutions that include both Web Application Firewalls and Intrusion Prevention Systems and evaluates their relative effectiveness in detecting, reporting and thwarting web attacks. The purpose of this study is to create a foundation for such comparative studies.

Constraints and Assumptions

The experiments in the study were designed with following constraints and assumptions: A typical end-user (hereinafter user) wants to protect a vulnerable web application or website, hereinafter called the security target. The user can detect vulnerabilities within the security target, using an automated tool. The user, while knowledgeable about the Web application security principles, is not familiar with any vendor specific details for the products being evaluated. The user does not fully understand the underlying technological details such-as deep understanding of HTTP protocol negotiations. The user only has limited time and budget to perform these evaluations. This study is not an exhaustive dissertation of various advanced capabilities of the evaluated web application firewalls and intrusion prevention systems.

Targets of Attack (TOA)

The study is a comparative analysis of primary results from a set of controlled tests conducted on a collection of vulnerable site. The vulnerable site used is called Target of Attack (TOA). The following TOAs were used in these experiments: Acunetix AspNet - http://testaspnet.vulnweb.com Acunetix TestPHP - http://testphp.vulnweb.com Cenzic Crackme - http://crackme.cenzic.com HP zerowebappsecurity - http://zero.webappsecurity.com IBM Testfire - http://demo.testfire.net NTO Web scan test - http://www.webscantest.com

These sites demonstrate manifestations of well known and well understood vulnerabilities.

Testing Tools
The study required use of at least one DAST product capable of creating filters for as many as possible of the above listed defensive products. The primary DAST products (vulnerability scanners) used were NT OBJECTives NTOSpider and Mavitunas Netsparker. NT OBJECTives NTODefend product was used for filter generation. There are other DAST vendors that have WAF integration such as Cenzic and Whitehat Security, and stand alone applications such as ThreadFix (previously called Vulnerability Manager) from Denim Group.

11

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

NT OBJECTives NTODefend product includes a QuickScan feature which tests for false positives (the blocking of good traffic) by sending both legitimate and attack payloads to confirm that only attack traffic is being blocked this was only used on two sites as a sanity check but expanding false positive would be important in future tests. The other tools and features were not used due to time and logistical considerations. Future testing should compare these solutions as well.

Experimental Tests Setup

The evaluation tests used the DAST scanners independently, as shown in Figure 1 to detect exploitable vulnerabilities in the TOAs. The study is comprised of the following tests:

TEST BL Baseline Test

A security scan the security targets of vulnerable systems for bugs using two These Scanners are the used for subsequent tests for each of the Evaluation Target WAF or IPS.

Figure 7

Figure 1 (above) depicts the WAF setup as it relates to the scanner and protected application location for Tests I and II. TEST I Scanning the security targets with a WAF/IPS in front of the vulnerable applications. In this test the WAF/IPS is configured to protect the application with approximately of one half day of effort. Vendor requested assistance is provided as necessary to ensure that the recommended protection policies are in place. This is the Tuned/Trained mode. TEST II Scanning the security targets with a WAF in front of the vulnerable applications. In this test the WAF is configured to protect the application with a minimum of one half day effort. Vendor requested assistance is provided as necessary to ensure that the recommended protection policies are in place. A third-party rule generator is then used to import policies in order to improve the blocking capability of the system. This is the DAST or Scanner Trained mode.

Vulnerabilities Included
Each WAF/IPS is different in its specific methodology for finding and preventing attacks. For the purposes of this report, the types of vulnerabilities that were counted were those that are useful against custom applications, and which most users care about. These are: SQL Injection / Blind SQL Injection Cross Site Scripting / Persistent Cross Site Scripting Command Injection CSRF / HTTP Response Splitting Arbitrary File Upload attacks Remote File Include (PHP Code Injection)

12

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

The vulnerabilities were recorded to cross reference which WAF's blocked which vulnerabilities in a simple format to track and compare the results. A complete listing of vulnerabilities blocked by each WAF was recorded to track the effectiveness of each technology. This resulted in a fairly high degree of confidence that the extent of Blocked and Missed numbers are complete.

Configuration Notes
For tuning and training the basic policy was enabled plus any settings that were recommended by the vendor in order to optimally configure the system in a reasonably effective protection mode. For many of the systems, I received direct assistance from the vendor in order to accomplish this. To then see how well customized training from an outside source would improve the protection, custom filters/rules were generated using the DAST tuning solution for each supported WAF and loaded them in and repeated the test. Again, this was done mostly when it was obvious that the basic tuning process of the WAF was going to involve a significant effort. The assumption was that protection would improve with training, and this was clearly proven during testing. The collected data is being made freely available for anyone to review and re-create. Additionally, it should be understood that more manual configuration effort could likely be used to improve the results of the WAF's further, but time constraints are always going to be an issue in real world use, as well as when compiling a study like this.

8. Future Directions
So what are the future directions? Surely we cannot do manual defense. Even today, there is a strong debate on whether or not programmers can be taught to create flawlessly secure programs [6]. As we have seen, automated defense systems that perform strictly signature or pattern based detection only achieve partial success at best, and will probably reach an upper bound that is well below the required threshold of risk mitigation. So what does it all mean? Is run-time automated defense doomed to failure? I dont think so. There are entities and companies doing research in interesting directions in computersciences, mathematics, artificial intelligence and other related areas that may prove to be much more powerful weapons in fight against attacks. I think that the true solutions lies not just in ever intractable construction of more and more complex pattern- matching systems, but in approaches that fundamentally redefine the problem and by doing so discover innovative approaches to solving this very real, very important issue of protecting Web Application Assets.

13

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

9. References
[1] Web Application Firewalls; Whats in a name?; Ramon Krikken, Copyright 2009 Burton Group; Presentation at InfoSec 2009 [2] Web App Firewalls: How to Evaluate, Buy, Implement; Mary Brandel; 2009-06-11; http://www.cio.com.au/article/ 307044/web_app_firewalls_how_evaluate_buy_implement [3] Brian Soliman Blog, Technical Articles, 2011-07-25; Common Criteria (CC) for Information Technology Security Evaluation; http://bryansoliman.wordpress.com/2011/07/25/common-criteria-cc-for-informationtechnology-security-evaluation/ [4] WebAppSec.org; 2006; Web Application Firewall Evaluation Criteria (WAFEC) 1.0; Project Coordinator: Ofre Shezaf; http://projects.webappsec.org/f/wasc-wafec-v1.0.pdf [5] WebAppSec.org; 2006; WAFEC Response Matrix; http://projects.webappsec.org/f/ WAFEC_Evaluation_Response_Matrix_1.0.xls [6] OWASP AppSec USA 2010 Keynote; Application Security Its a jungle out there [7] Time to Automate Web Defenses?; Robert Lemos; 2011-10-25; http://www.darkreading.com/vulnerability-management/167901026/security/attacks-breaches/231901651/time-toautomate-web-defenses.html

10. Bibliography
Brian Soliman Blog, Technical Articles, 2011-07-25; Common Criteria (CC) for Information Technology Security Evaluation http://bryansoliman.wordpress.com/2011/07/25/common-criteria-cc-for-information-technology-securityevaluation/ Brickman, J. (2010) Common Criteria a good concept in transformation http://community.ca.com/blogs/iam/ archive/2010/03/25/common-criteria-a-good-concept-in-transformation.aspx F5 Networks and WhiteHat Security Solutions (2008); Vulnerability Assessment Plus Web Application Firewall (VA+WAF)

11. Acknowledgements
Thanks to all the WAF and IPS vendors for all their graciousness and patience. Things always seem to take longer than you expect. The more scrutiny we place our solutions under, the better for everyone. This is a nascent space with a lot of potential. Special thanks to: Kasey Cross at Imperva Ido Berger and Tom Spector at F5 Networks Grant Murphy at Barracuda Networks Renaud Bidou at DenyAll Also I would like to thank NT OBJECTives and Mavituna Security for DAST scanner and filter support. In addition I would like to thank Amir Khan for helping host the systems and Ahmed Masud for valuable editorial commentary.

14

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

Appendix A: Definition of Terms and Acronyms

Blacklist - The list of items, that when matched by a pattern matching system that decides to authorize an operation, will result in rejection of such an authorization. Brute-force attack - An attack that simply tries all possible permutations of a given scenario to discover and exploit any possible vulnerabilities. Cross-site scripting - The ability to include executable code (typically Javascript code) in non Sameorigin contexts. Origin - Defined as the tuple (domain name, application layer protocol, port number) for purpose of resource access control. Resources (objects, methods, instances) can access other resources created from the same origin. See Same-origin context Same-origin context - A programmatic boundary in browser-side programming languages that restricts access to methods, properties and objects across different domains (origins). See also: Origin Deny-by-default - An authorization paradigm where conditions, input, output and processing actions must be explicitly allowed (using a Whitelist). The rationale behind deny-by-default paradigm is to significantly reduce the possibilities of unexpected or malicious behavior. Denial of Service attack - Any attack that affects the availability and normal usage of a service. DoS - Denial of Service: See also Denial of Service Attack DDoS - Distributed Denial of Service. This variation uses multiple clients to perform attacks: See also Denial of Service Attack Fuzzing - See fuzz testing Fuzz Testing - Fuzz testing or fuzzing is a software testing technique that applies random unexpected data to inputs of a computer program. The program is then monitored for unanticipated behavior. HTTP Response Splitting - An attacker's ability to send a single HTTP request that forces the web server to form an output stream, which is then interpreted by the target as two HTTP responses instead of one response. This type of vulnerability can be exploited to perform several web application based attacks. OS Command Injection - An injection attack that utilizes the lack of input validation on web applications that execute out to command line applications. An attacker can use such a vulnerability to execute arbitrary programs on the host operating system. SQL injection attack - SQL injection attack utilizes the lack of input validation vulnerability of web data that an application uses to build a SQL query string. An attacker can use such a vulnerability construct an input in such a way as to execute arbitrary code on the back-end database. Whitelist - See Deny-by-default XSS - See Cross-site Scripting

15

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

Appendix B: Test sites and vulnerability data

Summary of raw results

16

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

Acunetix AspNet - http://testaspnet.vulnweb.com

17

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

Acunetix TestPHP - http://testphp.vulnweb.com

18

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

Cenzic Crackme - http://crackme.cenzic.com

19

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

HP zerowebappsecurity - http://zero.webappsecurity.com

20

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

IBM Testfire - http://demo.testfire.net

21

Larry Suto November 2011

Analyzing The Effectiveness of Web Application Firewalls

NTO Web Scan Test - http://www.webscantest.com

22