GCN article: In defense of common criteria

In defense of common criteria

By William Jackson / GCN

THE COMMON CRITERIA Evaluation and Validation Scheme has been heavily criticized lately. Devised as an independent evaluation of security products against a set of standard criteria, Common Criteria has been faulted for being expensive and not providing a foolproof measure to increase security. Not everyone shares these views.

Mary Ann Davidson, chief security officer at Oracle, for one, feels Common Criteria has a number of strengths.

GCN: It seems as if the perceived value of a Common Criteria evaluation depends in large part on how a vendor approaches the process. Those that put the most into it get the best value from the investment. Is this true?
DAVIDSON: The value of assurance is the extent to which a vendor embraces it across its development processes. That said, since every vendor of [information technology] products claims, "Our product is secure: trust us!" having a third party validate the product against the Common Criteria is tremendously valuable to customers, who otherwise would have to rely on unproven security claims. Also, many vendors, including Oracle, view the Common Criteria as the starting point for assurance, not the ending point.

GCN: How do you use the Common Criteria evaluation to create a reliable, repeatable development process?
DAVIDSON: The Common Criteria allows vendors to start their evaluations with a lower Evaluation Assurance Level and improve their processes to meet a higher assurance level over time. The higher in assurance levels you go, the more aspects of your development process the evaluators validate, and thus you need more process to meet the requirements. This avoids an all-or-nothing benchmark that few vendors could meet and allows them to improve their assurance over time.

GCN: How helpful are automated vulnerability assessment tools in improving the quality of your products and in achieving evaluation?
DAVIDSON: Automated vulnerability assessment tools do not come into play in Common Criteria until you reach those Evaluation Assurance Levels that are higher than those mutually recognized under the Common Criteria Recognition Arrangement. The national schemes that use such tools do not release them to vendors, which means they are of no use in helping improve product security.

The main value of automated vulnerability assessment tools is finding and fixing problems during development, before products ship. Also, automated vulnerability assessment tools are just one component of a robust, comprehensive assurance program. Oracle uses multiple tools as part of its Software Security Assurance program.

GCN: What are the weaknesses of these tools, and why are they not required in Common Criteria?
DAVIDSON: There is no tool validation program. Anybody can create a tool and -- even if wildly inaccurate -- claim they found a problem, and the burden is on the product vendor to prove that there isn't a problem instead of on the tool vendor to prove its tool is accurate.

Automated tools can find, at best, half the common security defects in software, and they miss many design defects. Also, in a product with millions of lines of code, if the tool has a 90 percent false-positive rate, the vendor could spend thousands of hours in nonrecoverable time chasing false alarms instead of actually improving security. Finally, these tools do not validate whether a product has any useful security functionality.

GCN: How do you synchronize your development process with the National Information Assurance Partnership evaluation process so products are not outdated by the time they have been evaluated?
DAVIDSON: The National Information Assurance Partnership is merely one of the national schemes under which vendors can evaluate their products. Under the Common Criteria Recognition Arrangement, vendors can do an evaluation up to Evaluation Assurance Level four that is accepted in other venues. In Oracle's experience, we can evaluate a large, complex product like our database in about six months. The length of the evaluation cycle has not been an impediment to customer adoption of the product. A vendor going through an evaluation for the first time or who does not have well-developed development processes may take longer to go through an evaluation.

GCN: How do you select an evaluation laboratory? And is it possible to shop for labs to get a favorable evaluation?
DAVIDSON: Labs do not have the final say on whether a product completes an evaluation successfully; the national schemes that certify the labs do. A lab doing substandard work would face scrutiny by the national scheme. Generally, vendors shop for labs based on expertise and cost. We do our evaluations primarily in the United Kingdom and in Germany because we found these labs to have a higher level of expertise in Oracle software and an acceptable cost versus labs in other countries.

GCN: A frequent complaint of Common Criteria is that it focuses on process rather than the product. Has Common Criteria helped improve your products? If so, how?
DAVIDSON: Oracle continues to invest significant resources in building market-leading new functionality and products that we evaluate under the Common Criteria so that our evaluation-aware customers will feel comfortable using the products. We also continue to improve development processes; for example, we have formal processes addressing security vulnerabilities that we have included in our evaluations under flaw remediation.

GCN: What are the strengths and weaknesses in Common Criteria as it now stands?
DAVIDSON: Common Criteria has a number of strengths. Common Criteria considers both threats and the technical remedies needed to counter those threats; a product must have actual security functionality and assurance proof points for it. Common Criteria evaluation assurance levels are graduated, so that vendors can improve their assurance level over time. Also, secure development processes need to be demonstrable and repeatable.

Common Criteria is flexible. Vendors can assert a security claim, such as the use of automated vulnerability tools, through the target of evaluation they choose. And because of the Common Criteria Recognition Arrangement, evaluations are cost-effective. A vendor can do one evaluation that is recognized and accepted in multiple venues. Many vendors who complain about Common Criteria did not experience the pre-Common Criteria days when vendors evaluated the exact same product in multiple countries.

As far as challenges, Common Criteria is a committee-led organization representing 24 countries and is thus often slow to change. There is a lack of transparency in its structure, its decision-making and its technical review processes that the Common Criteria Vendors? Forum has raised. The organization is slowly starting to address these issues.

GCN: What changes would you like to see made in Common Criteria?
DAVIDSON: Any critical piece of software should be designed in consideration of what kinds of threats the product is likely to face and the appropriate technical remedy for those threats. As threats evolve, so should Common Criteria.

That said, the National Cyber Security Partnership recommended the use of automated vulnerability testing tools at lower assurance levels, which could be a useful change, provided that all the criteria are met. The tools themselves must be evaluated and validated for what they find and how well they find it. The vendor can only make claims about use of automated tools as part of an evaluation based only on the tools they actually use in development. The vendor must assure that the Common Criteria Recognition Arrangement still applies: The use of these tools is mutually accepted in multiple venues. And, finally, the vendor must continue to maintain control of its source code. That is, the company is protected by contractual relationships it has with its labs but is not required to give source to any other party.

GCN: How well is the National Information Assurance Partnership working with industry to make needed changes in the criteria and processes?
DAVIDSON: Since the National Information Assurance Partnership is just one of the national schemes under which vendors can do Common Criteria evaluations, vendors who want the Common Criteria improved should work within the Common Criteria Vendors' Forum. Oracle also believes it is duplicative and wasteful for individual schemes to want country-specific variants that would require vendors to evaluate their products only under those schemes.