As an organization’s representative for ‘Application Security Service line’, more often I provide presentations to customers who are key owners of Application Security Market in their respective organization. At times, this include CISO’s also.
During these times, the most often asked question is what kind of a value add, we can give in enabling them manage all their vulnerabilities, do security automation at the time help them improve the security posture of their applications year on year.
To do this first of all, one needs to have a clear understanding of where they are and where they need to go. Now, how does one know where they are? To do this, you first need a complete understanding of the assets you are trying to protect, your regulatory/compliance needs, your organization policies, what you need to protect and why you need to protect them. The which/when and how comes later through the help of Application Security Metrics.
First I will list down some of the metrics and then explain how to use this to figure where you are.
Application Security Metrics:
- Security Coverage
- Remediation Window or Vulnerability Age
- Vulnerability Trend by Month
- Vulnerability Density
- Vulnerability Distribution by severity, by status, by category.
- Rejection Rate
- Tool Efficiency
- Compliance Percentage
- Defect Recurrence Rate
- False Negatives/Positive Rate
Though there are more, we will cover these 10 for now.
- Security Coverage -> This helps you decide whether you are actually doing security assessments for all your assets. To do this, first you need a proper inventory on all your assets. Assets could be your applications, end points, network devices or the IT Infrastructure.
Unless you have total control of your entire asset, it would be impossible to protect them. A well known organization I had worked with had proper SDLC process but their HR department floated a website using a third party vendor which didn’t go through the release management process. The website was later hacked.
So, get all your assets first and get them through the SDLC/Release Management process. Then, make them go through security assessments to see if you are actually covering all.
2. Remediation Window or Vulnerability Age: This is the time it takes for a team to fix the vulnerability after it was detected. Or, this can be also called as your effective ‘internal zero day’ as you know that the vulnerability is there and a patch is not available yet.
3. Vulnerability Trend by Month: This helps in deciding how the vulnerabilities are introduced into the software month on month; whether you are improving, going down or whether its just adhoc.
4. Vulnerability Density: This helps in deciding how vulnerable your software is. It can be calculated not just for source code, but also for dynamic assessments and infrastructure security assessments.
5. Vulnerability Distribution by status, severity and category: Status is nothing but the stages your vulnerability is in once its created in the vulnerability management system. It could be New/Unresolved, Fixed, False Positive, etc; Severity is the risk rating that you assign. It could either be a 3 point rating like High/Medium/Low or have your customized rating. Category helps in deciding the concentration of vulnerabilities. Lets say that 70% of the vulnerabilities is in security configuration and 30% is in authorization. That helps in deciding where to channel your energy.
6. Rejection Rate: This again is an indicator to check how vulnerable and exploitable the software is.
7. Tool Efficiency: One thing that surprises me is that organizations buy a lot of tools and don’t use it to their efficient best. They either remain idle or organization thinks that the tool purchase didn’t go well. Unless you put in a metric to calculate the efficiency of the tool, you cannot assume anything here.
8. Compliance Percentage: This helps in compliance with respect to PCI, HIPAA etc.
9. Defect Recurrence Rate: If a closed vulnerability recurs again due to a broken fix; how do you know whether its the same vulnerability that was reported earlier or whether the ADM don’t effectively know how to fix the defect?
10. False Positives/Negatives: This again is tied to your tool and to the manual analysis that you do. Always choose a tool that gives a best trade off between false positives and false negatives.