WASC / OWASP References for all vulnerabilities

[andresriancho]

Introduction

vulns.py was the first step towards having all vulnerabilities referenced against OWASP / WASC or some other third-party.

Also, this is the first step towards having long vulnerability descriptions, URLs with references for each vulnerability, solutions for each vulnerability, etc.

What needs to be done

My idea about how to integrate all this data into w3af is simple:

• A plugin discovers a vulnerability and calls Vuln.from_mutant
• The from_mutant takes a vulnerability name as parameter (note that this parameter is the one stored in vulns.py and where a unique ID is assigned to it). Based on this vulnerability ID, we should load all the other vulnerability information from a file.

New methods that should be defined in info.py:

• get_long_desc(): Long description for the vulnerability
• get_vuln_references(): List of VulnReference objects which hold a URL pointing to useful resources related to the vulnerability and an optional description of the reference.
• get_vuln_solution(): A text describing how to solve this vulnerability
• get_wasc_id(): The WASC id for this vulnerability

If a plugin calls from_mutant of from_fr and the name is not found in the data source, an exception/warning should be raised.

Since some vulnerabilities have "different names" but are the same vulnerability (for example: "Cross-Site Scripting with Content-Security Policy enabled" vs. "Cross-Site Scripting") a parameter called id should be added to from_mutant and from_fr in order to reference the id in vulns.py and allow the plugin writer to put "anything in the name" but still get all the data (long_desc, vuln_references, vuln_solution, wasc_id) loaded.

JSON backend

All the data should be stored in individual JSON files, one for each vulnerability. These files should be stored in core/data/constants/vulns/en/ and the name should be <vuln-id>-<vulnerability-name>.json , where <vuln-id> is the ID that's present in vulns.py and the <vulnerability-name> is a human readable name.

When w3af searches for the long description of a vulnerability we'll use the filename's first part, <vuln-id>, and ignore the rest (which is just useful for us humans)

JSON files should look similar to this example:

{
  "id": 12345,
  "title": "Cross-Site Scripting",
  "description": "A very long description for Cross-Site Scripting",
  "solution": "A very long text explaining how to fix XSS vulnerabilities",
  "WASC": [0003],
  "CWE": [0003, 0007],
  "OWASP_TOP_10": {
        "2010": [1],
        "2013": [2]
  },
  "fix-effort": 60,
  "references": [
      {"URL": "http://foo.com/xss", "desc": "First reference to XSS vulnerability"},
      {"URL": "http://asp.net/xss", "desc": "How to fix XSS vulns in ASP.NET"},
      {"URL": "http://owasp.org/xss", "desc": ""},
    ]
  }

Notes about the JSON file:

• "id" holds the vulnerability ID from vulns.py. The contents of the "id" field must match filename and must be in vulns.py (both things MUST be unittested).
• The "OWASP" fields points to the OWASP Top10 category. We have different versions of this (2010 and 2013) to match the different releases of the OWASP Top10 project
• "fix-effort" gives the user an idea of how much time it will take to solve this vulnerability, it's stored in minures
• Since a long blob of unformatted text is hard to read for any user, the description and solution text MUST use RST for formatting. There are various python libraries which translate markdown to HTML, which will allow us to show the text in the GUI and or Web UI.

Long strings in JSON

JSON has an awful limitation: "No multi-line strings". In our case this is very limiting since we don't want to have awfully long lines in these sections:

  "description": "A very long description for Cross-Site Scripting",
  "solution": "A very long text explaining how to fix XSS vulnerabilities",

So, the parser should check if the description and solution fields are strings or arrays. If they are arrays, the contents of the array should be joined using \n. For example:

  "description": ["A very long description for Cross-Site Scripting",
                         "which has more than one line",
                         "and will be joined by the python parser"]

Python wrapper

It would be great to have a simple python wrapper that would return an object representation of the JSON object, with all the getters and setters, so that it can be easily used in our software.

Translations

One of the cool things about this architecture is that it will allow us to easily add translations. Just adding a new set of JSON files (keeping the vulnerability IDs) would work.

Implementation details:

• The English version will be in core/data/constants/vulns/
• Unix's environment variable LANG is used for setting the language
• If there is no translation for the current LANG then the English version is used
• Translations will be in core/data/constants/vulns/ru/ , core/data/constants/vulns/es/ , etc.
• The translation json files will override the parts of the main JSON file which is in English, for example, the main file contains:

  "title": "Cross-Site Scripting",

And then the Russian translation file (for the same vulnerability id) says:

  "title": "Cross-Site Scripting in Russian",

And if the LANG environment variable is set to Russian then the python wrapper should return Cross-Site Scripting in Russian as the title for this vulnerability. Any field from the main JSON file can be overridden (for example you can override a link to wikipedia to point to the XSS description in Russian).

Content

The code we write for adding meta-data to information/vulnerability objects should allow us to slowly migrate from our current state (no meta-data) to the final objective (meta-data for all vulnerabilities). In other words, the new methods added to Info shouldn't fail if there is no JSON for that vulnerability id, and the parser shouldn't crash if there is no such file.

The content of the JSON files could be retrieved from:

• http://cwe.mitre.org/data/xml/cwec_v2.8.xml.zip

The license for the CWE data is here and:

• Allows use in w3af
• Allows commercial use
• Requires us to show the license Any copy you make for such purposes is authorized provided that you reproduce MITRE’s copyright designation and this license in any such copy

Sent email to CWE contact asking if we can have a "Attribution-ShareAlike CC BY-SA" license for the JSON files.

Unittesting

• We should write a json-schema for our JSON, and then validate the JSON after each push with a unittest. See https://python-jsonschema.readthedocs.org/en/latest/ for more information about json-schemas
  • For each value in WASC/OWASP, make sure that we can generate a link like http://owasp.org/foo/<id>, the URL is valid, site is online, not 404
  • For each URL field in the schema, we need to check that:
  • It's a valid URL
  • The site is online, and no 404 is returned
  • fix-effort value is in the right format
  • The RST in both description and solution are well formed RST

[andresriancho]

A potential source for the data to generate the JSON might be: CWE's XML

[andresriancho]

@ST2Labs started to work on this

[andresriancho]

@ST2Labs ping!

[bhuvi11]

Hello @andresriancho,
I am curious to understand how is CWE mapping done for each vulnerability ID?
I saw the above post , it says we can find the json for vulnerability with the cwe id, but in the result i dont find cwe available
Can you help me to understand it more?