From 2ac2233e695efbb793adfeb86390727c32bcf032 Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 16 Nov 2018 14:41:55 +0000 Subject: [PATCH 1/3] Add change note for enhance visibility of security alerts and conversion to path-queries. --- change-notes/1.19/analysis-python.md | 38 ++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 change-notes/1.19/analysis-python.md diff --git a/change-notes/1.19/analysis-python.md b/change-notes/1.19/analysis-python.md new file mode 100644 index 000000000000..122d367c5bf1 --- /dev/null +++ b/change-notes/1.19/analysis-python.md @@ -0,0 +1,38 @@ +# Improvements to Python analysis + + +## General improvements + +> Changes that affect alerts in many files or from many queries +> For example, changes to file classification + +## New queries + +| **Query** | **Tags** | **Purpose** | +|-----------------------------|-----------|--------------------------------------------------------------------| +| *@name of query (Query ID)* | *Tags* |*Aim of the new query and whether it is enabled by default or not* | + +## Changes to existing queries + +All taint-tracking queries now support visualization of paths in QL for Eclipse. +Most security alerts are now visible on LGTM by default. + +| **Query** | **Expected impact** | **Change** | +|----------------------------|------------------------|------------------------------------------------------------------| +| Code injection (`py/code-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | +| Deserializing untrusted input (`py/unsafe-deserialization`) | Supports path visualization | No change to expected results | +| Information exposure through an exception (`py/stack-trace-exposure`) | Now visible on LGTM by default | No change to expected results | +| Reflected server-side cross-site scripting (`py/reflective-xss`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | +| SQL query built from user-controlled sources (`py/sql-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | +| Uncontrolled data used in path expression (`py/path-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | +| Uncontrolled command line (`py/command-line-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | +| URL redirection from remote source (`py/url-redirection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | + +## Changes to code extraction + +* *Series of bullet points* + +## Changes to QL libraries + +* *Series of bullet points* + From 95f1935eaa6adb16ad53226b7f20ba2ce32b1fea Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 23 Nov 2018 12:55:04 +0000 Subject: [PATCH 2/3] Python change notes: Merge in internal change notes. --- change-notes/1.19/analysis-python.md | 66 +++++++++++++++++++++++++--- 1 file changed, 61 insertions(+), 5 deletions(-) diff --git a/change-notes/1.19/analysis-python.md b/change-notes/1.19/analysis-python.md index 122d367c5bf1..7a9bc55026d7 100644 --- a/change-notes/1.19/analysis-python.md +++ b/change-notes/1.19/analysis-python.md @@ -6,11 +6,57 @@ > Changes that affect alerts in many files or from many queries > For example, changes to file classification +### Representation of the control flow graph + +The representation of the control flow graph (CFG) has been modified to better reflect the semantics of Python. + +The following statement types no longer have a CFG node for the statement itself, as their sub-expressions already contain all the +semantically significant information: + +* `ExprStmt` +* `If` +* `Assign` +* `Import` + +For example, the CFG for `if cond: foo else bar` now starts with the CFG node for `cond`. + +For the following statement types, the CFG node for the statement now follows the CFG nodes of its sub-expressions to better reflect the semantics: + +* `Print` +* `TemplateWrite` +* `ImportStar` + +For example the CFG for `print foo` (in Python 2) has changed from `print -> foo` to `foo -> print`, better reflecting the runtime behavior. + + +The CFG for the `with` statement has been re-ordered to more closely reflect the semantics. +For the `with` statement: +```python +with cm as var: + body +``` +The order of the CFG changes from: + + + cm + var + body + +to: + + cm + + var + body + +A new predicate `Stmt.getAnEntryNode()` has been added to make it easier to write reachability queries involving statements. + + ## New queries | **Query** | **Tags** | **Purpose** | |-----------------------------|-----------|--------------------------------------------------------------------| -| *@name of query (Query ID)* | *Tags* |*Aim of the new query and whether it is enabled by default or not* | +| Information exposure through an exception (`py/stack-trace-exposure`) | security, external/cwe/cwe-209, external/cwe/cwe-497 | Finds instances where information about an exception may be leaked to an external user. Enabled on LGTM by default. | ## Changes to existing queries @@ -19,20 +65,30 @@ Most security alerts are now visible on LGTM by default. | **Query** | **Expected impact** | **Change** | |----------------------------|------------------------|------------------------------------------------------------------| +| Assert statement tests the truth value of a literal constant (`py/assert-literal-constant`) | reliability, correctness | Checks whether an assert statement is testing the truth of a literal constant value. Not shown by default. | | Code injection (`py/code-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | | Deserializing untrusted input (`py/unsafe-deserialization`) | Supports path visualization | No change to expected results | -| Information exposure through an exception (`py/stack-trace-exposure`) | Now visible on LGTM by default | No change to expected results | +| Encoding error (`py/encoding-error`) | Better alert location | Alert is now shown at the position of the first offending character, rather than at the top of the file. | +| Missing call to \_\_init\_\_ during object initialization (`py/missing-call-to-init`) | Fewer false positive results | Results where it is likely that the full call chain has not been analyzed are no longer reported. | | Reflected server-side cross-site scripting (`py/reflective-xss`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | | SQL query built from user-controlled sources (`py/sql-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | | Uncontrolled data used in path expression (`py/path-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | | Uncontrolled command line (`py/command-line-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | -| URL redirection from remote source (`py/url-redirection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results | +| URL redirection from remote source (`py/url-redirection`) | Fewer false positive results and now supports path visualization | Taint is no longer tracked from the right hand side of binary expressions. In other words `SAFE + TAINTED` is now treated as safe. | + ## Changes to code extraction -* *Series of bullet points* +* Improved scalability: Scaling is near linear to at least 20 CPU cores. +* Five levels of logging can be selected: `CRITICAL`, `ERROR`, `WARN`, `INFO` and `DEBUG`. `WARN` is the default. +* The `-v` flag can be specified twice to increase logging level to `DEBUG` +* The `-q` flag has been added to reduce the logging level to `ERROR` or `CRITICAL` +* Log lines are now in the `[SEVERITY] message` style and never overlap. +* Extractor now outputs the location of the first offending character when an EncodingError is encountered. ## Changes to QL libraries -* *Series of bullet points* +* Taint tracking analysis now understands HTTP requests in the `twisted` library. +* The analysis now handles `isinstance` and `issubclass` tests involving the basic abstract base classes better. For example, the test `issubclass(list, collections.Sequence)` is now understood to be `True` +* Taint tracking automatically tracks tainted mappings and collections, without you having to add additional taint kinds. This means that custom taints are tracked from `x` to `y` in the following flow: `l = [x]; y =l[0]`. From 4f5cfbc336c1322b9b3b48dc1ef3c19eb634ac38 Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 23 Nov 2018 13:03:16 +0000 Subject: [PATCH 3/3] Correct change for extractor logging levels. --- change-notes/1.19/analysis-python.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/change-notes/1.19/analysis-python.md b/change-notes/1.19/analysis-python.md index 7a9bc55026d7..b5bd513d6f7d 100644 --- a/change-notes/1.19/analysis-python.md +++ b/change-notes/1.19/analysis-python.md @@ -80,9 +80,9 @@ Most security alerts are now visible on LGTM by default. ## Changes to code extraction * Improved scalability: Scaling is near linear to at least 20 CPU cores. -* Five levels of logging can be selected: `CRITICAL`, `ERROR`, `WARN`, `INFO` and `DEBUG`. `WARN` is the default. -* The `-v` flag can be specified twice to increase logging level to `DEBUG` -* The `-q` flag has been added to reduce the logging level to `ERROR` or `CRITICAL` +* Five levels of logging can be selected: `ERROR`, `WARN`, `INFO`, `DEBUG` and `TRACE`. `WARN` is the stand-alone default, but `INFO` will be used when run by LGTM. +* The `-v` flag can be specified multiple times to increase logging level by one per `-v`. +* The `-q` flag has been added and can be specified multiple times to reduce the logging level by one per `-q`. * Log lines are now in the `[SEVERITY] message` style and never overlap. * Extractor now outputs the location of the first offending character when an EncodingError is encountered.