Merge rc/1.20 into master

change-notes/1.20/analysis-python.md

@@ -1,19 +1,23 @@
 # Improvements to Python analysis
 
+## General improvements
 
- ## General improvements
+### Extractor changes
 
- > Changes that affect alerts in many files or from many queries
-> For example, changes to file classification
+The extractor now parses all Python code from a single unified grammar. This means that almost all Python code will be successfully parsed, even if mutually incompatible Python code is present in the same project. This also means that Python code for any version can be correctly parsed on a worker running any other supported version of Python. For example, Python 3.7 code is parsed correctly, even if the installed version of Python is only 3.5. This will reduce the number of syntax errors found in many projects.
+
+### Regular expression analysis improvements
+
+The Python `re` (regular expressions) module library has a couple of constants called `MULTILINE` and `VERBOSE` which determine the parsing of regular expressions. Python 3.6 changed the implementation of these constants, which resulted in false positive results for some queries. The relevant QL libraries have been updated to support both implementations which will remove false positive results from projects that use Python 3.6 and later versions.
+
+### API improvements
 
-The constants `MULTILINE` and `VERBOSE` in `re` module, are now understood for Python 3.6 and upward. 
-Removes false positives seen when using Python 3.6, but not when using earlier versions.
 The API has been improved to declutter the global namespace and improve discoverability and readability.
  * New predicates `ModuleObject::named(name)` and `ModuleObject.attr(name)` have been added, allowing more readable access to common objects. For example, `(any ModuleObject m | m.getName() = "sys").getAttribute("exit")` can be replaced with `ModuleObject::named("sys").attr("exit")`
- * The API for accessing builtin functions has been improved. Predicates of the form `theXXXFunction()`, such as `theLenFunction()`, have been deprecated in favour of `Object::builtin(name)`.
+ * The API for accessing builtin functions has been improved. Predicates of the form `theXXXFunction()`, such as `theLenFunction()`, have been deprecated in favor of `Object::builtin(name)`.
  * A configuration based API has been added for writing data flow and taint tracking queries. This is provided as a convenience for query authors who have written data flow or taint tracking queries for other languages, so they can use a similar format of query across multiple languages.
 
- ## New queries
+## New queries
 
  | **Query**                   | **Tags**  | **Purpose**                                                        |
 |-----------------------------|-----------|--------------------------------------------------------------------|
@@ -24,7 +28,7 @@ The API has been improved to declutter the global namespace and improve discover
 | Overly permissive file permissions (`py/overly-permissive-file`) | security, external/cwe/cwe-732 | Finds instances where a file is created with overly permissive permissions. Results are not shown on LGTM by default. |
 | Use of insecure SSL/TLS version (`py/insecure-protocol`) | security, external/cwe/cwe-327 | Finds instances where a known insecure protocol has been specified. Results are shown on LGTM by default. |
 
- ## Changes to existing queries
+## Changes to existing queries
 
  | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
@@ -35,11 +39,8 @@ The API has been improved to declutter the global namespace and improve discover
 | Unused import (`py/unused-import`) | Fewer false positive results | Results where the imported module is used in a `doctest` string are no longer reported. |
 | Unused import (`py/unused-import`) | Fewer false positive results | Results where the imported module is used in a type-hint comment are no longer reported. |
 
- ## Changes to code extraction
-
- * The extractor now parses all Python code from a single unified grammar. This means that almost all Python code will be successfully parsed, even if mutually incompatible Python code is present in the same project. This also means that Python code for any version can be correctly parsed on a worker running any other supported version of Python. For example, Python 3.7 code is parsed correctly, even if the installed version of Python is only 3.5.
 
- ## Changes to QL libraries
+## Changes to QL libraries
 
  * Added support for the `dill` pickle library.
  * Added support for the `bottle` web framework.

change-notes/1.20/extractor-javascript.md

@@ -2,25 +2,11 @@
 
 # Improvements to JavaScript analysis
 
-> NOTES
->
-> Please describe your changes in terms that are suitable for
-> customers to read. These notes will have only minor tidying up
-> before they are published as part of the release notes.
->
-> This file is written for lgtm users and should contain *only*
-> notes about changes that affect lgtm enterprise users. Add
-> any other customer-facing changes to the `studio-java.md`
-> file.
->
-
-## General improvements
-
 ## Changes to code extraction
 
-* Parallel extraction of JavaScript files (but not TypeScript files) on LGTM is now supported. The `LGTM_THREADS` environment variable can be set to indicate how many files should be extracted in parallel. If this variable is not set, parallel extraction is disabled.
-* The extractor now offers experimental support for [E4X](https://developer.mozilla.org/en-US/docs/Archive/Web/E4X), a legacy language extension developed by Mozilla.
-* The extractor now supports additional [Flow](https://flow.org/) syntax.
-* The extractor now supports [Nullish Coalescing](https://github.com/tc39/proposal-nullish-coalescing) expressions.
-* The extractor now supports [TypeScript 3.2](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-2.html).
-* The TypeScript extractor now handles the control-flow of logical operators and destructuring assignments more accurately.
+* Parallel extraction of JavaScript files (but not TypeScript files) on LGTM is now supported. If LGTM is configured to evaluate queries using multiple threads, then JavaScript files are also extracted using multiple threads.
+* Experimental support for [E4X](https://developer.mozilla.org/en-US/docs/Archive/Web/E4X), a legacy language extension developed by Mozilla, is available.
+* Additional [Flow](https://flow.org/) syntax is now supported.
+* [Nullish Coalescing](https://github.com/tc39/proposal-nullish-coalescing) expressions are now supported.
+* [TypeScript 3.2](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-2.html) is now supported.
+* The TypeScript extractor now handles the control flow of logical operators and destructuring assignments more accurately.

change-notes/1.20/support/versions-compilers.csv

@@ -13,4 +13,4 @@ Java,"Java 6 to 11 [2]_.","javac (OpenJDK and Oracle JDK)
 Eclipse compiler for Java (ECJ) batch compiler",``.java``
 JavaScript,ECMAScript 2018 or lower,Not applicable,"``.js``, ``.jsx``, ``.mjs``, ``.es``, ``.es6``, ``.htm``, ``.html``, ``.xhm``, ``.xhtml``, ``.vue``, ``.json`` [3]_."
 Python,"2.7, 3.5, 3.6, 3.7",Not applicable,``.py``
-TypeScript [4]_.,"2.6, 2.7, 2.8, 2.9, 3.0, 3.1",Standard TypeScript compiler,"``.ts``, ``.tsx``"
+TypeScript [4]_.,"2.6, 2.7, 2.8, 2.9, 3.0, 3.1, 3.2",Standard TypeScript compiler,"``.ts``, ``.tsx``"

csharp/ql/src/API Abuse/DisposeNotCalledOnException.ql

@@ -49,18 +49,28 @@ predicate isTriedAgainstException(ControlFlowElement cfe, ExceptionClass ec) {
   )
 }
 
+private class DisposeCall extends MethodCall {
+  DisposeCall() { this.getTarget() instanceof DisposeMethod }
+}
+
+private predicate reachesDisposeCall(DisposeCall disposeCall, DataFlow::Node node) {
+  DataFlow::localFlowStep(node, DataFlow::exprNode(disposeCall.getQualifier()))
+  or
+  exists(DataFlow::Node mid | reachesDisposeCall(disposeCall, mid) |
+    DataFlow::localFlowStep(node, mid)
+  )
+}
+
 /**
  * Holds if `disposeCall` disposes the object created by `disposableCreation`.
  */
-predicate disposeReachableFromDisposableCreation(MethodCall disposeCall, Expr disposableCreation) {
+predicate disposeReachableFromDisposableCreation(DisposeCall disposeCall, Expr disposableCreation) {
   // The qualifier of the Dispose call flows from something that introduced a disposable into scope
   (
     disposableCreation instanceof LocalScopeDisposableCreation or
     disposableCreation instanceof MethodCall
   ) and
-  DataFlow::localFlowStep+(DataFlow::exprNode(disposableCreation),
-    DataFlow::exprNode(disposeCall.getQualifier())) and
-  disposeCall.getTarget() instanceof DisposeMethod
+  reachesDisposeCall(disposeCall, DataFlow::exprNode(disposableCreation))
 }
 
 class MethodCallThatMayThrow extends MethodCall {
@@ -73,7 +83,7 @@ ControlFlowElement getACatchOrFinallyClauseChild() {
   result = getACatchOrFinallyClauseChild().getAChild()
 }
 
-from MethodCall disposeCall, Expr disposableCreation, MethodCallThatMayThrow callThatThrows
+from DisposeCall disposeCall, Expr disposableCreation, MethodCallThatMayThrow callThatThrows
 where
   disposeReachableFromDisposableCreation(disposeCall, disposableCreation) and
   // The dispose call is not, itself, within a dispose method.

csharp/ql/src/semmle/code/csharp/Assignable.qll

@@ -76,6 +76,11 @@ class AssignableRead extends AssignableAccess {
     not nameOfChild(_, this)
   }
 
+  pragma[noinline]
+  private ControlFlow::Node getAnAdjacentReadSameVar() {
+    Ssa::Internal::adjacentReadPairSameVar(this.getAControlFlowNode(), result)
+  }
+
   /**
    * Gets a next read of the same underlying assignable. That is, a read
    * that can be reached from this read without passing through any other reads,
@@ -102,7 +107,7 @@ class AssignableRead extends AssignableAccess {
    */
   AssignableRead getANextRead() {
     forex(ControlFlow::Node cfn | cfn = result.getAControlFlowNode() |
-      Ssa::Internal::adjacentReadPairSameVar(this.getAControlFlowNode(), cfn)
+      cfn = this.getAnAdjacentReadSameVar()
     )
   }
 

csharp/ql/src/semmle/code/csharp/dataflow/DataFlow.qll

@@ -704,6 +704,7 @@ module DataFlow {
       )
     }
 
+    pragma[nomagic]
     private ControlFlowElement getANonExactScopeChild(ControlFlowElement scope) {
       scope = getAScope(false) and
       result = scope

javascript/extractor/src/com/semmle/js/extractor/ASTExtractor.java

@@ -42,7 +42,6 @@
 import com.semmle.js.ast.Identifier;
 import com.semmle.js.ast.IfStatement;
 import com.semmle.js.ast.ImportDeclaration;
-import com.semmle.js.ast.ImportNamespaceSpecifier;
 import com.semmle.js.ast.ImportSpecifier;
 import com.semmle.js.ast.InvokeExpression;
 import com.semmle.js.ast.JumpStatement;
@@ -1449,13 +1448,7 @@ public Label visit(ImportDeclaration nd, Context c) {
     public Label visit(ImportSpecifier nd, Context c) {
       Label lbl = super.visit(nd, c);
       visit(nd.getImported(), lbl, 0, IdContext.label);
-      visit(
-          nd.getLocal(),
-          lbl,
-          1,
-          nd instanceof ImportNamespaceSpecifier
-              ? IdContext.varAndNamespaceDecl
-              : IdContext.varAndTypeAndNamespaceDecl);
+      visit(nd.getLocal(), lbl, 1, IdContext.varAndTypeAndNamespaceDecl);
       return lbl;
     }
 

javascript/extractor/src/com/semmle/js/extractor/ScopeManager.java

@@ -17,7 +17,6 @@
 import com.semmle.js.ast.Identifier;
 import com.semmle.js.ast.IfStatement;
 import com.semmle.js.ast.ImportDeclaration;
-import com.semmle.js.ast.ImportNamespaceSpecifier;
 import com.semmle.js.ast.ImportSpecifier;
 import com.semmle.js.ast.LabeledStatement;
 import com.semmle.js.ast.LetExpression;
@@ -559,9 +558,7 @@ public Void visit(ImportDeclaration nd, Void c) {
 
       @Override
       public Void visit(ImportSpecifier nd, Void c) {
-        return visit(
-            nd.getLocal(),
-            nd instanceof ImportNamespaceSpecifier ? DeclKind.varAndNamespace : DeclKind.all);
+        return visit(nd.getLocal(), DeclKind.all);
       }
 
       @Override

javascript/extractor/tests/es2015/output/trap/import5.js.trap

@@ -65,45 +65,49 @@ isModule(#20001)
 isES2015Module(#20001)
 #20021=@"var;{foo};{#20020}"
 variables(#20021,"foo",#20020)
-#20022=@"local_namespace_name;{foo};{#20020}"
-local_namespace_names(#20022,"foo",#20020)
+#20022=@"local_type_name;{foo};{#20020}"
+local_type_names(#20022,"foo",#20020)
+#20023=@"local_namespace_name;{foo};{#20020}"
+local_namespace_names(#20023,"foo",#20020)
 variables(#20021,"foo",#20020)
-local_namespace_names(#20022,"foo",#20020)
-#20023=*
-stmts(#20023,27,#20001,0,"import  ...  'foo';")
-hasLocation(#20023,#20003)
-stmtContainers(#20023,#20001)
+local_type_names(#20022,"foo",#20020)
+local_namespace_names(#20023,"foo",#20020)
 #20024=*
-exprs(#20024,4,#20023,-1,"'foo'")
-hasLocation(#20024,#20015)
-enclosingStmt(#20024,#20023)
-exprContainers(#20024,#20001)
-literals("foo","'foo'",#20024)
+stmts(#20024,27,#20001,0,"import  ...  'foo';")
+hasLocation(#20024,#20003)
+stmtContainers(#20024,#20001)
 #20025=*
-exprs(#20025,85,#20023,0,"* as foo")
-#20026=@"loc,{#10000},1,8,1,15"
-locations_default(#20026,#10000,1,8,1,15)
-hasLocation(#20025,#20026)
-enclosingStmt(#20025,#20023)
+exprs(#20025,4,#20024,-1,"'foo'")
+hasLocation(#20025,#20015)
+enclosingStmt(#20025,#20024)
 exprContainers(#20025,#20001)
-#20027=*
-exprs(#20027,78,#20025,1,"foo")
-hasLocation(#20027,#20011)
-enclosingStmt(#20027,#20023)
-exprContainers(#20027,#20001)
-literals("foo","foo",#20027)
-decl(#20027,#20021)
-namespacedecl(#20027,#20022)
+literals("foo","'foo'",#20025)
+#20026=*
+exprs(#20026,85,#20024,0,"* as foo")
+#20027=@"loc,{#10000},1,8,1,15"
+locations_default(#20027,#10000,1,8,1,15)
+hasLocation(#20026,#20027)
+enclosingStmt(#20026,#20024)
+exprContainers(#20026,#20001)
 #20028=*
-entry_cfg_node(#20028,#20001)
-#20029=@"loc,{#10000},1,1,1,0"
-locations_default(#20029,#10000,1,1,1,0)
-hasLocation(#20028,#20029)
-#20030=*
-exit_cfg_node(#20030,#20001)
-hasLocation(#20030,#20019)
-successor(#20023,#20030)
-successor(#20025,#20023)
-successor(#20028,#20025)
+exprs(#20028,78,#20026,1,"foo")
+hasLocation(#20028,#20011)
+enclosingStmt(#20028,#20024)
+exprContainers(#20028,#20001)
+literals("foo","foo",#20028)
+decl(#20028,#20021)
+typedecl(#20028,#20022)
+namespacedecl(#20028,#20023)
+#20029=*
+entry_cfg_node(#20029,#20001)
+#20030=@"loc,{#10000},1,1,1,0"
+locations_default(#20030,#10000,1,1,1,0)
+hasLocation(#20029,#20030)
+#20031=*
+exit_cfg_node(#20031,#20001)
+hasLocation(#20031,#20019)
+successor(#20024,#20031)
+successor(#20026,#20024)
+successor(#20029,#20026)
 numlines(#10000,1,1,0)
 filetype(#10000,"javascript")