Skip to content

JavaScript: Make configuration of parallel extraction consistent with parallel evaluation. #1039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
semmle-qlci merged 1 commit into from
Mar 5, 2019
Merged

JavaScript: Make configuration of parallel extraction consistent with parallel evaluation. #1039

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion change-notes/1.20/extractor-javascript.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

## Changes to code extraction

* Extraction of JavaScript files (but not TypeScript files) on LGTM is now parallelized. By default, the extractor uses as many threads as there are processors, but this can be overridden by setting the `LGTM_INDEX_THREADS` environment variable. In particular, setting `LGTM_INDEX_THREADS` to 1 disables parallel extraction.
* Parallel extraction of JavaScript files (but not TypeScript files) on LGTM is now supported. The `LGTM_THREADS` environment variable can be set to indicate how many files should be extracted in parallel. If this variable is not set, parallel extraction is disabled.
* The extractor now offers experimental support for [E4X](https://developer.mozilla.org/en-US/docs/Archive/Web/E4X), a legacy language extension developed by Mozilla.
* The extractor now supports additional [Flow](https://flow.org/) syntax.
* The extractor now supports [Nullish Coalescing](https://github.com/tc39/proposal-nullish-coalescing) expressions.
Expand Down
10 changes: 5 additions & 5 deletions javascript/extractor/src/com/semmle/js/extractor/AutoBuild.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
* <li><code>LGTM_INDEX_TYPESCRIPT</code>: whether to extract TypeScript
* <li><code>LGTM_INDEX_FILETYPES</code>: a newline-separated list of ".extension:filetype" pairs
* specifying which {@link FileType} to use for the given extension
* <li><code>LGTM_INDEX_THREADS</code>: the maximum number of files to extract in parallel
* <li><code>LGTM_THREADS</code>: the maximum number of files to extract in parallel
* <li><code>LGTM_TRAP_CACHE</code>: the path of a directory to use for trap caching
* <li><code>LGTM_TRAP_CACHE_BOUND</code>: the size to bound the trap cache to
* </ul>
Expand Down Expand Up @@ -163,9 +163,9 @@
* following environment variables are available:
*
* <ul>
* <li><code>LGTM_INDEX_THREADS</code> determines how many threads are used for parallel
* <li><code>LGTM_THREADS</code> determines how many threads are used for parallel
* extraction of JavaScript files (TypeScript files cannot currently be extracted in
* parallel). If left unspecified, the extractor uses as many threads as there are cores.
* parallel). If left unspecified, the extractor uses a single thread.
* <li><code>LGTM_TRAP_CACHE</code> and <code>LGTM_TRAP_CACHE_BOUND</code> can be used to specify
* the location and size of a trap cache to be used during extraction.
* </ul>
Expand Down Expand Up @@ -405,8 +405,8 @@ public void run() throws IOException {
}

private void startThreadPool() {
int defaultNumThreads = Runtime.getRuntime().availableProcessors();
int numThreads = Env.systemEnv().getInt("LGTM_INDEX_THREADS", defaultNumThreads);
int defaultNumThreads = 1;
int numThreads = Env.systemEnv().getInt("LGTM_THREADS", defaultNumThreads);
if (numThreads > 1) {
System.out.println("Parallel extraction with " + numThreads + " threads.");
threadPool = Executors.newFixedThreadPool(numThreads);
Expand Down