Switched 'ts.performance' to a mixed mode only uses native performance APIs when necessary

[Ron Buckton (rbuckton)]

Andrew Casey (@amcasey) recently discovered that the Web Performance API in NodeJS adds quite a bit more overhead to the total compile time than what we were doing before. This changes our use of native performance APIs in the following ways:

• Revert back to calculating our own mark times and measure durations in ts.performance rather than using PerformanceObserver.
• Conditionally call the native mark and measure when any of the following are true:
  • We are running in the browser (since a debugger can start a cpu profile at any time)
  • We are running in NodeJS with Node's --cpu-prof option specified (to capture user timing events in the profile)
  • We are running in NodeJS with V8's --prof option specified (to capture user timing events in the profile)
  • We are running in NodeJS with our own --generateCpuProfile option specified (to capture user timing events in the profile)
  • We are running in NodeJS and sys.debugMode is true (since a debugger can start a cpu profile at any time)

If none of those cases are true, we will not call the native mark and measure methods. I've filed nodejs/diagnostics#464 to track the performance issue we encountered in NodeJS, and it sounds like they plan to have a patch available in the near future that improves their implementation.

[Andrew Casey (amcasey)]

The original repro had emit time tripled from a TS 4.0 baseline. With this change, they're back within 10% of TS 4.0. Since the repro is noisy and the number of other changes is huge, I'm satisfied.

[Andrew Casey (amcasey)]

TypeScript Bot (@typescript-bot) perf test this

[TypeScript Bot (typescript-bot)]

Heya Andrew Casey (@amcasey), I've started to run the perf test suite on this PR at a20bfa2. You can monitor the build here.

Update: The results are in!

[TypeScript Bot (typescript-bot)]

Andrew Casey (@amcasey)
The results of the perf run you requested are in!

Here they are:

Comparison Report - master..42586

Metric	master	42586	Delta	Best	Worst
Angular - node (v10.16.3, x64)
Memory used	346,510k (± 0.02%)	346,430k (± 0.01%)	-80k (- 0.02%)	346,337k	346,561k
Parse Time	1.97s (± 0.31%)	1.92s (± 0.54%)	-0.05s (- 2.39%)	1.90s	1.94s
Bind Time	0.82s (± 0.73%)	0.82s (± 0.41%)	-0.00s (- 0.61%)	0.81s	0.83s
Check Time	5.00s (± 0.48%)	5.00s (± 0.79%)	-0.00s (- 0.02%)	4.91s	5.09s
Emit Time	5.32s (± 0.50%)	5.24s (± 0.48%)	-0.08s (- 1.41%)	5.20s	5.33s
Total Time	13.11s (± 0.20%)	12.98s (± 0.47%)	-0.13s (- 0.98%)	12.85s	13.12s
Compiler-Unions - node (v10.16.3, x64)
Memory used	214,941k (± 0.05%)	214,895k (± 0.08%)	-45k (- 0.02%)	214,428k	215,075k
Parse Time	0.78s (± 0.61%)	0.78s (± 0.88%)	-0.00s (- 0.13%)	0.76s	0.79s
Bind Time	0.50s (± 0.68%)	0.49s (± 1.00%)	-0.00s (- 0.60%)	0.48s	0.50s
Check Time	10.66s (± 0.92%)	10.65s (± 0.36%)	-0.01s (- 0.08%)	10.58s	10.77s
Emit Time	2.32s (± 1.21%)	2.35s (± 0.75%)	+0.04s (+ 1.51%)	2.32s	2.39s
Total Time	14.25s (± 0.67%)	14.27s (± 0.27%)	+0.03s (+ 0.20%)	14.19s	14.37s
Monaco - node (v10.16.3, x64)
Memory used	355,390k (± 0.02%)	355,305k (± 0.03%)	-85k (- 0.02%)	355,061k	355,598k
Parse Time	1.59s (± 0.61%)	1.55s (± 0.34%)	-0.04s (- 2.52%)	1.54s	1.56s
Bind Time	0.73s (± 1.14%)	0.72s (± 0.46%)	-0.01s (- 0.83%)	0.71s	0.73s
Check Time	5.14s (± 0.56%)	5.13s (± 0.46%)	-0.01s (- 0.23%)	5.07s	5.19s
Emit Time	2.84s (± 1.15%)	2.77s (± 0.42%)	-0.08s (- 2.67%)	2.74s	2.79s
Total Time	10.29s (± 0.55%)	10.16s (± 0.28%)	-0.13s (- 1.26%)	10.10s	10.24s
TFS - node (v10.16.3, x64)
Memory used	308,242k (± 0.02%)	308,106k (± 0.01%)	-136k (- 0.04%)	307,958k	308,176k
Parse Time	1.22s (± 0.46%)	1.21s (± 0.74%)	-0.01s (- 0.99%)	1.19s	1.23s
Bind Time	0.68s (± 0.65%)	0.68s (± 0.49%)	-0.00s (- 0.29%)	0.67s	0.69s
Check Time	4.60s (± 0.52%)	4.60s (± 0.48%)	0.00s ( 0.00%)	4.56s	4.65s
Emit Time	2.94s (± 1.13%)	2.92s (± 1.32%)	-0.02s (- 0.85%)	2.85s	3.00s
Total Time	9.44s (± 0.60%)	9.41s (± 0.52%)	-0.03s (- 0.37%)	9.32s	9.55s
material-ui - node (v10.16.3, x64)
Memory used	496,244k (± 0.01%)	496,237k (± 0.01%)	-7k (- 0.00%)	496,062k	496,375k
Parse Time	2.05s (± 0.78%)	1.98s (± 0.37%)	🟩-0.07s (- 3.56%)	1.96s	1.99s
Bind Time	0.66s (± 1.32%)	0.65s (± 0.68%)	-0.01s (- 0.91%)	0.64s	0.66s
Check Time	13.96s (± 0.64%)	14.00s (± 0.77%)	+0.04s (+ 0.28%)	13.84s	14.32s
Emit Time	0.00s (± 0.00%)	0.00s (± 0.00%)	0.00s ( NaN%)	0.00s	0.00s
Total Time	16.67s (± 0.59%)	16.64s (± 0.67%)	-0.04s (- 0.22%)	16.46s	16.95s
Angular - node (v12.1.0, x64)
Memory used	323,945k (± 0.10%)	323,990k (± 0.02%)	+45k (+ 0.01%)	323,868k	324,084k
Parse Time	1.95s (± 0.59%)	1.90s (± 0.68%)	-0.05s (- 2.77%)	1.86s	1.93s
Bind Time	0.81s (± 1.92%)	0.80s (± 0.93%)	-0.01s (- 1.23%)	0.79s	0.82s
Check Time	4.89s (± 0.56%)	4.89s (± 0.44%)	0.00s ( 0.00%)	4.84s	4.93s
Emit Time	5.49s (± 1.03%)	5.39s (± 0.51%)	-0.09s (- 1.71%)	5.35s	5.47s
Total Time	13.13s (± 0.71%)	12.98s (± 0.37%)	-0.16s (- 1.18%)	12.89s	13.11s
Compiler-Unions - node (v12.1.0, x64)
Memory used	200,282k (± 0.06%)	200,370k (± 0.06%)	+88k (+ 0.04%)	200,077k	200,623k
Parse Time	0.77s (± 0.52%)	0.77s (± 1.07%)	-0.00s (- 0.26%)	0.75s	0.78s
Bind Time	0.50s (± 0.99%)	0.50s (± 0.68%)	+0.00s (+ 0.20%)	0.49s	0.50s
Check Time	9.77s (± 0.22%)	9.78s (± 0.58%)	+0.01s (+ 0.13%)	9.68s	9.98s
Emit Time	2.34s (± 1.09%)	2.35s (± 2.25%)	+0.00s (+ 0.13%)	2.27s	2.51s
Total Time	13.38s (± 0.31%)	13.39s (± 0.62%)	+0.02s (+ 0.13%)	13.24s	13.67s
Monaco - node (v12.1.0, x64)
Memory used	337,577k (± 0.02%)	337,415k (± 0.05%)	-162k (- 0.05%)	336,774k	337,570k
Parse Time	1.57s (± 0.37%)	1.53s (± 0.69%)	-0.03s (- 2.17%)	1.52s	1.56s
Bind Time	0.71s (± 0.78%)	0.70s (± 0.63%)	-0.01s (- 0.85%)	0.69s	0.71s
Check Time	4.94s (± 0.28%)	4.93s (± 0.59%)	-0.01s (- 0.24%)	4.87s	5.00s
Emit Time	2.87s (± 0.67%)	2.85s (± 0.88%)	-0.02s (- 0.59%)	2.82s	2.94s
Total Time	10.08s (± 0.32%)	10.01s (± 0.47%)	-0.07s (- 0.68%)	9.94s	10.18s
TFS - node (v12.1.0, x64)
Memory used	292,441k (± 0.02%)	292,346k (± 0.01%)	-95k (- 0.03%)	292,263k	292,443k
Parse Time	1.24s (± 0.61%)	1.22s (± 0.60%)	-0.02s (- 1.46%)	1.20s	1.23s
Bind Time	0.65s (± 0.46%)	0.66s (± 0.76%)	+0.00s (+ 0.46%)	0.64s	0.66s
Check Time	4.51s (± 0.61%)	4.51s (± 0.50%)	0.00s ( 0.00%)	4.45s	4.56s
Emit Time	2.94s (± 0.48%)	2.91s (± 0.81%)	-0.03s (- 1.19%)	2.86s	2.97s
Total Time	9.35s (± 0.28%)	9.29s (± 0.51%)	-0.05s (- 0.57%)	9.18s	9.40s
material-ui - node (v12.1.0, x64)
Memory used	473,316k (± 0.08%)	473,346k (± 0.05%)	+30k (+ 0.01%)	472,399k	473,530k
Parse Time	2.05s (± 0.37%)	1.99s (± 0.52%)	🟩-0.06s (- 3.02%)	1.96s	2.01s
Bind Time	0.64s (± 0.76%)	0.64s (± 0.81%)	-0.00s (- 0.47%)	0.63s	0.65s
Check Time	12.64s (± 1.10%)	12.50s (± 0.42%)	-0.14s (- 1.12%)	12.37s	12.61s
Emit Time	0.00s (± 0.00%)	0.00s (± 0.00%)	0.00s ( NaN%)	0.00s	0.00s
Total Time	15.34s (± 0.90%)	15.13s (± 0.33%)	-0.21s (- 1.37%)	15.02s	15.24s
Angular - node (v14.15.1, x64)
Memory used	322,740k (± 0.01%)	322,608k (± 0.00%)	-132k (- 0.04%)	322,582k	322,644k
Parse Time	1.96s (± 0.46%)	1.91s (± 0.30%)	-0.05s (- 2.80%)	1.90s	1.92s
Bind Time	0.86s (± 0.46%)	0.85s (± 0.73%)	-0.01s (- 1.39%)	0.84s	0.87s
Check Time	4.90s (± 0.45%)	4.89s (± 0.53%)	-0.00s (- 0.08%)	4.84s	4.95s
Emit Time	5.54s (± 0.41%)	5.47s (± 0.42%)	-0.06s (- 1.17%)	5.39s	5.50s
Total Time	13.26s (± 0.35%)	13.12s (± 0.28%)	-0.14s (- 1.06%)	13.01s	13.19s
Compiler-Unions - node (v14.15.1, x64)
Memory used	201,492k (± 0.59%)	201,488k (± 0.59%)	-5k (- 0.00%)	199,638k	203,496k
Parse Time	0.80s (± 0.73%)	0.79s (± 0.63%)	-0.00s (- 0.25%)	0.79s	0.81s
Bind Time	0.53s (± 0.76%)	0.53s (± 0.84%)	-0.00s (- 0.19%)	0.52s	0.54s
Check Time	9.81s (± 0.34%)	9.79s (± 0.42%)	-0.02s (- 0.16%)	9.70s	9.90s
Emit Time	2.34s (± 1.33%)	2.35s (± 1.29%)	+0.01s (+ 0.34%)	2.29s	2.45s
Total Time	13.47s (± 0.32%)	13.46s (± 0.43%)	-0.01s (- 0.11%)	13.34s	13.62s
Monaco - node (v14.15.1, x64)
Memory used	336,828k (± 0.01%)	336,765k (± 0.01%)	-63k (- 0.02%)	336,724k	336,868k
Parse Time	1.61s (± 0.79%)	1.57s (± 0.80%)	-0.05s (- 2.92%)	1.53s	1.60s
Bind Time	0.74s (± 0.51%)	0.73s (± 0.31%)	-0.01s (- 0.82%)	0.72s	0.73s
Check Time	4.84s (± 0.44%)	4.85s (± 0.27%)	+0.01s (+ 0.17%)	4.82s	4.88s
Emit Time	2.93s (± 0.60%)	2.89s (± 0.60%)	-0.04s (- 1.23%)	2.85s	2.92s
Total Time	10.11s (± 0.33%)	10.03s (± 0.29%)	-0.08s (- 0.80%)	9.96s	10.09s
TFS - node (v14.15.1, x64)
Memory used	291,592k (± 0.00%)	291,568k (± 0.00%)	-24k (- 0.01%)	291,543k	291,593k
Parse Time	1.28s (± 1.36%)	1.24s (± 0.80%)	-0.04s (- 2.97%)	1.22s	1.26s
Bind Time	0.69s (± 0.64%)	0.69s (± 0.48%)	-0.00s (- 0.29%)	0.68s	0.70s
Check Time	4.49s (± 0.41%)	4.49s (± 0.30%)	-0.00s (- 0.11%)	4.46s	4.53s
Emit Time	3.05s (± 0.65%)	3.07s (± 0.75%)	+0.02s (+ 0.56%)	3.01s	3.11s
Total Time	9.52s (± 0.30%)	9.49s (± 0.27%)	-0.03s (- 0.34%)	9.41s	9.52s
material-ui - node (v14.15.1, x64)
Memory used	471,867k (± 0.06%)	472,219k (± 0.00%)	+353k (+ 0.07%)	472,169k	472,251k
Parse Time	2.13s (± 0.50%)	2.06s (± 0.52%)	🟩-0.07s (- 3.15%)	2.03s	2.08s
Bind Time	0.70s (± 0.57%)	0.70s (± 0.53%)	-0.00s (- 0.57%)	0.69s	0.70s
Check Time	12.71s (± 0.74%)	12.59s (± 0.55%)	-0.12s (- 0.91%)	12.45s	12.76s
Emit Time	0.00s (± 0.00%)	0.00s (± 0.00%)	0.00s ( NaN%)	0.00s	0.00s
Total Time	15.54s (± 0.62%)	15.35s (± 0.46%)	-0.19s (- 1.21%)	15.22s	15.54s

System

Machine Name	ts-ci-ubuntu
Platform	linux 4.4.0-198-generic
Architecture	x64
Available Memory	16 GB
Available Memory	8 GB
CPUs	4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz

Hosts

• node (v10.16.3, x64)
• node (v12.1.0, x64)
• node (v14.15.1, x64)

Scenarios

• Angular - node (v10.16.3, x64)
• Angular - node (v12.1.0, x64)
• Angular - node (v14.15.1, x64)
• Compiler-Unions - node (v10.16.3, x64)
• Compiler-Unions - node (v12.1.0, x64)
• Compiler-Unions - node (v14.15.1, x64)
• Monaco - node (v10.16.3, x64)
• Monaco - node (v12.1.0, x64)
• Monaco - node (v14.15.1, x64)
• TFS - node (v10.16.3, x64)
• TFS - node (v12.1.0, x64)
• TFS - node (v14.15.1, x64)
• material-ui - node (v10.16.3, x64)
• material-ui - node (v12.1.0, x64)
• material-ui - node (v14.15.1, x64)

Benchmark	Name	Iterations
Current	42586	10
Baseline	master	10

[Ron Buckton (rbuckton)]

Perf numbers look good to me

[Andrew Casey (amcasey)]

It looks like nothing in the perf suite does a meaningful amount of emit, which is probably why we missed the regression. Should we add something? (Unfortunately, I don't have an example in mind.)