13 111 files
jobSize |
wall |
| 16 |
218 s |
| 150 |
86 s |
| 300 |
75 s |
279 files (a single module of the same project)
jobSize |
chunks |
workers |
wall |
| 16 |
18 |
14 |
14.1 s |
| 50 |
6 |
6 |
8.9 s |
| 100 |
3 |
3 |
9.5 s |
| 150 |
2 |
2 |
10.9 s |
| 300 |
1 |
1 |
12.2 s |
On 279 files even a single worker - jobSize: 300, no parallelism at all - beats the current default running on 14, because starting 14 processes costs more than the parallelism returns.
The optimum does move with corpus size (~50 for 279 files, 300+ for 13 111), but 100 is close to both, and anything in 100-200 beats 16 everywhere I measured.
So yes, increasing default job size to 100 might be a good idea, but:
jobSize and timeoutSeconds interact. A worker has to finish a whole chunk inside the per-job timeout, and 16 -> 150 multiplies the work per chunk by ~10. With a heavy rule set and a cold cache I hit exactly that - Child process timed out after 120 seconds, exit 1 - at jobSize: 150, where 16 completed. If the default goes up, the default timeout likely needs to go up with it, otherwise the failure mode moves from "slow" to "red build".
Originally posted by @webard in #8489 (comment)
13 111 files
jobSize279 files (a single module of the same project)
jobSizeOn 279 files even a single worker -
jobSize: 300, no parallelism at all - beats the current default running on 14, because starting 14 processes costs more than the parallelism returns.The optimum does move with corpus size (~50 for 279 files, 300+ for 13 111), but
100is close to both, and anything in 100-200 beats16everywhere I measured.So yes, increasing default job size to 100 might be a good idea, but:
jobSizeandtimeoutSecondsinteract. A worker has to finish a whole chunk inside the per-job timeout, and 16 -> 150 multiplies the work per chunk by ~10. With a heavy rule set and a cold cache I hit exactly that -Child process timed out after 120 seconds, exit 1 - atjobSize: 150, where16completed. If the default goes up, the default timeout likely needs to go up with it, otherwise the failure mode moves from "slow" to "red build".Originally posted by @webard in #8489 (comment)