futures.as_completed leaks memory

[visjedi]

I am using your concurrent.futures package. I schedule about 200,000 tasks and execute them on multiple cores using ProcessPoolExecutor. The memory usage explodes because as_completed does not release references to the futures as the tasks are completed.
In my use case, the individual tasks themselves handle images and matrices. So I dont know how I can send you a reproducible minimal example.
However, This seems to be a known fixed bug : https://bugs.python.org/issue27144
Any thoughts on whether this has been backported in this package?

[agronholm]

Process pooling on Python 2.7 has some reliability issues which cannot be fixed in the scope of this library. I strongly suggest that you migrate to Python 3. That aside, I don't recall this specific problem being addressed. If you want this to happen, it would be faster if you opened a PR based on upstream changes.

[cowbert]

This wasn't fixed in PR #66 ? Does the extra array instantiation in the re-implementation of yield from continue to cause increased memory fragmentation (we're creating a 1 element array containing the future then popping it in the yield to free the future ref, but the array ref still exists)?

[agronholm]

@visjedi Can you confirm that this happens on the latest release (3.2.0)?

[mowshon]

If you use the standard module “concurrent.futures” and want to simultaneously process several million data, then a queue of workers will take up all the free memory.

You can use bounded-pool-executor.
https://github.com/mowshon/bounded_pool_executor

[agronholm]

@visjedi since the changes have been backported from the issue you linked to, is there anything else you would like me to do, or can I close this issue?