Add configuration for setting the degree of compilation parallelism. #51

Activity

1. tbenthompson commented on Dec 18, 2020

   

   tbenthompson

   on Dec 18, 2020

   Last edited by tbenthompson

   Owner

   More actions

   Currently, parallelism is off by default. If you turn parallelism on, the number of cores is equal to multiprocessing.cpu_count().

   This would be quite easy to modify. I have two proposals for how we could do this:

   • Change the parallelism flag to instead take a parameter indicating the number of threads to use. Ideally, this would be done in a way that maintains backward compatibility with the current True/False behavior. So, perhaps, True == multiprocessing.cpu_count, False = 1, and then any integer uses that number of threads.
   • A slightly different, but similar proposal would be to just add another configuration field that contains the number of threads to use.

   Some locations in the code that would be useful for this:
   

   cppimport/cppimport/templating.py

   Line 46 in 00c40d9

   parallel = False,

   
   

   cppimport/cppimport/build_module.py

   Line 57 in 00c40d9

   def parallel_compile(self, sources, output_dir = None, macros = None,

   
   

   cppimport/cppimport/build_module.py

   Line 150 in 00c40d9

   parallelize = cfg.get('parallel') and py33orgreater

   I would be excited to accept a pull request if you have the time to implement this!

   👍React with 👍1Reacted by Yuan Yao
2. yaoyuannnn

   closed this as completedon Jan 7, 2021
3. tbenthompson

   reopened this on Jan 7, 2021
4. tbenthompson

   changed the title [-]Is there any option to limit the threads cppimport uses for compilation[/-] [+]Add configuration for setting the degree of compilation parallelism.[/+] on Jan 7, 2021
5. tbenthompson commented on Jan 7, 2021

   

   tbenthompson

   on Jan 7, 2021

   Owner

   More actions

   Reopened because this is something I want to do eventually.