Repository navigation
Fork-Union for parallelization? #1556
Description
I came across this project which seems quite interesting. The author claims substantial speed-ups compared to using Rayon for parallelization. Does this seem like something worth investigating?
Activity
akern40 commentedon Dec 14, 2025
There are some really impressive numbers there! Thank you for sharing. I would welcome some experimental code, e.g., an experimental branch, that tried to look at how ForkUnion could improve the performance of our parallel iterators. I don't have the time right now to dedicate to it, but if you're interested I'd be happy to review and discuss!
nilgoyette commentedon Dec 15, 2025
I read their readme and I can't find "Microsoft" or "Windows" anywhere. Did I miss it? I understand that this feature would be locked behind a feature gate, but if the most used OS is not supported, I don't think it's worth putting much effort on this project. At least, not with the current throughput we have.
Impressive numbers indeed. I'm curious to test on my non-ndarray projects!
varchasgopalaswamy commentedon Dec 15, 2025
I can't say I've ever programmed on Windows, so I didn't think to check (and wouldn't know where to start even if I did!) I see this issue ashvardanian/ForkUnion#26 suggesting that Windows support was added.
@akern40 I don't have time myself right now, but I am adding rust based acceleration to my work projects so I could in the future. Frankly, ndarray+Rayon is already giving me ~ order of magnitude speedup over my previous code with very little pain, so I'm already quite happy!
akern40 commentedon Dec 15, 2025
Well I'm glad to hear that the codebase is delivering! Let's leave this issue open as a marker that this might be worth returning to?