You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Freeze the resolved interpreter for child processes #796
When PythonCall resolves its interpreter via CondaPkg, each child process that loads
PythonCall re-resolves the same CondaPkg environment from scratch. Spawn several at once —
e.g. Distributed workers for parallel data loading (MLUtils DataLoader(...; num_workers=N)
over a PythonCall-backed dataset) — and they serialize on CondaPkg's lock, spamming CondaPkg: Waiting for lock to be freed and redoing work the parent already did.
PythonCall already has the fix, but gates it behind CI=true and flags it as a hack
(src/C/context.jl, v0.9.35):
# HACK: If we are using CondaPkg, prevent child processes from using it by explicitly# setting the executable. ...# A better solution may be to use some environment variable to "freeze" CondaPkg in# child processes.# Only done when CI=true since it's a hack.if (get(ENV, "CI", "false") =="true") && (CTX.which ===:CondaPkg)
ENV["JULIA_PYTHONCALL_EXE"] = CTX.exe_path::Stringend
Since JULIA_PYTHONCALL_EXE is exactly the "freeze" env var the comment asks for (child
processes inherit ENV, and getpref_exe reads it), the same line solves the general case.
Proposal
Set JULIA_PYTHONCALL_EXE whenever CTX.which === :CondaPkg, not only under CI — ideally
behind a preference (e.g. freeze_child_processes, default true) so it can be disabled. get! already leaves a user-set value untouched.
Workaround
A downstream package can do this itself after init:
if PythonCall.C.CTX.which ===:CondaPkgget!(ENV, "JULIA_PYTHONCALL_EXE", PythonCall.python_executable_path()::String)
end
Activity