You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CustomContainerTrainingJob.run drops max_wait_duration=0 instead of requesting indefinite DWS wait #7067
Thanks for maintaining this library. CustomContainerTrainingJob.run(max_wait_duration=0) drops the value because the high-level training-job implementation uses a truthiness check. This prevents callers from requesting the documented indefinite wait for DWS Flex Start.
Environment details
OS type and version: macOS
Python version: 3.12
google-cloud-aiplatform versions: reproduced in 1.148.1; source inspection confirms the same behavior in 1.150.0, 1.163.0, and current main
Steps to reproduce
Create a CustomContainerTrainingJob using any valid project, staging bucket, and container image.
Run it with Flex Start and max_wait_duration=0.
Inspect the scheduling inputs produced by _prepare_training_task_inputs_and_output_dir or the resulting TrainingPipeline request.
That commit did not update the CustomContainerTrainingJob path in training_jobs.py.
Suggested fix
Use an explicit is not None check in _prepare_training_task_inputs_and_output_dir, consistent with the lower-level implementation, and add tests covering omitted, zero, and positive durations.
Stack trace
No exception is raised. The problem is a silent request-semantics change: 0 is serialized as omission.
Thus, SDK input 0 produces exactly the same final request value as omission.
Protobuf presence control
I also constructed Scheduling directly to confirm that the API type can represent explicit zero and distinguishes it from omission:
Construction
HasField(max_wait_duration)
JSON
omitted
False
strategy only
explicit 0s
True
strategy plus max_wait_duration: 0s
This rules out proto3 default-value behavior as the cause. The value is lost earlier by the truthiness conversion in training_jobs.py; explicit zero would survive serialization if that conversion preserved it.
Activity
someshfengade commentedon Aug 12, 2026
The official
CustomContainerTrainingJob.run()reference states thatmax_wait_duration=0waits indefinitely and that the omitted default is 30 minutes:https://docs.cloud.google.com/python/docs/reference/aiplatform/1.148.1/google.cloud.aiplatform.CustomContainerTrainingJob#google_cloud_aiplatform_CustomContainerTrainingJob_run
Final serialized scheduling results
max_wait_durationNoneNone0None1440014400sThus, SDK input
0produces exactly the same final request value as omission.Protobuf presence control
I also constructed
Schedulingdirectly to confirm that the API type can represent explicit zero and distinguishes it from omission:HasField(max_wait_duration)False0sTruemax_wait_duration: 0sThis rules out proto3 default-value behavior as the cause. The value is lost earlier by the truthiness conversion in
training_jobs.py; explicit zero would survive serialization if that conversion preserved it.