[Chocksy]

lean cloud push fails on big projects. PushManager._push_metadata sends every source file in one form-encoded projects/update POST, and the nginx in front of www.quantconnect.com/api/v2 rejects any request body over 1 MiB with a 413. The CLI logs Cannot push ... and still exits 0, so scripts don't notice the cloud kept the old code.

We hit it with a 103-file C# project: 925 KB of source, 1,163,065 bytes once form-encoded. I binary-searched the cap on a throwaway project and it's exactly 1,048,576 bytes; 1,048,577 gets the nginx 413 Request Entity Too Large page on both projects/update and files/update. The docs allow up to 250 files of 256 KB each per project, so a project can sit well inside its quota and still not fit in one request.

Related: #573 and #597 (which made the 413 message clearer).

Change

When projects/update returns 413, PushManager now:

1. reads the cloud files with files/read,
2. deletes cloud files that no longer exist locally (library files are skipped), then creates new files and updates changed ones, with files/delete, files/create and files/update,
3. sends the same projects/update call without files last (name, description, parameters, libraries, encryption key).

Every one of those calls sends codeSourceId the same way projects/update does: "cli" normally, nothing with --force. FileClient.create, update and delete got an optional code_source_id for this. The API only checks the collaboration lock on files/* when codeSourceId is sent, and it doesn't check a projects/update without files, so the files go first: if another session holds the lock, the first file call is refused and nothing is written.

Deletes go first among the file calls, so a rename in a project at its file quota doesn't go over it. Unchanged files are skipped (compared with rstrip(), so leading whitespace still counts). Small projects keep the single fast request; only a 413 takes this path. Any other error is re-raised as before.

I went with "fall back on 413" instead of checking the size up front, so the CLI doesn't hardcode the server's nginx limit.

Partial failures

The per-file path isn't atomic the way projects/update is, so failures are reported by how far the push got:

• First request fails: nothing was written. The error says nothing was pushed and keeps the server's text, so the --force hint still shows on a lock.
• A later request fails: a new PartialPushError says the project was only partly pushed and names the failing request, for example creating 'research.ipynb' failed. If only the final metadata request fails, it says all files were pushed but the project details weren't.
• push_projects warns per project and moves on, as before, then raises PartialPushError at the end if any project was only partly pushed. So lean cloud backtest/optimize/live --push stop instead of running half-pushed code, and lean cloud push exits 1 in that case.

Tests

• test_push_projects_pushes_files_one_by_one_when_the_project_is_too_large (with and without --force): checks the create, update and delete calls, code_source_id on each, the exact call order (deletes, file writes, metadata last), and that trailing whitespace is ignored but leading whitespace isn't.
• test_push_projects_leaves_the_project_unchanged_when_the_first_file_push_fails: a lock refusal on the first call; no metadata update, plain error, --force hint.
• test_push_projects_raises_when_a_file_push_fails_after_other_files_were_pushed and test_push_projects_raises_when_only_the_project_details_fail_after_the_files_were_pushed: the partial-push messages and the raise.
• test_push_projects_does_not_push_files_one_by_one_on_other_errors: a 400 doesn't trigger the fallback.
• test_get_cloud_project_does_not_return_a_partly_pushed_project: --push commands stop on a partial push.
• tests/components/api/test_file_client.py: codeSourceId is sent only when given.

python -m pytest: 2461 passed, 412 skipped (master: 2448 passed, 412 skipped). static_analysis.py passes.

Live runs (throwaway project, the 103-file project)

Lock test, with the web IDE holding the edit lock and a changed Main.cs and description:

• without --force: refused on the first file call (updating 'Main.cs' failed, nothing was pushed: The project could not be saved as another collaborator controls the write permission...). The cloud description and Main.cs were unchanged afterwards.
• with --force: pushed; both updated in the cloud.

From the first version of this PR (same per-file sync, before the order and codeSourceId changes):

• first push: 413, then 103 files/create and 1 files/delete (a stale file); afterwards all 103 cloud files were byte-identical to the local ones. It took about 2.5 minutes with --verbose on.
• second push, no changes: 0 file requests, 5 seconds.
• after editing Main.cs: 1 files/update, nothing else.

Notes

• If a large project's only change is metadata, the fallback sends just the metadata-only projects/update, which the server doesn't lock-check. The single-request path always includes files, so it is checked there. I think this belongs on the server side (check the lock whenever codeSourceId is sent).
• Encryption: the CLI encrypts with the key's fixed IV, so unchanged files in an encrypted project aren't re-uploaded. The key id now goes in the last request; files/create accepts ciphertext in a project that isn't marked encrypted yet. I haven't run a full --encrypt push of a large project end to end.
• Separate from this PR: a failed push (other than a partial one) still only logs a warning and lean cloud push exits 0. Happy to open an issue for that.

[quantconnect-kronos-reviewer]

Thanks for the clear write-up and the measurements. Falling back only on a 413 keeps small projects on the one-request path, and the tests show the fallback is reached and that other errors still re-raise. One blocking problem, and two smaller points.

Blocking: the fallback skips the collaborator write-permission check. This answers the open question in your notes. I checked how the API handles this. The collaboration-lock check (the one behind You do not have write permission ... push with --force) is tied to codeSourceId. projects/update only runs it when files is in the request, and files/create, files/update and files/delete only run it when codeSourceId is sent.

On the fallback path, the metadata-only projects/update has no files, so the check doesn't run. FileClient never sends codeSourceId, so the per-file calls skip it too. As a result, a plain lean cloud push (without --force) of a large project silently overwrites and deletes files while another collaborator holds the write lock, which today's single-request push refuses to do. That can lose someone else's code.

Please add an optional code_source_id to FileClient.create, update and delete, sent as codeSourceId the way ProjectClient.update does. Pass update_args.get("code_source_id") through _push_files_one_by_one so --force keeps working as it does now. Please also extend the 413 test to assert that the per-file calls receive code_source_id="cli" without --force and None with it.

Smaller points (not blocking on their own):

• Partial failure leaves the cloud worse off than before. Deletes run first, so if a later files/create or files/update fails, the deleted files are already gone. For example, a single file over the 1 MiB body limit still returns 413 per file, and so do validation errors. Before this change, the cloud project was left untouched. You note it isn't atomic, but please at least make the warning say the project was only partly pushed, so users know to fix the problem and push again. You could also make the delete-first order apply only when the project is near its file quota, but that's your call.
• I checked encryption: the CLI encrypts with the key's fixed IV, so the ciphertext is the same on every push. Unchanged files in an encrypted project won't be re-uploaded on each push, which answers your second note.

CI: every finished job passed; the last three matrix jobs were still running when I submitted this review.

Other repositories: none need a change. This is client-side only and uses existing files/* endpoints.

Risk: risk:high. This changes what lean cloud push writes to and deletes from users' cloud projects, and as it stands it can overwrite a collaborator's code. Passing codeSourceId with a test for it, plus an end-to-end push of a project over the limit that shows a lock held by another session is still respected, would be the evidence to lower it.

[quantconnect-kronos-reviewer]

This request has no files, so the collaboration-lock check that code_source_id="cli" triggers on the single-request path doesn't run here. Together with the per-file calls below, which send no codeSourceId at all, the non---force large-project push no longer respects another collaborator's write lock.

[quantconnect-kronos-reviewer]

files/create, files/update and files/delete only check the write lock when codeSourceId is sent. Please add an optional code_source_id to these FileClient methods and pass update_args.get("code_source_id") through, so --force keeps its current meaning.

[Chocksy]

Thanks, the lock point was right. It's fixed, and I tested it end to end against a real lock.

codeSourceId on the per-file calls. FileClient.create, update and delete now take an optional code_source_id and send codeSourceId only when it's set. The fallback passes update_args.get("code_source_id") through, so it's "cli" without --force and absent with it. The 413 test is parametrized over --force and checks every per-file call. A new tests/components/api/test_file_client.py checks the request parameters.

Order changed: files first, metadata last. While testing I found that a metadata-only projects/update with codeSourceId=cli goes through even while another session holds the lock (no files, no check, as you said). If it ran first, a description or parameter change would get past the lock before the first file call was refused. So it now runs last. Deletes still go first among the file calls, because of the per-project file quota.

End-to-end lock test. I used a throwaway project with the same 103-file, 1.1 MB C# project. The web IDE held the edit lock, and I changed Main.cs and the description locally:

$ lean cloud push --project lean-cli-413-lock
[1/1] Pushing 'lean-cli-413-lock'
'lean-cli-413-lock' is too large to push in one request, pushing its changed files one by one
Cannot push 'lean-cli-413-lock': updating 'Main.cs' failed, nothing was pushed: The project could not be saved as another collaborator controls the write permission. Request write permission from your collaborator #...
Please pull any required changes and push with --force

Reading the project back afterwards, the cloud description and Main.cs were unchanged. Then with --force:

$ lean cloud push --project lean-cli-413-lock --force
[1/1] Pushing 'lean-cli-413-lock'
'lean-cli-413-lock' is too large to push in one request, pushing its changed files one by one
Successfully updated name, description, files, and libraries for 'lean-cli-413-lock'

Both were updated in the cloud.

Partial failures.

• If the first request fails, nothing was written. The error says "nothing was pushed" and keeps the server's text, so the --force hint still shows.
• If a later request fails, a new PartialPushError says the project was only partly pushed and names the request that failed (for example creating 'research.ipynb' failed). If only the final metadata request fails, it says all files went up but the project details didn't.
• push_projects still warns per project and moves on to the next one, then raises PartialPushError at the end. Without that, lean cloud backtest/optimize/live --push would carry on and run half-pushed code. Now they stop, and lean cloud push exits 1 in that case.

One more fix. The changed-file check compares with rstrip() now instead of strip(), so a change in leading whitespace (Python indentation) isn't skipped.

Encryption. Thanks for checking the fixed IV. Since the key id now goes in the last request, I checked that files/create accepts AES ciphertext in a project that isn't marked encrypted yet. It does, so a first --encrypt push of a large project won't be refused on the file calls. I haven't run a full --encrypt push of a large project end to end.

A question for you. When a large project's only change is metadata (description, parameters, libraries), the fallback sends just the metadata-only projects/update, and the server doesn't lock-check that. The single-request path always includes files, so it is checked there. I left the client as it is. Would it make sense for the server to check the lock on projects/update whenever codeSourceId is sent, even without files?

Tests: the full suite passes (2461 passed, 412 skipped) and static_analysis.py is clean. Each new test fails without its fix. I've updated the description to match.