Skip to content

feat(datacontroller): add Kubernetes controller for declarative data … - #5000

Open
aburan28 wants to merge 2 commits into
dragonflyoss:mainfrom
aburan28:claude/k8s-data-lifecycle-controller-1oze0n
Open

aburan28 wants to merge 2 commits into
dragonflyoss:mainfrom
aburan28:claude/k8s-data-lifecycle-controller-1oze0n

Conversation

@aburan28

@aburan28 aburan28 commented Sep 8, 2026

Copy link
Copy Markdown

…and dataset lifecycle management

Add a controller-runtime based component that manages data in the Dragonfly P2P network declaratively through two CRDs in the data.d7y.io group:

  • Dataset declares file and image sources, a distribution policy and an optional inline lifecycle. The reconciler distributes the sources with preheat jobs, refreshes them on an interval or cron schedule, verifies them with get_task jobs, expires them after a TTL (purge or retain) and purges them with delete_task jobs when they are removed or the Dataset is deleted, guarded by a finalizer. Task IDs are computed locally the same way the schedulers do, including blob digest based IDs, so verify and purge address exactly the distributed tasks. Failures are retried with backoff.
  • DataLifecyclePolicy applies a lifecycle to Datasets selected by labels, with priority based tie breaking and per-Dataset overrides.

The controller talks to the manager open API with a personal access token, reads all credentials from Secrets, records events and metrics, and reports phase, conditions and per-source state in the status.

Also add the datacontroller command, configuration, Dockerfile, deployment manifests with CRDs and RBAC, samples, build and release wiring, a design document and unit tests against a fake API server and manager.

Claude-Session: https://claude.ai/code/session_01GaXA6phNbdRVaiRGeJv1Xc

Description

Related Issue

Motivation and Context

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation Update (if none of the other choices apply)

Checklist

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.

…and dataset lifecycle management

Add a controller-runtime based component that manages data in the Dragonfly
P2P network declaratively through two CRDs in the data.d7y.io group:

- Dataset declares file and image sources, a distribution policy and an
  optional inline lifecycle. The reconciler distributes the sources with
  preheat jobs, refreshes them on an interval or cron schedule, verifies
  them with get_task jobs, expires them after a TTL (purge or retain) and
  purges them with delete_task jobs when they are removed or the Dataset is
  deleted, guarded by a finalizer. Task IDs are computed locally the same way
  the schedulers do, including blob digest based IDs, so verify and purge
  address exactly the distributed tasks. Failures are retried with backoff.
- DataLifecyclePolicy applies a lifecycle to Datasets selected by labels,
  with priority based tie breaking and per-Dataset overrides.

The controller talks to the manager open API with a personal access token,
reads all credentials from Secrets, records events and metrics, and reports
phase, conditions and per-source state in the status.

Also add the datacontroller command, configuration, Dockerfile, deployment
manifests with CRDs and RBAC, samples, build and release wiring, a design
document and unit tests against a fake API server and manager.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GaXA6phNbdRVaiRGeJv1Xc
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

This PR is stale because it has been open 60 days with no activity.

@github-actions github-actions Bot added the Stale label Nov 8, 2026
@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 54.92115% with 829 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.20%. Comparing base (5ace19f) to head (95968f1).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
...roller/apis/data/v1alpha1/zz_generated.deepcopy.go 0.00% 355 Missing ⚠️
datacontroller/controllers/dataset_controller.go 78.12% 118 Missing and 36 partials ⚠️
datacontroller/controllers/operations.go 56.33% 102 Missing and 29 partials ⚠️
datacontroller/datacontroller.go 0.00% 55 Missing ⚠️
cmd/datacontroller/cmd/root.go 0.00% 39 Missing ⚠️
datacontroller/dragonfly/client.go 70.07% 26 Missing and 12 partials ⚠️
...ller/controllers/datalifecyclepolicy_controller.go 53.33% 22 Missing and 6 partials ⚠️
datacontroller/config/config.go 85.50% 5 Missing and 5 partials ⚠️
internal/dflog/loginit.go 0.00% 9 Missing ⚠️
cmd/datacontroller/main.go 0.00% 2 Missing ⚠️
... and 4 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #5000      +/-   ##
==========================================
- Coverage   62.80%   62.20%   -0.60%     
==========================================
  Files         230      243      +13     
  Lines       22866    24705    +1839     
==========================================
+ Hits        14360    15367    +1007     
- Misses       7863     8605     +742     
- Partials      643      733      +90     
Flag Coverage Δ
unittests 62.20% <54.92%> (-0.60%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
cmd/datacontroller/main.go 0.00% <0.00%> (ø)
...er/apis/data/v1alpha1/datalifecyclepolicy_types.go 0.00% <0.00%> (ø)
datacontroller/apis/data/v1alpha1/dataset_types.go 0.00% <0.00%> (ø)
datacontroller/controllers/lifecycle.go 98.23% <98.23%> (ø)
datacontroller/metrics/metrics.go 0.00% <0.00%> (ø)
internal/dflog/loginit.go 0.00% <0.00%> (ø)
datacontroller/config/config.go 85.50% <85.50%> (ø)
...ller/controllers/datalifecyclepolicy_controller.go 53.33% <53.33%> (ø)
datacontroller/dragonfly/client.go 70.07% <70.07%> (ø)
cmd/datacontroller/cmd/root.go 0.00% <0.00%> (ø)
... and 4 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants