Repository navigation
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Description of the change
I tried to add a StatefulSet with HARP to run exapps on kubernetes.
My tests were successful.
StatefulSet code:
apiVersion: v1
kind: ConfigMap
metadata:
name: docker-daemon-config
data:
daemon.json: |
{
"memory": "512m"
}
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app: clouduvet
name: clouduvet-harp
spec:
serviceName: harp
replicas: 1
selector:
matchLabels:
app: clouduvet
template:
metadata:
labels:
app: clouduvet
spec:
volumes:
- name: docker-config
configMap:
name: docker-daemon-config
restartPolicy: Always
securityContext:
fsGroup: 1000
containers:
- name: appapi-harp
image: ghcr.io/nextcloud/nextcloud-appapi-harp:release
imagePullPolicy: Always
env:
- name: DOCKER_HOST
value: unix:///run/user/1000/docker.sock
- name: HP_SHARED_KEY
valueFrom:
secretKeyRef:
name: harp-shared-key
key: shared-key
- name: NC_INSTANCE_URL
value: "http://nextcloud.nextcloud.svc.cluster.local"
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "250m"
memory: "512Mi"
#securityContext:
# privileged: true
#---
#apiVersion: v1
#kind: Pod
#metadata:
# name: appapi-harp
# labels:
# app: appapi-harp
#spec:
# restartPolicy: Always
# hostNetwork: false
# containers:
# - name: appapi-harp
# image: ghcr.io/nextcloud/nextcloud-appapi-harp:release
# imagePullPolicy: IfNotPresent
# env:
# - name: HP_SHARED_KEY
# valueFrom:
# secretKeyRef:
# name: harp-shared-key
# key: shared-key
# - name: NC_INSTANCE_URL
# value: "http://nextcloud.nextcloud.svc.cluster.local"
# ports:
# - containerPort: 8780
# - containerPort: 8782
# volumeMounts:
# - name: docker-sock
# mountPath: /var/run/docker.sock
# - name: certs
# mountPath: /certs
# volumes:
# - name: docker-sock
# hostPath:
# path: /var/run/docker.sock
# type: Socket
# - name: certs
# hostPath:
# path: /absolute/path/to/certs # Change to absolute path on host
# type: Directory
It allowed me to expose a docker rootless using harp. I put that here mostly so others may be able to reference it, feel free to use it :)
You also need to change the service and the ingress.
Benefits
ExApps support.
Possible drawbacks
The StatefulState approach is losely based on what I've done for gitlab runners on my cluster, it might not be the best approach.
I cannot guarantee that the manifest I provided can be used across every clusters.
Additional information
I am willing to create a pull request for this change. However, I'd appreciate some guidance on how to expose the docker socket of the host node in a manner that is compatible with a maximum of setups.
Reacted by Eric Daly, Adrian Chuck, Térence Chateigné and Oli Oeli
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
pschichtel commentedon Oct 26, 2025
I wonder if something like kubedock would work nicely to get ExApps to work without having to mess with docker in kubernetes.
On your proposal: From reading the docs it seems that HaRP itself is a stateless application that could be deployed as a Deployment and replicated, right? Also it doesn't seem to carry a docker daemon itself and the DOCKER_HOST variable doesn't seem to do anything.
Dual-0 commentedon Mar 22, 2026
This is a great discussion. I just spent a significant amount of time successfully deploying HaRP + AppAPI on a strict Talos Kubernetes cluster (using FluxCD) and wanted to share my findings too...
I went with a different approach using Docker-in-Docker (DinD) native sidecar. This completely avoids the security and compatibility headaches of mounting the host's
/var/run/docker.sock, making it compatible with any modern Kubernetes distribution.Here we go with some more details:
The Docker-in-Docker (DinD) Approach
Instead of relying on the host, we can deploy a dedicated
docker:dindcontainer as a native Kubernetes sidecaremptyDirvolume for the/var/runsocket. HaRP talks to the sidecar viaunix:///var/run/docker.sock.StatefulSet?
HaRP seems stateless @pschichtel. I think a **StatefulSet with
volumeClaimTemplatesis necessary:/certs, it generates a new CA. The surviving ExApp containers (inside the DinD volume) will try to reconnect with the old certificates, fail TLS verification and become permanently orphaned. (hit me hard serval times)Internal Heartbeat
A major issue for me while running HaRP in Kubernetes was the Nextcloud heartbeat. Nextcloud checks if ExApps are alive by hitting
/exapps/<app>/heartbeat. If you use your public URL, traffic leaves the cluster... If you use the internalhttp://nextcloud.nextcloud.svc...URL, the Nextcloud Nginx sidecar throws a404because it doesn't know what/exapps/is.I fixed this with a configuration change in the Nextcloud Helm config (via
serverBlockCustomin the Helm chart) to route/exapps/internally directly to HaRP, while spoofing the publicHostheader.The Complete Setup
1. The Nextcloud Helm Chart Nginx Patch
2. The HaRP + DinD StatefulSet
(You still need an Ingress resource to route
yourdomain.com/exapps/to the8780port of this service.Feedback is highly recommend :)
pschichtel commentedon Mar 26, 2026
dind still feels like a hack. If I finally end up needing ExApps in my setup I will definitely attempt this with kubedock. I think your STS is a good basis even for that, "just" change it to a Deployment and replace dind with kubedock. Image storage (and caching) would be handled by kubernetes/cri, complete with automatic garbage collection. Also isolating ExApps using NetworkPolicy would be easily possible.
alexbarcelo commentedon Sep 28, 2026
I was browsing around ExApps and realized that, recently, https://github.com/nextcloud/HaRP includes information about Kubernetes.
Does this change things? Because, with my shallow understanding, @Kakise proposal relied on some hackish (IMHO, although it was the only sane option, if I understood it correctly) dind. But with last HaRP versions, this helm could leverage that and configure HaRP while setting the appropriate flags.
pschichtel commentedon Sep 28, 2026
Oh, yes native kubernetes support makes any hacks using dind or kubedock obsolete.
Dual-0 commentedon Sep 30, 2026
@alexbarcelo yes it changes everything. I recently migrated my setup and can share two things relevant to this issue:
Since HaRP v0.4.0 there is a native Kubernetes backend: instead of talking to a Docker engine (in my example DinD), HaRP manages ExApps directly through the Kubernetes API as native
Deployment,ServiceandPersistentVolumeClaim:my config snippet:
The only RBAC it needs is a namespaced
Rolein the ExApp namespace, exact permission set documented in the HaRP README wich bound to a ServiceAccount in the HaRP pod's namespace (no cluster-scoped). Register the daemon afterwards with:Result: No Docker socket, no
privileged: true, no rootless-Docker-in-Docker/Cluster. ExApps become visible, policy-able Kubernetes workloads. I can share my full production manifests (Deployment + RBAC + Gateway API HTTPRoute) as a reference if that helps.pschichtel commentedon Sep 30, 2026
@Dual-0 sounds great, I think a pr would be a nice way to share this.