Skip to content

Feature: add HARP/ExApps support #776

Open

Description

@Kakise

Description of the change

I tried to add a StatefulSet with HARP to run exapps on kubernetes.
My tests were successful.

StatefulSet code:

apiVersion: v1
kind: ConfigMap
metadata:
  name: docker-daemon-config
data:
  daemon.json: |
    {
      "memory": "512m"
    }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: clouduvet
  name: clouduvet-harp
spec:
  serviceName: harp
  replicas: 1
  selector:
    matchLabels:
      app: clouduvet
  template:
    metadata:
      labels:
        app: clouduvet
    spec:
      volumes:
        - name: docker-config
          configMap:
            name: docker-daemon-config
      restartPolicy: Always
      securityContext:
        fsGroup: 1000
      containers:
        - name: appapi-harp
          image: ghcr.io/nextcloud/nextcloud-appapi-harp:release
          imagePullPolicy: Always
          env:
            - name: DOCKER_HOST
              value: unix:///run/user/1000/docker.sock
            - name: HP_SHARED_KEY
              valueFrom:
                secretKeyRef:
                  name: harp-shared-key
                  key: shared-key
            - name: NC_INSTANCE_URL
              value: "http://nextcloud.nextcloud.svc.cluster.local"
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "250m"
              memory: "512Mi"
          #securityContext:
          #  privileged: true
#---
#apiVersion: v1
#kind: Pod
#metadata:
#  name: appapi-harp
#  labels:
#    app: appapi-harp
#spec:
#  restartPolicy: Always
#  hostNetwork: false
#  containers:
#    - name: appapi-harp
#      image: ghcr.io/nextcloud/nextcloud-appapi-harp:release
#      imagePullPolicy: IfNotPresent
#      env:
#        - name: HP_SHARED_KEY
#          valueFrom:
#            secretKeyRef:
#              name: harp-shared-key
#              key: shared-key
#        - name: NC_INSTANCE_URL
#          value: "http://nextcloud.nextcloud.svc.cluster.local"
#      ports:
#        - containerPort: 8780
#        - containerPort: 8782
#      volumeMounts:
#        - name: docker-sock
#          mountPath: /var/run/docker.sock
#        - name: certs
#          mountPath: /certs
#  volumes:
#    - name: docker-sock
#      hostPath:
#        path: /var/run/docker.sock
#        type: Socket
#    - name: certs
#      hostPath:
#        path: /absolute/path/to/certs  # Change to absolute path on host
#        type: Directory

It allowed me to expose a docker rootless using harp. I put that here mostly so others may be able to reference it, feel free to use it :)
You also need to change the service and the ingress.

Benefits

ExApps support.

Possible drawbacks

The StatefulState approach is losely based on what I've done for gitlab runners on my cluster, it might not be the best approach.
I cannot guarantee that the manifest I provided can be used across every clusters.

Additional information

I am willing to create a pull request for this change. However, I'd appreciate some guidance on how to expose the docker socket of the host node in a manner that is compatible with a maximum of setups.

Activity

  1. pschichtel commented on Oct 26, 2025

    @pschichtel

    I wonder if something like kubedock would work nicely to get ExApps to work without having to mess with docker in kubernetes.

    On your proposal: From reading the docs it seems that HaRP itself is a stateless application that could be deployed as a Deployment and replicated, right? Also it doesn't seem to carry a docker daemon itself and the DOCKER_HOST variable doesn't seem to do anything.

  2. Dual-0 commented on Mar 22, 2026

    @Dual-0

    This is a great discussion. I just spent a significant amount of time successfully deploying HaRP + AppAPI on a strict Talos Kubernetes cluster (using FluxCD) and wanted to share my findings too...

    I went with a different approach using Docker-in-Docker (DinD) native sidecar. This completely avoids the security and compatibility headaches of mounting the host's /var/run/docker.sock, making it compatible with any modern Kubernetes distribution.

    Here we go with some more details:

    The Docker-in-Docker (DinD) Approach

    Instead of relying on the host, we can deploy a dedicated docker:dind container as a native Kubernetes sidecar

    • The DinD sidecar provides an isolated Docker engine just for ExApps.
    • HaRP and DinD share an emptyDir volume for the /var/run socket. HaRP talks to the sidecar via unix:///var/run/docker.sock.

    StatefulSet?

    HaRP seems stateless @pschichtel. I think a **StatefulSet with volumeClaimTemplates is necessary:

    • Many ExApps download gigabytes of Docker images. If deployed as a stateless Deployment, every Pod restart wipes the downloaded images and destroys the running ExApp containers, causing massive downtime.
    • More important, on first boot, HaRP generates self-signed CA and TLS certificates for the FRP tunnel to the ExApps. If the pod restarts and loses /certs, it generates a new CA. The surviving ExApp containers (inside the DinD volume) will try to reconnect with the old certificates, fail TLS verification and become permanently orphaned. (hit me hard serval times)

    Internal Heartbeat

    A major issue for me while running HaRP in Kubernetes was the Nextcloud heartbeat. Nextcloud checks if ExApps are alive by hitting /exapps/<app>/heartbeat. If you use your public URL, traffic leaves the cluster... If you use the internal http://nextcloud.nextcloud.svc... URL, the Nextcloud Nginx sidecar throws a 404 because it doesn't know what /exapps/ is.
    I fixed this with a configuration change in the Nextcloud Helm config (via serverBlockCustom in the Helm chart) to route /exapps/ internally directly to HaRP, while spoofing the public Host header.


    The Complete Setup

    1. The Nextcloud Helm Chart Nginx Patch

        nginx:
          config:
            serverBlockCustom: |
              # Route internal heartbeats directly to HaRP without leaving the cluster
              location /exapps/ {
                proxy_pass http://nextcloud-harp.nextcloud-harp.svc.cluster.local:8780;
                
                # Force the public Host header so HaRP recognizes the route
                proxy_set_header Host "cloud.yourdomain.com";
                
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_read_timeout 3600s;
                proxy_send_timeout 3600s;
                proxy_buffering off;
              }

    2. The HaRP + DinD StatefulSet

    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: nextcloud-harp
      labels:
        app: nextcloud-harp
    spec:
      serviceName: nextcloud-harp
      replicas: 1
      selector:
        matchLabels:
          app: nextcloud-harp
      template:
        metadata:
          labels:
            app: nextcloud-harp
        spec:
          volumes:
            - name: docker-socket
              emptyDir: {}
          initContainers:
            - name: dind
              image: docker:29.3.0-dind-alpine3.23
              restartPolicy: Always
              securityContext:
                privileged: true    # Required for DinD
              command:
                - /bin/sh
                - -c
                # Note: Talos OS requires nftables for Docker bridge NAT
                - "apk add --no-cache nftables iptables && dockerd-entrypoint.sh"
              env:
                # Setting this prevents an intentional 15s startup delay in Docker 29+
                - name: DOCKER_TLS_CERTDIR
                  value: "/certs/docker-tls"
              volumeMounts:
                - name: docker-socket
                  mountPath: /var/run
                - name: docker-storage
                  mountPath: /var/lib/docker
                # Must share the same PVC as HaRP
                - name: harp-data
                  mountPath: /certs
              startupProbe:
                exec:
                  command:["docker", "info"]
                initialDelaySeconds: 5
                periodSeconds: 5
                failureThreshold: 12
    
          containers:
            - name: harp
              image: ghcr.io/nextcloud/nextcloud-appapi-harp:release
              env:
                - name: HP_SHARED_KEY
                  valueFrom:
                    secretKeyRef:
                      name: nextcloud-harp-secret
                      key: harp-shared-key
                # Use the internal Nextcloud service
                - name: NC_INSTANCE_URL
                  value: "http://nextcloud.nextcloud.svc.cluster.local:8080"
                - name: HP_TRUSTED_PROXY_IPS
                  value: "10.0.0.0/8"
                - name: DOCKER_HOST
                  value: "unix:///var/run/docker.sock"
              ports:
                - name: exapps-http
                  containerPort: 8780
                - name: frp-server
                  containerPort: 8782
              volumeMounts:
                - name: docker-socket
                  mountPath: /var/run
                - name: harp-data
                  mountPath: /certs
    
      volumeClaimTemplates:
        - metadata:
            name: docker-storage
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 10Gi # Space for large AI ExApp images
        - metadata:
            name: harp-data
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 32Mi # Just for FRP certificates

    (You still need an Ingress resource to route yourdomain.com/exapps/ to the 8780 port of this service.

    Feedback is highly recommend :)

  3. pschichtel commented on Mar 26, 2026

    @pschichtel

    dind still feels like a hack. If I finally end up needing ExApps in my setup I will definitely attempt this with kubedock. I think your STS is a good basis even for that, "just" change it to a Deployment and replace dind with kubedock. Image storage (and caching) would be handled by kubernetes/cri, complete with automatic garbage collection. Also isolating ExApps using NetworkPolicy would be easily possible.

  4. alexbarcelo commented on Sep 28, 2026

    @alexbarcelo

    I was browsing around ExApps and realized that, recently, https://github.com/nextcloud/HaRP includes information about Kubernetes.

    Does this change things? Because, with my shallow understanding, @Kakise proposal relied on some hackish (IMHO, although it was the only sane option, if I understood it correctly) dind. But with last HaRP versions, this helm could leverage that and configure HaRP while setting the appropriate flags.

  5. pschichtel commented on Sep 28, 2026

    @pschichtel

    Oh, yes native kubernetes support makes any hacks using dind or kubedock obsolete.

  6. Dual-0 commented on Sep 30, 2026

    @Dual-0

    @alexbarcelo yes it changes everything. I recently migrated my setup and can share two things relevant to this issue:


    Since HaRP v0.4.0 there is a native Kubernetes backend: instead of talking to a Docker engine (in my example DinD), HaRP manages ExApps directly through the Kubernetes API as native Deployment, Service and PersistentVolumeClaim:

    my config snippet:

    env:
      - name: HP_K8S_ENABLED
        value: "true"
      - name: HP_K8S_NAMESPACE
        value: "nextcloud-exapps"
      - name: HP_K8S_STORAGE_CLASS
        value: "longhorn"
      - name: HP_K8S_DEFAULT_STORAGE_SIZE
        value: "1Gi"

    The only RBAC it needs is a namespaced Role in the ExApp namespace, exact permission set documented in the HaRP README wich bound to a ServiceAccount in the HaRP pod's namespace (no cluster-scoped). Register the daemon afterwards with:

    php occ app_api:daemon:register k8s_harp "Kubernetes HaRP" kubernetes-install \
      http nextcloud-harp:8780 https://cloud.example.com --harp \
      --harp_shared_key "<key>" --k8s --set-default

    Result: No Docker socket, no privileged: true, no rootless-Docker-in-Docker/Cluster. ExApps become visible, policy-able Kubernetes workloads. I can share my full production manifests (Deployment + RBAC + Gateway API HTTPRoute) as a reference if that helps.

  7. pschichtel commented on Sep 30, 2026

    @pschichtel

    @Dual-0 sounds great, I think a pr would be a nice way to share this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions