Skip to content

Memcached sample is not passing in the operator-sdk scorecard bundle test #4564

Closed

Description

@camilamacedo86

Bug Report

Memcached sample is not passing in the operator-sdk scorecard bundle test.

What did you do?

$ cd testdata/go/v3/memcached-operator
$ kind create cluster
$ operator-sdk scorecard bundle --output=json 

What did you expect to see?

  • All scorecard bundle tests are called
  • The sample pass in all tests
  • The e2e test with scorecard calling all available tests

What did you see instead? Under which circumstances?

See the errors faced:

            "name": "olm-status-descriptors",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\nLoaded 1 Custom Resources from alm-examples\n",
            "state": "fail",
            "errors": [
              "memcacheds.cache.example.com does not have a status descriptor"

            "name": "olm-crds-have-resources",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\n",
            "state": "fail",
            "errors": [
              "Owned CRDs do not have resources specified"

Following the full result:

$ operator-sdk scorecard bundle --output=json 
{
  "kind": "TestList",
  "apiVersion": "scorecard.operatorframework.io/v1alpha3",
  "items": [
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-crds-have-resources"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-crds-have-resources-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-crds-have-resources",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\n",
            "state": "fail",
            "errors": [
              "Owned CRDs do not have resources specified"
            ]
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-status-descriptors"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-status-descriptors-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-status-descriptors",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\nLoaded 1 Custom Resources from alm-examples\n",
            "state": "fail",
            "errors": [
              "memcacheds.cache.example.com does not have a status descriptor"
            ]
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "basic-check-spec"
        ],
        "labels": {
          "suite": "basic",
          "test": "basic-check-spec-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "basic-check-spec",
            "state": "pass"
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-bundle-validation"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-bundle-validation-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-bundle-validation",
            "log": "time=\"2021-02-22T11:17:19Z\" level=debug msg=\"Found manifests directory\" name=bundle-test\ntime=\"2021-02-22T11:17:19Z\" level=debug msg=\"Found metadata directory\" name=bundle-test\ntime=\"2021-02-22T11:17:19Z\" level=debug msg=\"Getting mediaType info from manifests directory\" name=bundle-test\ntime=\"2021-02-22T11:17:19Z\" level=info msg=\"Found annotations file\" name=bundle-test\ntime=\"2021-02-22T11:17:19Z\" level=info msg=\"Could not find optional dependencies file\" name=bundle-test\n",
            "state": "pass"
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-spec-descriptors"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-spec-descriptors-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-spec-descriptors",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\nLoaded 1 Custom Resources from alm-examples\n",
            "state": "fail",
            "errors": [
              "size does not have a spec descriptor"
            ],
            "suggestions": [
              "Add a spec descriptor for size"
            ]
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-crds-have-validation"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-crds-have-validation-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-crds-have-validation",
            "log": "Loaded 1 Custom Resources from alm-examples\nLoaded CustomresourceDefinitions: [\u0026CustomResourceDefinition{ObjectMeta:{memcacheds.cache.example.com      0 0001-01-01 00:00:00 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[] map[controller-gen.kubebuilder.io/version:v0.4.1] [] []  []},Spec:CustomResourceDefinitionSpec{Group:cache.example.com,Names:CustomResourceDefinitionNames{Plural:memcacheds,Singular:memcached,ShortNames:[],Kind:Memcached,ListKind:MemcachedList,Categories:[],},Scope:Namespaced,Versions:[]CustomResourceDefinitionVersion{CustomResourceDefinitionVersion{Name:v1alpha1,Served:true,Storage:true,Schema:\u0026CustomResourceValidation{OpenAPIV3Schema:\u0026JSONSchemaProps{ID:,Schema:,Ref:nil,Description:Memcached is the Schema for the memcacheds API,Type:object,Format:,Title:,Default:nil,Maximum:nil,ExclusiveMaximum:false,Minimum:nil,ExclusiveMinimum:false,MaxLength:nil,MinLength:nil,Pattern:,MaxItems:nil,MinItems:nil,UniqueItems:false,MultipleOf:nil,Enum:[]JSON{},MaxProperties:nil,MinProperties:nil,Required:[],Items:nil,AllOf:[]JSONSchemaProps{},OneOf:[]JSONSchemaProps{},AnyOf:[]JSONSchemaProps{},Not:nil,Properties:map[string]JSONSchemaProps{apiVersion: {  \u003cnil\u003e APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources string   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},kind: {  \u003cnil\u003e Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds string   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},metadata: {  \u003cnil\u003e  object   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},spec: {  \u003cnil\u003e MemcachedSpec defines the desired state of Memcached object   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[foo:{  \u003cnil\u003e Foo is an example field of Memcached. Edit memcached_types.go to remove/update string   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e} size:{  \u003cnil\u003e Size defines the number of Memcached instances integer int32  nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e}] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},status: {  \u003cnil\u003e MemcachedStatus defines the observed state of Memcached object   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[nodes:{  \u003cnil\u003e Nodes store the name of the pods which are running Memcached instances array   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] \u0026JSONSchemaPropsOrArray{Schema:\u0026JSONSchemaProps{ID:,Schema:,Ref:nil,Description:,Type:string,Format:,Title:,Default:nil,Maximum:nil,ExclusiveMaximum:false,Minimum:nil,ExclusiveMinimum:false,MaxLength:nil,MinLength:nil,Pattern:,MaxItems:nil,MinItems:nil,UniqueItems:false,MultipleOf:nil,Enum:[]JSON{},MaxProperties:nil,MinProperties:nil,Required:[],Items:nil,AllOf:[]JSONSchemaProps{},OneOf:[]JSONSchemaProps{},AnyOf:[]JSONSchemaProps{},Not:nil,Properties:map[string]JSONSchemaProps{},AdditionalProperties:nil,PatternProperties:map[string]JSONSchemaProps{},Dependencies:JSONSchemaDependencies{},AdditionalItems:nil,Definitions:JSONSchemaDefinitions{},ExternalDocs:nil,Example:nil,Nullable:false,XPreserveUnknownFields:nil,XEmbeddedResource:false,XIntOrString:false,XListMapKeys:[],XListType:nil,XMapType:nil,},JSONSchemas:[]JSONSchemaProps{},} [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e}] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},},AdditionalProperties:nil,PatternProperties:map[string]JSONSchemaProps{},Dependencies:JSONSchemaDependencies{},AdditionalItems:nil,Definitions:JSONSchemaDefinitions{},ExternalDocs:nil,Example:nil,Nullable:false,XPreserveUnknownFields:nil,XEmbeddedResource:false,XIntOrString:false,XListMapKeys:[],XListType:nil,XMapType:nil,},},Subresources:\u0026CustomResourceSubresources{Status:\u0026CustomResourceSubresourceStatus{},Scale:nil,},AdditionalPrinterColumns:[]CustomResourceColumnDefinition{},Deprecated:false,DeprecationWarning:nil,},},Conversion:nil,PreserveUnknownFields:false,},Status:CustomResourceDefinitionStatus{Conditions:[]CustomResourceDefinitionCondition{},AcceptedNames:CustomResourceDefinitionNames{Plural:,Singular:,ShortNames:[],Kind:,ListKind:,Categories:[],},StoredVersions:[],},}]\n",
            "state": "pass"
          }
        ]
      }
    }
  ]
}

Enviroment:

  • operator-sdk master branch: aafa2d5

Activity

  1. estroz commented on Feb 22, 2021

    @estroz
    Contributor

    These tests can be made to pass by CSV markers added by the sample generator to Go operator samples. Another way to make changes to make these pass needs implementing for Ansible/Helm samples, ex. kustomize patch files on the CSV.

  2. added
    area/testingIssue related to testing the operator-sdk and subcomponents
    on Feb 22, 2021
  3. added this to the v1.6.0 milestone on Feb 22, 2021
  4. modified the milestones: v1.6.0, v1.7.0 on Mar 29, 2021
  5. modified the milestones: v1.7.0, Backlog on Apr 28, 2021
  6. camilamacedo86 commented on Jun 21, 2021

    @camilamacedo86
    ContributorAuthor

    I am removing myself as assigned from this one and adding help wanted and good first issue since follows what required to get done.

    To achieve that we need to improve the mocks in https://github.com/operator-framework/operator-sdk/tree/master/hack/generate/samples to add what is required to pass in the scorecard for these test and then change the tests in

    "olm-crds-have-validation": v1alpha3.FailState,
    "olm-crds-have-resources": v1alpha3.FailState,
    "olm-spec-descriptors": v1alpha3.FailState,
    "olm-status-descriptors": v1alpha3.FailState,
    , All should pass in this case.

    Also, please feel free to ping me if you need help to work on that.

  7. removed their assignment
    on Jun 21, 2021
  8. added
    good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Jun 21, 2021
  9. openshift-bot commented on Sep 20, 2021

    @openshift-bot

    Issues go stale after 90d of inactivity.

    Mark the issue as fresh by commenting /remove-lifecycle stale.
    Stale issues rot after an additional 30d of inactivity and eventually close.
    Exclude this issue from closing by commenting /lifecycle frozen.

    If this issue is safe to close now please do so with /close.

    /lifecycle stale

  10. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Sep 20, 2021
  11. openshift-bot commented on Oct 20, 2021

    @openshift-bot

    Stale issues rot after 30d of inactivity.

    Mark the issue as fresh by commenting /remove-lifecycle rotten.
    Rotten issues close after an additional 30d of inactivity.
    Exclude this issue from closing by commenting /lifecycle frozen.

    If this issue is safe to close now please do so with /close.

    /lifecycle rotten
    /remove-lifecycle stale

  12. added
    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.
    and removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Oct 20, 2021
  13. openshift-bot commented on Nov 19, 2021

    @openshift-bot

    Rotten issues close after 30d of inactivity.

    Reopen the issue by commenting /reopen.
    Mark the issue as fresh by commenting /remove-lifecycle rotten.
    Exclude this issue from closing again by commenting /lifecycle frozen.

    /close

  14. openshift-ci commented on Nov 19, 2021

    @openshift-ci

    @openshift-bot: Closing this issue.

    Details

    In response to this:

    Rotten issues close after 30d of inactivity.

    Reopen the issue by commenting /reopen.
    Mark the issue as fresh by commenting /remove-lifecycle rotten.
    Exclude this issue from closing again by commenting /lifecycle frozen.

    /close

    Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

  15. camilamacedo86 commented on Feb 14, 2022

    @camilamacedo86
    ContributorAuthor

    /lifecycle frozen

  16. added
    lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.
    and removed
    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.
    on Feb 14, 2022
  17. ima1mai commented on Feb 17, 2022

    @ima1mai
    Contributor

    I could reproduce the error locally. I'll try looking into it.

  18. added 2 commits that reference this issue on Apr 7, 2022
    653cdd9
    cee93c9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/testingIssue related to testing the operator-sdk and subcomponentsgood first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions