namespace scope not working as expected

[yong-jie-gong]

What happened:

1. configure nginx ingress controller with specified ingress class as below

     /nginx-ingress-controller
      --watch-namespace=test1
      --annotations-prefix=ingress.kubernetes.io
      --ingress-class=test-local

2. create one one service account such as "nginx-ingress-controller-sa" whose don't have get/list/watch permission on cluster resource "networking.k8s.io/ingressclasses"
3. start nginx-ingress-controller with service account nginx-ingress-controller-sa
4. create one global ingress in the same namespace for other global ingress controller

apiVersion: networking.k8s.io/v1
kind: Ingress
  annotations:
    kubernetes.io/ingress.class: alb
  name: test-global
  namespace: test1

5. ingress object "test-global" will be caught by nginx-ingrss-controller and write into nginx configuration file by mistake, this new path rule cause conflict with other default local ingress path rule

but if I restart nginx-ingress-controller, "test-global" path rule will be removed from nginx configuration. it means nginx-ingress-controller behave different on runtime and fresh start.

What you expected to happen:
s nginx-ingress-controller should work in consistent way on runtime and fresh start.

if ingress object doesn't belongs current nginx-ingress-controller, it should be not caught and write into nginx path rule

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
nginx-ingress-controller --version

NGINX Ingress controller
Release: v1.5.1
Build: git-f966ccc43
Repository: github
nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.7-eks-fb459a0", GitCommit:"c240013134c03a740781ffa1436ba2688b50b494", GitTreeState:"clean", BuildDate:"2022-10-24T20:40:13Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.8-eks-ffeb93d", GitCommit:"abb98ec0631dfe573ec5eae40dc48fd8f2017424", GitTreeState:"clean", BuildDate:"2022-11-29T18:45:03Z", GoVersion:"go1.18.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:
NA

• Cloud provider or hardware configuration: AWS

• OS (e.g. from /etc/os-release): NA

• Kernel (e.g. uname -a): 4.18.0-305.el8.x86_64 Basic structure  #1 SMP Thu Apr 29 08:54:30 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed: helm

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

[longwuyuan]

/retitle namespace scope not working as expected

namespace-scope is something to be looked at. After the stabilization work is completed, I think this will get some attention

/triage accepted

[yong-jie-gong]

@longwuyuan Raised one PR #9712 to fix this issue

[longwuyuan]

@yong-jie-gong I think the details of the problem you are hinting at, are not explained here. Also I made a comment on your PR.

Please help out here. Please write a step-by-step instructions procedure here in such a way that someone can create a default minikube cluster or kind cluster and then copy/paste from your instructions to reproduce the problem. This is because you have made several assumptions in your post that actually are a deep dive into the configuration of the controller, but none of the impacting design aspects are clearly explained. For example, you can look at the helm chart methijd of installing the controller and the key:value pairs related to that in the values.yaml . There is also the service-account object in the static manifest

ingress-nginx/deploy/static/provider/cloud/deploy.yaml

Line 11 in 4f74e03

kind: ServiceAccount

and related permissions . That requires a caveat and explaining for customized service-account and permissions for the custom service-account

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[davidcurrie]

@longwuyuan - I have hit this issue and have a simple recreate. I believe this is also the same issue described in #9636.

1. Deploy the controller scoped to a single namespace:

helm install ingress-nginx ingress-nginx -n ingress-nginx --create-namespace --repo https://kubernetes.github.io/ingress-nginx \
  --set controller.service.type=ClusterIP,controller.ingressClassResource.name=test,controller.scope.enabled=true,controller.admissionWebhooks.enabled=false,rbac.scope=true --skip-crds

2. Create an Ingress resource with a mismatched ingress class name:

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  namespace: ingress-nginx
spec:
  ingressClassName: wrong
  rules:
    - host: www.example.com
      http:
        paths:
          - pathType: Prefix
            backend:
              service:
                name: example-service
                port:
                  number: 80
            path: /
EOF

3. The controller logs indicate that the Ingress is being ignored:

I1213 22:14:30.827574       6 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="ingress-nginx/example" error="ingress does not contain a valid IngressClass"

4. Make some update to the Ingress e.g.

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  namespace: ingress-nginx
spec:
  ingressClassName: wrong
  rules:
    - host: www.example.com
      http:
        paths:
          - pathType: Prefix
            backend:
              service:
                name: example-service-changed
                port:
                  number: 80
            path: /
EOF

5. The controller now attempts to process the Ingress resource even though the class name doesn't match:

I1213 22:15:20.974712       6 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-nginx", Name:"example", UID:"a7bad8f1-b8f9-4edb-8a1a-13f0f762f671", APIVersion:"networking.k8s.io/v1", ResourceVersion:"21415", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W1213 22:15:20.975048       6 controller.go:1108] Error obtaining Endpoints for Service "ingress-nginx/example-service-change": no object matching key "ingress-nginx/example-service-change" in local store
I1213 22:15:20.975186       6 controller.go:190] "Configuration changes detected, backend reload required"
I1213 22:15:21.051719       6 controller.go:210] "Backend successfully reloaded"

I see that the multi-ingress controller docs state:

But, when user has deployed with scope.enabled, then the ingress class resource field is not used.

It doesn't really make it clear what "not used" means, but I'd expect the behavior to be consistent. Either it should mean that the Ingress is never processed (unless the deprecated annotation is used), or it is always processed.

To give a bit more background on my scenario: I have a cluster-wide ingress controller that is used to provide external access to the cluster. For a specific application, I want to use the upstream-hash-by capability of nginx to provide consistent routing across the pods behind a service. To support this, the application deploys its own nginx controller that specifies a unique ingress class name and is scoped to the namespace in which the application runs. The RBAC permissions are also scoped to the namespace. The application defines an Ingress resource, which specifies the required ingress class name via annotation (as the controller doesn't not create an IngressClass).

This all works fine, except that the application also defines another Ingress resource in the same namespace that specifies the ingress class for the cluster-wide controller. As per the recreate above, when the resource is first created, everything is fine. It is ignored by the namespaced controller and the cluster-wide controller picks it up and processes it. Unfortunately, when the cluster-wide controller controller updates the status, that causes the namespaced controller to kick off a sync, even though the ingress class does not match. The resource then flip-flops between the two controllers.

I believe the fix in the linked PR is good. It would mean that, in the update scenario, it would still check the annotation on the resource and consequently ignore it.

[longwuyuan]

@davidcurrie Please explicitly state what the issue is for you. Are you trying to install a instance of the ingress-nginx controller that processes ingresses only from one namespace ?

• I have to check what real world case there is for creating a service type ClusterIP
• I have to check what are the default values of the flags you are using in the helm install command
• In case your final goal is to have multiple controller instances in the same cluster then try this https://kubernetes.github.io/ingress-nginx/user-guide/k8s-122-migration/#how-can-i-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster

[davidcurrie]

@longwuyuan - answers as follows:

Are you trying to install a instance of the ingress-nginx controller that processes ingresses only from one namespace ?

Yes. The complicating factors though are that a) there are other ingress resources in that same namespace that should be processed by a cluster-wide controller, and b) the namespaced controller should not have any cluster-wide permissions. In particular, the namespaced controller cannot read IngressClass.

I have to check what real world case there is for creating a service type ClusterIP

As I said above, the application has a requirement for sharding when one component is calling another. We want to use nginx's upstream-hash-by to provide the sharding. The problem exists whatever the service type but this is our motivation for having a second controller that is namespaced.

I have to check what are the default values of the flags you are using in the helm install command

The recreate above contains all of the flags necessary to recreate the problem.

In case your final goal is to have multiple controller instances in the same cluster then try this

This works because it does not set rbac.scope=true. This means that it can read IngressClass and, as a consequence, a call to GetIngressClass is made on the update at

ingress-nginx/internal/ingress/controller/store/store.go

Lines 475 to 478 in 6ff70f0

	if !icConfig.IgnoreIngressClass {
	_, errOld = store.GetIngressClass(oldIng, icConfig)
	classCur, errCur = store.GetIngressClass(curIng, icConfig)
	}

. What the PR is suggesting is that the update should always call GetIngressClass because, even if it can't read IngressClass resources (which that function handles anyway) it may still be able to usefully process annotations.

[longwuyuan]

• I can't elaborate too much without doing some real hard and long digging. But in summary, the limiting of a controller to a namespace has been a grey area for a while, and not enough users are dependent on limiting a controller instance to a namespace, so it has been on the backburner. Too many reasons to explain why at a short notice.
• I think that you can install multiple ingress-controllers in the clusters, without bothering about limiting scope to a namespace, and just use the ingressClassName field, in the ingress resource, as it is intended. It will be helpful to know if you don't want to do this and specifically why it would not work for you
• I am yet to look at the PR you mentioned, but I already know of the wide range of impacts, for namespace scoping a controller instance. That the PRmay or may not have convered is a different topic of discussion. I will look at it to know more. But I am bringing this up to state that we don't even know if we want to go in the direction of limiting the scope of a controller to a namespace, in the current context and intended future design aspects of the controller

[davidcurrie]

@longwuyuan, thanks for providing some background. As previously stated, we're using this nginx instance for internal traffic within the application. As such, we deploy the controller as part of what is essentially a packaged application. To date, we have avoided creating cluster-scoped resources, making it easier to persuade administrators to deploy the application in a multi-tenant cluster.

I appreciate that this is very definitely an edge case. What bothers me most here is the inconsistency in behavior. If the controller does not have permission to list IngressClass, then on the addition of an Ingress resource it currently uses the information that it can access (i.e. the kubernetes.io/ingress.class annotation) and otherwise ignores resources, but on an update it processes all resources regardless.

On both an add and an update, I would argue that it should ignore Ingress resources that specify annotations that do not match the class name specified on the controller. What should happen if there is no annotation could go either way. It could ignore the resource as it does on add today, or it could process the resource as it does on update. Either way, they should both do the same thing!

Note that none of the above is really about namespacing, it's just about what happens when it can't list IngressClass.

[kfox1111]

We've run into the same behavior as we run an ingress controller per tenant namespace to restrict the blast damage of any configuration issues (lots, saved us from them many times).

[longwuyuan]

@davidcurri tht udpate is helpful. I think that a ton of info needs to be exchanged for triaging what you explained. I will try to list in not any particular order

• If I assume that you are not installing the ingress-nginx controller as per docs here https://kubernetes.github.io/ingress-nginx/deploy/ but instead bundling it with your app, then those details become super important. Reason being a reader here will know details of the config and the impact of the config (customizations that are supported but not tested or known commonly).

• Related to above comment, if the install was as per docs and upgrade was also as per docs, then the resulting key:value pairs of ingress related object specs are predicatable. But if you installed in a certain way and the upgrade changed some values, then what you describe is expected

• The important key:value specs of the controller, in a multi-instance (on the same cluster) can be viewed here https://kubernetes.github.io/ingress-nginx/user-guide/k8s-122-migration/#how-can-i-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster . Specs like controllerValue, controllerName , isDefault etc come into play but can be commented on only after comparing the kubectl describe po controllerpodname (and servicename etc) of both your own packaged install and the upgrade. You can compare by doing a helm install ...... --dry-run=client, if you upgrade using helm

• Upgrades are working for me and for multiple users in general, in the use case that the install as well as upgrade was as per docs. Since upgrades break for you and the original author of this issue, my current thoughts with limited data is that the upgrade changes some key:value specs and that change has gone undetected, causing this problem. This is in the context of that PR you mentioned as in, I need to see the details before I myself suspect that there is a bug or a feature needing code-change.

• Once the issue description is adequate for a reader to reproduce, hopefully copy/pasting from a reproduce procedure you can provide, I think some actionable task will emerge. If the PR you mention is the action needed, then I think it will get gratefully reviewed.

• Please review if you need to open a new issue with more precise title & description (answers to the questions asked in the new issue template). If not then kindly mention the original description here matches your problem (but I don't currently see it that way)

• After the reproduce procedure is available, I can try to reproduce on minikube/kind

• I am not certain how to proceed with your pre-upgrade ingress resource does not spec ingressClassName with a annotation or the dedicated field. That part is a complicated description, for example https://kubernetes.github.io/ingress-nginx/user-guide/k8s-122-migration/#what-is-the-flag-watch-ingress-without-class

[davidcurrie]

@longwuyuan - I'm only describing what we are doing as motivation as to why this matters to us. In terms of reproduction, please just follow the steps in #9662 (comment) (the configuration we package in our application is based on a helm template from this command).

I believe what I'm describing is exactly the same as in the issue description. The key points are that the controller does not have permission to list IngressClass and that there is then a difference in behavior when an add is detected (annotation respected) vs an update (annotation is ignored). If the controller does have permission to list IngressClass (which is what you get if you follow the instructions in the docs) then the annotation is respected in both cases.

[longwuyuan]

Thank you @davidcurrie . I need to go backwards towards basics as I am not able to understand the problem that needs to be solved in the code. This is how I understand the related current feature of the controller in v1.9.4 ;

• User installs controller instance with a ingressClass and other distinct configs per instance
• User creates ingress object with either a annotation or the ingressClassName filed referring to any particlar ingressClass
• Controller instance is configured with only one ingressClass and the respective controller processes the rules in the ingress object that is configured with the respective ingressClassName

If I am reading the info here right, I have to guess that the description of the problem to be solved is not detailed enough

• I changed a ingress object after earlier creating it with a non-existing ingressClassName and I saw the controller log a message ignoring it on both the create as well as the edit events

The timestamps are different for the create as well as the edit events in the screenshot

• I do have a service existing with the name test0. I don't know if you had a service named "example-service-changed", in the context of the error message indicating that a endpoint does not exist for that service

 Error obtaining Endpoints for Service "ingress-nginx/example-service-change"

Previously, in the ingress configuration, I had set the backend service name to the actually existing service "test0". Now I edited the ingress object yet again and changed the backend service name to a non-existing service called "test1". And even this was ignored by the controller

Hence I think that in my test, I saw the controller ignore a ingress correctly, as expected.

Since this is not working for you, the issue description needs to be more detailed for example, output of below commands in one single post ;

• Install command as already shown above
• helm ls -A
• kubectl get deploy,svc,po -A
• kubectl describe po for each of the controller pods
• kubectl get ingressclass
• App, svc create command as shown in my screenshot with image nginx:alpine or httpbun etc
• Ingress create command
• kubectl describe ingress output
• Edit the ingress and show kubectl describe ingress again
• The logs of all the controller pods that are being tailed with -f from beginning to end

But this is all my opinion. You may just wait for comments from others on this issue and the related PR

[davidcurrie]

@longwuyuan - additional information as requested...

• Install command as already shown above (I've modified it slightly to simplify the scenario further)

helm install ingress-nginx ingress-nginx -n ingress-nginx --create-namespace --repo https://kubernetes.github.io/ingress-nginx --set controller.ingressClassResource.enabled=false,controller.scope.enabled=true,rbac.scope=true --skip-crds

• helm ls -A

NAME         	NAMESPACE    	REVISION	UPDATED                             	STATUS  	CHART              	APP VERSION
ingress-nginx	ingress-nginx	1       	2023-12-20 16:48:37.243605 +0000 UTC	deployed	ingress-nginx-4.8.3	1.9.4

• kubectl get deploy,svc,po -A

NAMESPACE       NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx   deployment.apps/ingress-nginx-controller   1/1     1            1           33s
kube-system     deployment.apps/coredns                    1/1     1            1           46h

NAMESPACE       NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default         service/kubernetes                           ClusterIP      10.96.0.1        <none>        443/TCP                      46h
ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.99.188.74     <pending>     80:31884/TCP,443:31212/TCP   33s
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.100.210.231   <none>        443/TCP                      33s
kube-system     service/kube-dns                             ClusterIP      10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       46h

NAMESPACE       NAME                                            READY   STATUS    RESTARTS      AGE
ingress-nginx   pod/ingress-nginx-controller-699745fc87-sbh4s   1/1     Running   0             33s
kube-system     pod/coredns-565d847f94-lxbcb                    1/1     Running   0             46h
kube-system     pod/etcd-minikube                               1/1     Running   0             46h
kube-system     pod/kube-apiserver-minikube                     1/1     Running   0             46h
kube-system     pod/kube-controller-manager-minikube            1/1     Running   0             46h
kube-system     pod/kube-proxy-pkkzn                            1/1     Running   0             46h
kube-system     pod/kube-scheduler-minikube                     1/1     Running   0             46h
kube-system     pod/storage-provisioner                         1/1     Running   1 (46h ago)   46h

• kubectl describe po for each of the controller pods

Name:             ingress-nginx-controller-699745fc87-sbh4s
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.49.2
Start Time:       Wed, 20 Dec 2023 16:48:42 +0000
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.9.4
                  helm.sh/chart=ingress-nginx-4.8.3
                  pod-template-hash=699745fc87
Annotations:      <none>
Status:           Running
IP:               10.244.0.28
IPs:
  IP:           10.244.0.28
Controlled By:  ReplicaSet/ingress-nginx-controller-699745fc87
Containers:
  controller:
    Container ID:  docker://fa3454d538253237ae135cf60386432e8b73071427889db1ec457db2df5fb940
    Image:         registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --watch-namespace=$(POD_NAMESPACE)
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Wed, 20 Dec 2023 16:48:42 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-699745fc87-sbh4s (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bn7jd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-bn7jd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From                      Message
  ----    ------     ----  ----                      -------
  Normal  Scheduled  60s   default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-699745fc87-sbh4s to minikube
  Normal  Pulled     60s   kubelet                   Container image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" already present on machine
  Normal  Created    60s   kubelet                   Created container controller
  Normal  Started    60s   kubelet                   Started container controller
  Normal  RELOAD     58s   nginx-ingress-controller  NGINX reload triggered due to a change in configuration

• kubectl get ingressclass

No resources found

• App, svc create command as shown in my screenshot with image nginx:alpine or httpbun etc (I didn't actually create any pods/services)

• Ingress create command (important that this is in the same namespace as the controller given it is scoped)

kubectl create ing test -n ingress-nginx --rule test.example.com/"*"=test:80 --annotation=kubernetes.io/ingress.class=wrong

• kubectl describe ingress output

Name:             test
Labels:           <none>
Namespace:        ingress-nginx
Address:
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host              Path  Backends
  ----              ----  --------
  test.example.com
                    /   test:80 (<error: endpoints "test" not found>)
Annotations:        kubernetes.io/ingress.class: wrong
Events:             <none>

• Edit the ingress and show kubectl describe ingress again (just changed the backend service name - note the sync event from the controller)

Name:             test
Labels:           <none>
Namespace:        ingress-nginx
Address:
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host              Path  Backends
  ----              ----  --------
  test.example.com
                    /   changed:80 (<error: endpoints "changed" not found>)
Annotations:        kubernetes.io/ingress.class: wrong
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  Sync    3s    nginx-ingress-controller  Scheduled for sync

• The logs of all the controller pods that are being tailed with -f from beginning to end

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.4
  Build:         846d251814a09d8a5d8d28e2e604bfc7749bcb49
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W1220 16:48:42.826798       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1220 16:48:42.826925       7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I1220 16:48:42.830586       7 main.go:249] "Running in Kubernetes cluster" major="1" minor="25" git="v1.25.3" state="clean" commit="434bfd82814af038ad94d62ebe59b133fcb50506" platform="linux/arm64"
I1220 16:48:42.959344       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
W1220 16:48:42.960797       7 main.go:111] No permissions to list and get Ingress Classes: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope, IngressClass feature will be disabled
W1220 16:48:42.964522       7 main.go:125] Unable to get NODE information: nodes "minikube" is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot get resource "nodes" in API group "" at the cluster scope
I1220 16:48:43.001544       7 ssl.go:536] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1220 16:48:43.010706       7 nginx.go:260] "Starting NGINX Ingress controller"
I1220 16:48:43.014006       7 event.go:298] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"a642aff7-7f7a-49aa-a490-9026abfd7a5c", APIVersion:"v1", ResourceVersion:"82328", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I1220 16:48:44.212559       7 nginx.go:303] "Starting NGINX process"
I1220 16:48:44.212625       7 leaderelection.go:245] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I1220 16:48:44.212895       7 nginx.go:323] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I1220 16:48:44.213131       7 controller.go:190] "Configuration changes detected, backend reload required"
I1220 16:48:44.216557       7 leaderelection.go:255] successfully acquired lease ingress-nginx/ingress-nginx-leader
I1220 16:48:44.216643       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-699745fc87-sbh4s"
I1220 16:48:44.249971       7 controller.go:210] "Backend successfully reloaded"
I1220 16:48:44.250072       7 controller.go:221] "Initial sync, sleeping for 1 second"
I1220 16:48:44.250126       7 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-699745fc87-sbh4s", UID:"078b0462-2edd-46d3-a62b-00f5e8a96e40", APIVersion:"v1", ResourceVersion:"82353", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W1220 16:54:14.906608       7 controller.go:331] ignoring ingress test in ingress-nginx based on annotation : ingress class annotation is not equal to the expected by Ingress Controller
I1220 16:54:14.906650       7 main.go:107] "successfully validated configuration, accepting" ingress="ingress-nginx/test"
I1220 16:54:14.911561       7 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="ingress-nginx/test" error="ingress class annotation is not equal to the expected by Ingress Controller"
W1220 16:55:10.704428       7 controller.go:331] ignoring ingress test in ingress-nginx based on annotation : ingress class annotation is not equal to the expected by Ingress Controller
I1220 16:55:10.704455       7 main.go:107] "successfully validated configuration, accepting" ingress="ingress-nginx/test"
W1220 16:55:10.707085       7 controller.go:1108] Error obtaining Endpoints for Service "ingress-nginx/changed": no object matching key "ingress-nginx/changed" in local store
I1220 16:55:10.707205       7 controller.go:190] "Configuration changes detected, backend reload required"
I1220 16:55:10.707097       7 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-nginx", Name:"test", UID:"a54708bd-55ea-41c6-a763-f5c06459beaa", APIVersion:"networking.k8s.io/v1", ResourceVersion:"82740", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1220 16:55:10.753886       7 controller.go:210] "Backend successfully reloaded"
I1220 16:55:10.754123       7 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-699745fc87-sbh4s", UID:"078b0462-2edd-46d3-a62b-00f5e8a96e40", APIVersion:"v1", ResourceVersion:"82353", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

At 16:55:10.704428 you see it ignore the newly created ingress as the annotation does not match that for the controller. At 16:55:10.707097 you see it being processed after the update.

[longwuyuan]

@davidcurrie thanks for the update.

This message

event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-nginx", Name:"test", UID:"a54708bd-55ea-41c6-a763-f5c06459beaa", APIVersion:"networking.k8s.io/v1", ResourceVersion:"82740", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1220 16:55:10.753886       7 controller.go:210] "Backend successfully reloaded"

does not mean that a request was sent to the controller and the controller matched the rules fro the ingress object named as "test". But if you expected to never ever see the name of the ingress called "test" ever again, in any log message of the controller, then that is very deep dive.

If you want to do that deep dive to describe a problem or bug in the code, then please edit the above post and create the app backend and send requests, first with the backend as test and next with the backend as changed. Correspondingly edit the logs also. If the request gets proper response code of 200, then it is proof that that the ingress was processed by the controller.

On a completely different observation, even though you have explained already, I still do not understand the practical need to disable the ingressClassResource. I suspect that the practical use of that spec --set controller.ingressClassResource.enabled=false is in the context of multiple ingressClasses existing and one of them being disabled.

Secondly I suspect that the spec controller.scope.enabled=true is retained by legacy and needs a developer to comment on, but its current state and expected behavioral impact on the controller is not clear to me. We do not have much priority for limiting the scope of the controller to a namespace as all users have practically switched to specifying the ingressClassName to control which ingress is processed by which controller instance, in a multi instance cluster.

Please wait for comments from others and kindly join the community meeting.

I could be wrong but my summary is ;

1. You don't want to use ingressClass annotation or ingressClassName field in a ingress object.
2. You want a controller instance to watch ingresses only in one namespace.
   If above assumptions are correct, then I have nothing more to comment as both of them are complicated grey areas that need to be discussed in a zoom meeting etc

[davidcurrie]

I can assure you that, if the backend service did exist then the controller would have configured nginx to route to it. I'm just trying to keep the recreate as simple as possible.

The desire to not use IngressClass is precisely because we want to keep everything scoped to the namespace. An IngressClass is not scoped to the namespace, nor are the permissions required to view them.

As you'll see from the linked PR in the corda organisation, our workaround for this was to give the controller permission to read IngressClass (i.e. the equivalent of setting controller.scope.enabled=true but leaving rbac.scope=false). It doesn't matter that we're still using the annotation-based approach and that there actually no IngressClass involved, this is sufficient to make things work, as just having the ability to read the IngressClass resources is enough to send the update down a different code path where it does check the annotation. That's all that the proposed fix above does - change it so that the annotation is always checked, regardless of whether IngessClass can be read.

[longwuyuan]

there is a flag to the controller like "--watch-ingress-without-class" https://github.com/search?q=repo%3Akubernetes%2Fingress-nginx%20without-class&type=code

[longwuyuan]

• This is unfortunately one area that never got sorted out

• Its clear that when users want multi-tenancy type of installs of this controller then limiting the scope to namespace is imperative. And thus the need to deal with the ingressClass as it is clusterScoped. And therein comes the flag "watch-ingress-without-class". But I still don't know if you can install the recent releases of the controller and avoid creating the incgressClass object.

• But this note is to make an update here that due to acute shortage of resources like developer-time, the namespace-scoping is not scoring high in the list of priorities. Securing the controller by default for all reported use-cases, implementing the Gteway-API are highest priorities now. We are even deprecating long supported popular features because there is just no way to maintain and support features that are far away from the Ingress-API implications.

• Ideally it will be good to close this issue as all users who posted messages here are either using the controller with the current code of namespace-scoped controller or have moved away from using namespace-scoped controller. To be frank, this link here https://kubernetes.github.io/ingress-nginx/faq/#multiple-controller-in-one-cluster , describes how to distinguish different instances of the controller installs in the same cluster. And this method makes all the RBAC and namespace-scoping go away completely.

• When a cluster has multiple tenants and all nodes of the cluster expose their TCP/IP stack to all the tenants, then the paradigm of segregating tenants is only a idea of data posted in a ETCD database but really a segregation at layer3 or layer5 etc

• Please do comment if its acceptable to close this issue or keep it open as tracking a very distant possibility that sometime in future, there will be enough developer time available to sort out namespace-scoping (in case anything does need to be sorted out).

• If there are no updates, then I will look at closing the issue

[davidcurrie]

I won't rehash the problem description. It's a shame the original PR was not merged, as I believe it was the correct fix. I understand that this is an edge case, though, and if there is no intention of implementing this simple change imminently, I'd close this issue.

[longwuyuan]

@davidcurrie thank you for the updated comment. The namespace-scoping is closer to the Ingres-API unlike a fringe implication, so closing this issue is not fair. The problem thought is resources available to work on it. So lets leave this open to track namespace-scoping.