Ingress question and initial install UI issue

[ksully-execute]

I have an new on-prem (kubeadm, couchbase) 1015-1 install running, but it doesn't seem like the nginx instance is hooked up to the auth-server or any of the other endpoints.
When I try to hit the documented endpoints:

Auth server https://fqdn/.well-known/openid-configuration
-fido2 https://fqdn/.well-known/fido2-configuration
-scim https://fqdn/.well-known/scim-configuration

and my ingress is exposed on a given ip, which i have dns set up on that ip as well....
When I hit that endpoint using the fdqn, I get a 404
curl -k https:///.well-known/openid-configuration
I would think that this endpoint is coming from the auth-server service, which in my case is a ClusterIP running on 10.108.170.78:8080
so shouldn't this work against that IP?
curl http://10.108.170.78:8080/.well-known/openid-configuration
but I get a 404 there too.

Nginx log shows that it's missing an active endpoint:
W0724 18:05:14.123896 7 controller.go:1207] Service "jans/auth-server" does not have any active Endpoint. │ W0724 18:05:14.123998 7 controller.go:1207] Service "jans/config-api" does not have any active Endpoint. │
Nginx was running prior to the jans install.
I see messages earlier in the log atoub updating the Ingress status for the endpoints, and pointing to the right IP'
I0724 18:03:43.444129 7 status.go:300] "updating Ingress status" namespace="jans" ingress="janssen-nginx-ingress-openid-config" curren │ │ tValue=null newValue=[{"ip":"172.16.1.7"}] │ │ I0724 18:03:43.444128 7 status.go:300] "updating Ingress status" namespace="jans" ingress="janssen-nginx-ingress-device-code" currentV │ │ alue=null newValue=[{"ip":"172.16.1.7"}] │ │ I0724 18:03:43.444207 7 status.go:300] "updating Ingress status" namespace="jans" ingress="janssen-nginx-ingress-webfinger" currentVal │ │ ue=null newValue=[{"ip":"172.16.1.7"}] │ │ I0724 18:03:43.444435 7 status.go:300] "updating Ingress status" namespace="jans" ingress="janssen-nginx-ingress-uma2-config" currentV │ │ alue=null newValue=[{"ip":"172.16.1.7"}] │ │ I0724 18:03:43.444670 7 status.go:300] "updating Ingress status" namespace="jans" ingress="janssen-nginx-ingress-webdiscovery" current │ │ Value=null newValue=[{"ip":"172.16.1.7"}] │ │ I0724 18:03:43.445660 7 status.go:300] "updating Ingress status" namespace="jans" ingress="janssen-nginx-ingress-u2f-config" currentVa │ │ lue=null newValue=[{"ip":"172.16.1.7"}] │

I have the lbIP and fqdn defined in helm, though I have questions on that (below).
Anybody got a clue?

====and further on the ingress
It's a bit confusing, having to specify the lbIP whether the fqdn is registered or not....as per the docs (https://docs.jans.io/head/admin/install/helm-install/google-gke/#jans-installation-using-helm).
And stranger, am used to the product creating the ingress itself, and that ingress being advertised through the ingress.
Q: Does the install create the ingress? If not, why not?
Q: Why, if the fqdn is dns registered, do you have to specify a lbIP?
Does the product install use the fqdn? Why is the ingress not just created, and leave the dns part to the user install process?
Am not sure how to proceed, guess I can create an ingress with a sample app first then assign the lbIP and update dns as I have been doing...

[misba7]

There's 2 types of ingress.
Ingress controller: nginx for example.. which YOU deploy
Ingress resource: where you map to a k8s service, define paths, annotations..etc

And here we come into your questions

A1: Yes, jans deploys ingress(resources) defined here

A2: LB stands in front of the k8s services.. that's what we'll always call to reach the jans k8s services > jans k8s pods
So it's always needed.

fqdn if registered, it will get mapped to lbip inside the pods as described here

....

Mimic FQDN/domain is registered section and make sure you added the full configuration correctly(LbIp, isfqdnregistered flag, fqdn, hosts..etc)

Double check the deployed helm installation using for example helm get values and see if your configuration is as intended

[ksully-execute]

But, and perhaps a dumb question, but why does the product use the lb address or fdqn at all? Shouldn’t it be using the internal clusterIP off the internal services and associated internal dns?

[moabu]

Because atm the internal jans services require an fqdn as microservices. Meaning forexample it would actually call the https://FQDN/.well-known/openid-configuration endpoint for certain configurations. Its an upstream requirement. The lb address is just a way to map it internally if the fqdn is not registered . The fqdn in short is a requirement for the idp backend and java services .

[ksully-execute]

Ok, that makes sense (if that’s all that the external entry point is used for, otherwise that’s an unnecessary point of attack). Still bothered by the fact that the lbIP is required if the fqdn is registered (you can get IP through the hostname lookup), but ok. Give me your thoughts though, because it’s still a chicken and egg issue for me……in on on-prem environment, the fqdn would not be registered first since the ingress establishes the lbIP (through metallb in my case). And since Jans is creating the ingress, I won’t know the lbIP till after it does that. So what to do? I could create an ingress first, deploying some little sample app to make that occur, and therefore have the lbip and register the fqdn as the next step, then install jans…seems like a lot of run around. Any other thoughts?

[ksully-execute]

Back to trying to get nginx to work first. After install, the nginx ingress isn't responding correctly.
Looking at the service/auth-server, I tried to hit that directly (if this is right), directly to the cluster-ip, and get:

 curl -k https://10.102.201.146:8080/jans-auth/sys/health-check
curl: (35) error:0A00
[override.txt](https://github.com/JanssenProject/jans/files/12197352/override.txt)
010B:SSL routines::wrong version number

Have I missed a step to create certs or something?
override.txt

[misba7]

Yes. There is a mistake in your override.yaml.

cnPersistenceType: couchbase is defined once inside global.
You defined it twice.. the other one was outside global, at line 68.. this stops further yaml configuration to be overrided.

So please remove that.

That's why I mentioned to double check your deployed configuration using helm get values.

[ksully-execute]

Oops, thanks for catching that, will correct and report back

[ksully-execute]

removed that line, and have things running, and I can now access the well known endpoints and health check successfully!
Question though: When I do a k get ingress I see these, and they don't list the ingress class, is that correct?
NAME CLASS HOSTS ADDRESS PORTS AGE janssen-nginx-ingress-auth-server <none> jans.executeinc.com 172.16.1.7 80, 443 96s janssen-nginx-ingress-config-api <none> jans.executeinc.com 172.16.1.7 80, 443 96s janssen-nginx-ingress-device-code <none> jans.executeinc.com 172.16.1.7 80, 443 96s janssen-nginx-ingress-firebase-messaging <none> jans.executeinc.com 172.16.1.7 80, 443 96s janssen-nginx-ingress-openid-config <none> jans.executeinc.com 172.16.1.7 80, 443 96s janssen-nginx-ingress-u2f-config <none> jans.executeinc.com 172.16.1.7 80, 443 96s janssen-nginx-ingress-uma2-config <none> jans.executeinc.com 172.16.1.7 80, 443 96s janssen-nginx-ingress-webdiscovery <none> jans.executeinc.com 172.16.1.7 80, 443 96s janssen-nginx-ingress-webfinger <none> jans.executeinc.com 172.16.1.7 80, 443 96s

[misba7]

We were passing the ingress class as an annotation which is deprecated now and it should be passed as spec.ingressClassName

The fix is pushed and you will find the ingress class name in the next jans helm release.

[ksully-execute]

Running 1.0.17-dev
went back to nginx since I can't get the cilium ingress to work correctly.
The checks for auth-server are fine:
curl -k https://jans.executeinc.com/jans-auth/sys/health-check curl -k https://jans.executeinc.com/jans-auth/.well-known/openid-configuration
And I have the fido2 and scim services and ingresses there:
janssen-nginx-ingress-u2f-config
Name: janssen-nginx-ingress-u2f-config │ │ Labels: app=janssen-nginx-ingress-u2f-config │ │ app.kubernetes.io/managed-by=Helm │ │ Namespace: jans │ │ Address: 172.16.1.7 │ │ Ingress Class: nginx │ │ Default backend: <default> │ │ TLS: │ │ tls-certificate terminates jans.executeinc.com │ │ Rules: │ │ Host Path Backends │ │ ---- ---- -------- │ │ jans.executeinc.com │ │ /.well-known/fido-configuration auth-server:8080 (10.0.1.130:8080) │ │ Annotations: meta.helm.sh/release-name: janssen │ │ meta.helm.sh/release-namespace: jans │ │ nginx.ingress.kubernetes.io/proxy-read-timeout: 300 │ │ nginx.ingress.kubernetes.io/rewrite-target: /jans-auth/restv1/fido-configuration │ │ nginx.ingress.kubernetes.io/ssl-redirect: false │ │ Events: │ │ Type Reason Age From Message │ │ ---- ------ ---- ---- ------- │ │ Normal Sync 39m (x2 over 40m) nginx-ingress-controller Scheduled for sync

and scim likewise.
Why am I not getting a response to the:
curl -k https://jans.executeinc.com/.well-known/fido2-configuration
I get a 404.

[ksully-execute]

1.0.16-1 seems to work better, all four health checks are workikng