Solved

Create a multi cluster


Userlevel 2
Badge

Team,

 

I have 2 ROSA clusters and want to create a multi cluster environment with them. I selected 1 cluster as master and executed the following command to get the context and got the below output.

[cloudshell-user@ip-10-0-146-129 ~]$ kubectl config get-contextsCURRENT   NAME                                                                         CLUSTER                                             AUTHINFO                                                          NAMESPACE
          default/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin        api-rosa4keshava-vidk-p1-openshiftapps-com:6443     cluster-admin/api-rosa4keshava-vidk-p1-openshiftapps-com:6443     default
          default/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443/cluster-admin      api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443   cluster-admin/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443   default
          demo-project/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin   api-rosa4keshava-vidk-p1-openshiftapps-com:6443     cluster-admin/api-rosa4keshava-vidk-p1-openshiftapps-com:6443     demo-project
*         kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin      api-rosa4keshava-vidk-p1-openshiftapps-com:6443     cluster-admin/api-rosa4keshava-vidk-p1-openshiftapps-com:6443     kasten-io
          kasten-io/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443/cluster-admin    api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443   cluster-admin/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443   kasten-io
          mysql/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin          api-rosa4keshava-vidk-p1-openshiftapps-com:6443     cluster-admin/api-rosa4keshava-vidk-p1-openshiftapps-com:6443     mysql
          pgsql/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin          api-rosa4keshava-vidk-p1-openshiftapps-com:6443     cluster-admin/api-rosa4keshava-vidk-p1-openshiftapps-com:6443     pgsql

Now when I try to execute the below set-primary command it errors out

./k10multicluster setup-primary --context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --name=api-rosa4keshava-vidk-p1-openshiftapps-com:6443

Bootstrapping Primary Cluster...
Getting Primary Cluster Config...
Verifying cluster parameters: api-rosa4keshava-vidk-p1-openshiftapps-com:6443
ERROR: a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*') (name: api-rosa4keshava-vidk-p1-openshiftapps-com:6443)

I cluster name I have specified looks correct for me. Kindly let me know what I am missing and how can I correct this.

Thanks,

Prasanna

icon

Best answer by FRubens 31 October 2022, 20:13

View original

17 comments

Userlevel 6
Badge +2

Thanks @Prasanna Keshava for posting your question here.

The name in the k10multicluster tool specifies a name that you want to use to identify your cluster.

This creates a K8s customer resource with the kind `cluster` and its name has to follow RFC 1123 subdomain naming convention.

In this case, lets say you want to name your cluster as `rosa4keshava-primary`, you can change the command as mentioned below.

./k10multicluster setup-primary --context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --name=rosa4keshava-primary

 

Userlevel 2
Badge

Hi Ganesh,

Thanks for your reply. When I executed the command “kubectl config get-contexts” I got the below output

*         kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin      api-rosa4keshava-vidk-p1-openshiftapps-com:6443     cluster-admin/api-rosa4keshava-vidk-p1-openshiftapps-com:6443     kasten-io

Where CURRENT = *, NAME = kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin, CLUSTER = api-rosa4keshava-vidk-p1-openshiftapps-com:6443, AUTHINFO = cluster-admin/api-rosa4keshava-vidk-p1-openshiftapps-com:6443 and NAMESPACE = kasten-io

Now when I execute the cmd “k10multicluster setup-primary \ --context=<primary_cluster_context_name> \ --name=<primary_cluster_name>”

--name= name of the cluster which I got in the above cmd output which is aas below.

api-rosa4keshava-vidk-p1-openshiftapps-com:6443

Just want to know where did u get the name “--name=rosa4keshava-primary” and how was it accepted. I would want to know this.

 

Thanks,

Prasanna

Userlevel 4
Badge +2

Hello @Prasanna Keshava ,

As @jaiganeshjk , has stated in his reply, the "--name" is a name of your choice that is used to identify your cluster, it was not extracted from your context, this "name" can be anything that helps you to identify your cluster, you can think of it as a label for your cluster.

The parameter you have to fill with the information you collected from your context is --context , that according with your context output would be:

--context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin 

Hope it answer your question.

Fernando R.

Userlevel 6
Badge +2

@Prasanna Keshava Hope Fernando has clarified your queries. 

If you are satisfied with the answer that Fernando gave, Please mark his comment as an answer to your question.

Userlevel 2
Badge

Hello Team,

With your help I could successfully create the primary node in the multi cluster environment. The command used was “./k10multicluster setup-primary --context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --name=rosa4keshava-primar”

but when I try to add another cluster as secondary cluster to this environment using the below cmd

./k10multicluster bootstrap --primary-context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --primary-name=rosa4keshava-primary --secondary-context=kasten-io/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443/cluster-admin --secondary-name=rosa4prasanna1-secondary --secondary-cluster-ingress=k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10” 

It errors out telling “ERROR: Invalid cluster ingress, missing scheme (cluster_name: rosa4prasanna1-secondary).

Please let me know what am I missing

Thanks,

Prasanna

Userlevel 6
Badge +2

@Prasanna Keshava , the ingress option should be a proper URL like http://k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10/ or  https://k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10/ based on your route configuration.

 

You should also add the root CA for your ingress controller certificates if you are using a self signed certificate in K10’s trust-store or skip TLS verification of ingress resource using --secondary-cluster-ingress-tls-insecure=true    

Userlevel 2
Badge

Hello Ganesh,

 

Thanks for your quick response. I tried the below command as suggested by you.

./k10multicluster bootstrap --primary-context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --primary-name=rosa4keshava-primary --secondary-context=kasten-io/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443/cluster-admin --secondary-name=rosa4prasanna1-secondary --secondary-cluster-ingress=https://k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10 --secondary-cluster-ingress-tls-insecure=true

But this errors out as below

ERROR: Ingress connection failed (cluster_name: rosa4prasanna1-secondary)
  -> Cluster ingress is unavailable

Thanks,

Prasanna

Userlevel 6
Badge +2

K10s default ingress path is `/k10/`

Can you try adding a `/` in the end in the ingress path provided.
 

Userlevel 2
Badge

when I do that I get the below error.

ERROR: Could not setup secondary cluster (cluster_name: rosa4prasanna1-secondary)
  -> Cluster config already exists (cluster_name: rosa4prasanna1-secondary)

So changed the secondary cluster name as “--secondary-name=rosa4prasanna1-second”

now if fails with the same error as before. So adding / has made no difference I feel

Thanks,

Prasanna

 

Userlevel 6
Badge +2

Can you try to access your secondary cluster ingress from your browser using the above URL ?
Do you see the login page ?

 

Can you confirm if you have tls enabled in the route ?
If not, you should probably use http url for the same.

Userlevel 6
Badge +2

Also, You should probably remove the stale secondary cluster config

 

k10multicluster disconnect --primary-name rosa4keshava-primary --secondary-name rosa4prasanna1-secondary

k10multicluster disconnect --primary-name rosa4keshava-primary --secondary-name rosa4prasanna1-second

Userlevel 2
Badge

[cloudshell-user@ip-10-0-153-58 ~]$ ./k10multicluster disconnect --primary-name rosa4keshava-primary --secondary-name rosa4prasanna1-second
Removing Secondary Cluster...
Getting Primary Cluster Config...
Verifying cluster parameters: rosa4keshava-primary
Getting Secondary Cluster Config...
Verifying cluster parameters: rosa4prasanna1-second
Cleaning up secondary multicluster configuration rosa4prasanna1-second from primary rosa4keshava-primary
Removing Clusters Complete!
[cloudshell-user@ip-10-0-153-58 ~]$ 

The above cmd got executed successfully. Also with name “rosa4prasanna1-secondary ws also scuucessful. But when I tried adding it as secondary cluster it was erroring out saying “ERROR: Could not setup secondary cluster (cluster_name: rosa4prasanna1-second)
  -> Cluster config already exists (cluster_name: rosa4prasanna1-second)”

Then I changed the secondary cluster name to rosa4prasanna1-sec and it was successful.

In all scenarios I have used http://

Not sure what’s the issue.

[cloudshell-user@ip-10-0-153-58 ~]$ ./k10multicluster bootstrap --primary-context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --primary-name=rosa4keshava-primary --secondary-context=kasten-io/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443/cluster-admin --secondary-name=rosa4prasanna1-sec --secondary-cluster-ingress=http://k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10/ --secondary-cluster-ingress-tls-insecure=true
Bootstrapping Clusters...
Getting Primary Cluster Config...
Verifying cluster parameters: rosa4keshava-primary
Getting Secondary Cluster Config...
Verifying cluster parameters: rosa4prasanna1-sec
Setting up secondary multicluster configuration: rosa4prasanna1-sec
Validating secondary API server connection...
Contacting secondary cluster...
Validating secondary ingress connection...
Bootstrapping Clusters Complete!

Thanks,
Prasanna

 

Userlevel 6
Badge +2

Can you go to your primary cluster Multicluster dashboard and see if you have the only one secondary cluster ?
you can remove the stale clusters from the UI as well.

Let me know if rosa4keshava-secondary and rosa4keshava-second doesn’t get removed from the UI.

I should be able to help you remove them forcibly.

Userlevel 2
Badge

I can see all the 3 secondary cluster names in the primary cluster k10 dashboard.

All 3 of them are in “cluster unreachable state”

Userlevel 2
Badge

Also tried removing the cluster from the k10 UI, after opting for removal the heading against the cluster say “Pending Removal” but can see that on the dashboard. Even the working secondary cluster is in in-accessible state in the primary cluster k10 dashboard. Let me know how this can be resolved.

Thanks,

Prasanna

Userlevel 2
Badge

Hello team,

Can I get some help on this.

Thanks,

Prasanna

Userlevel 6
Badge +2

The secondary cluster might be stuck on the finalization stage of debootstrap.

Please run the below command with the secondary cluster names that you want to remove.

 

kubectl -n kasten-io-mc patch cluster.dist.kio.kasten.io rosa4keshava-secondary -p '{"metadata":{"finalizers": null }}' --type=merge

 

kubectl -n kasten-io-mc patch cluster.dist.kio.kasten.io rosa4keshava-second -p '{"metadata":{"finalizers": null }}' --type=merge

 

And what do you see in the working secondary cluster ? Does it say `cluster unreachable` or some other errors ?

Would you be able to share a screenshot for the same ?

Comment