Create a multi cluster

Thanks @Prasanna Keshava for posting your question here.

The name in the k10multicluster tool specifies a name that you want to use to identify your cluster.

This creates a K8s customer resource with the kind `cluster` and its name has to follow RFC 1123 subdomain naming convention.

In this case, lets say you want to name your cluster as `rosa4keshava-primary`, you can change the command as mentioned below.

./k10multicluster setup-primary --context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --name=rosa4keshava-primary

Hi Ganesh,

Thanks for your reply. When I executed the command “kubectl config get-contexts” I got the below output

*         kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin      api-rosa4keshava-vidk-p1-openshiftapps-com:6443     cluster-admin/api-rosa4keshava-vidk-p1-openshiftapps-com:6443     kasten-io

Where CURRENT = *, NAME = kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin, CLUSTER = api-rosa4keshava-vidk-p1-openshiftapps-com:6443, AUTHINFO = cluster-admin/api-rosa4keshava-vidk-p1-openshiftapps-com:6443 and NAMESPACE = kasten-io

Now when I execute the cmd “k10multicluster setup-primary \ --context=<primary_cluster_context_name> \ --name=<primary_cluster_name>”

--name= name of the cluster which I got in the above cmd output which is aas below.

api-rosa4keshava-vidk-p1-openshiftapps-com:6443

Just want to know where did u get the name “--name=rosa4keshava-primary” and how was it accepted. I would want to know this.

Thanks,

Prasanna

@Prasanna Keshava Hope Fernando has clarified your queries. 

If you are satisfied with the answer that Fernando gave, Please mark his comment as an answer to your question.

Hello Team,

With your help I could successfully create the primary node in the multi cluster environment. The command used was “./k10multicluster setup-primary --context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --name=rosa4keshava-primar”

but when I try to add another cluster as secondary cluster to this environment using the below cmd

“./k10multicluster bootstrap --primary-context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --primary-name=rosa4keshava-primary --secondary-context=kasten-io/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443/cluster-admin --secondary-name=rosa4prasanna1-secondary --secondary-cluster-ingress=k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10” 

It errors out telling “ERROR: Invalid cluster ingress, missing scheme (cluster_name: rosa4prasanna1-secondary).

Please let me know what am I missing

Thanks,

Prasanna

@Prasanna Keshava , the ingress option should be a proper URL like http://k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10/ or  https://k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10/ based on your route configuration.

You should also add the root CA for your ingress controller certificates if you are using a self signed certificate in K10’s trust-store or skip TLS verification of ingress resource using --secondary-cluster-ingress-tls-insecure=true    

Hello Ganesh,

Thanks for your quick response. I tried the below command as suggested by you.

./k10multicluster bootstrap --primary-context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --primary-name=rosa4keshava-primary --secondary-context=kasten-io/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443/cluster-admin --secondary-name=rosa4prasanna1-secondary --secondary-cluster-ingress=https://k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10 --secondary-cluster-ingress-tls-insecure=true

But this errors out as below

ERROR: Ingress connection failed (cluster_name: rosa4prasanna1-secondary)
  -> Cluster ingress is unavailable

Thanks,

Prasanna

K10s default ingress path is `/k10/`

Can you try adding a `/` in the end in the ingress path provided.
 

when I do that I get the below error.

ERROR: Could not setup secondary cluster (cluster_name: rosa4prasanna1-secondary)
  -> Cluster config already exists (cluster_name: rosa4prasanna1-secondary)

So changed the secondary cluster name as “--secondary-name=rosa4prasanna1-second”

now if fails with the same error as before. So adding / has made no difference I feel

Thanks,

Prasanna

Can you try to access your secondary cluster ingress from your browser using the above URL ?
Do you see the login page ?

Can you confirm if you have tls enabled in the route ?
If not, you should probably use http url for the same.

Also, You should probably remove the stale secondary cluster config

k10multicluster disconnect --primary-name rosa4keshava-primary --secondary-name rosa4prasanna1-secondary

k10multicluster disconnect --primary-name rosa4keshava-primary --secondary-name rosa4prasanna1-second

[cloudshell-user@ip-10-0-153-58 ~]$ ./k10multicluster disconnect --primary-name rosa4keshava-primary --secondary-name rosa4prasanna1-second
Removing Secondary Cluster...
Getting Primary Cluster Config...
Verifying cluster parameters: rosa4keshava-primary
Getting Secondary Cluster Config...
Verifying cluster parameters: rosa4prasanna1-second
Cleaning up secondary multicluster configuration rosa4prasanna1-second from primary rosa4keshava-primary
Removing Clusters Complete!
[cloudshell-user@ip-10-0-153-58 ~]$ 

The above cmd got executed successfully. Also with name “rosa4prasanna1-secondary ws also scuucessful. But when I tried adding it as secondary cluster it was erroring out saying “ERROR: Could not setup secondary cluster (cluster_name: rosa4prasanna1-second)
  -> Cluster config already exists (cluster_name: rosa4prasanna1-second)”

Then I changed the secondary cluster name to rosa4prasanna1-sec and it was successful.

In all scenarios I have used http://

Not sure what’s the issue.

[cloudshell-user@ip-10-0-153-58 ~]$ ./k10multicluster bootstrap --primary-context=kasten-io/api-rosa4keshava-vidk-p1-openshiftapps-com:6443/cluster-admin --primary-name=rosa4keshava-primary --secondary-context=kasten-io/api-rosa4prasanna1-1m23-p1-openshiftapps-com:6443/cluster-admin --secondary-name=rosa4prasanna1-sec --secondary-cluster-ingress=http://k10-route-kasten-io.apps.rosa4prasanna1.1m23.p1.openshiftapps.com/k10/ --secondary-cluster-ingress-tls-insecure=true
Bootstrapping Clusters...
Getting Primary Cluster Config...
Verifying cluster parameters: rosa4keshava-primary
Getting Secondary Cluster Config...
Verifying cluster parameters: rosa4prasanna1-sec
Setting up secondary multicluster configuration: rosa4prasanna1-sec
Validating secondary API server connection...
Contacting secondary cluster...
Validating secondary ingress connection...
Bootstrapping Clusters Complete!

Thanks,
Prasanna

Can you go to your primary cluster Multicluster dashboard and see if you have the only one secondary cluster ?
you can remove the stale clusters from the UI as well.

Let me know if rosa4keshava-secondary and rosa4keshava-second doesn’t get removed from the UI.

I should be able to help you remove them forcibly.

I can see all the 3 secondary cluster names in the primary cluster k10 dashboard.

All 3 of them are in “cluster unreachable state”

Also tried removing the cluster from the k10 UI, after opting for removal the heading against the cluster say “Pending Removal” but can see that on the dashboard. Even the working secondary cluster is in in-accessible state in the primary cluster k10 dashboard. Let me know how this can be resolved.

Thanks,

Prasanna

Hello team,

Can I get some help on this.

Thanks,

Prasanna

The secondary cluster might be stuck on the finalization stage of debootstrap.

Please run the below command with the secondary cluster names that you want to remove.

kubectl -n kasten-io-mc patch cluster.dist.kio.kasten.io rosa4keshava-secondary -p '{"metadata":{"finalizers": null }}' --type=merge

kubectl -n kasten-io-mc patch cluster.dist.kio.kasten.io rosa4keshava-second -p '{"metadata":{"finalizers": null }}' --type=merge

And what do you see in the working secondary cluster ? Does it say `cluster unreachable` or some other errors ?

Would you be able to share a screenshot for the same ?