I have installed ICP CE 2.1.0 on a google cloud VM and the installation has gone well-no errors in installation process. When accessing the GUI I am able to see deployments and services but as soon as I access any part of the Catalog I get a blank white page with the text:
{"statusCode":401,"details":"Unexpected response code 401 from request:\nGET https://xx.xxx.xxx.xx:8443/console/api/v1/header?serviceId=catalog-ui&dev=false&accessUrl=https%3A%2F%2Fxx.xxx.xxx.xx%3A8443* ...... }
I have tried killing the individual pods but I get same error. When looking a the pod logs for the catalog-ui I have error 500 messages.
Has anyone experienced this or can tell my why this is the case? Understand that a cloud VM is not the best use case maybe but it should work?
Can you confirm the version level of ICP? Your post mentioned "ICP CE 2.1.0" but if you can check the user icon (top right corner) and click About, we should be able to see the full version details.
Reason for asking is that, at the 2100 level there was an intermittent catalog issue just like you describe. Generally it was caused by resource constraints on the k8s.
Details for ICP 2103, which is the latest available release:
https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0.3/getting_started/whats_new.html
Related
I recently updated our CDK code to move our OpenSearch cluster from version 1.3 to 2.3. The cluster itself seems to have upgraded to a healthy state and is still accessible / usable by our application, but CloudFormation failed when attempting to update our domain resource with:
Resource handler returned message: "Resource handler returned message: "Invalid request provided: DP Nodes are OOS, Tags operation is not allowed"
This kicked the stack into UPDATE_ROLLBACK_FAILED, which is not allowed. The cluster cannot be downgraded back to 1.3.
I'm struggling to find any information about this error it's kicking out and not quite sure how to resolve it to unblock the CloudFormation stack.
Things I have tried:
Digging through CloudWatch logs only revealed information pertaining to queries.
Forcing the rollback to occur without Domain resource. This got me back to an UPDATE_COMPLETE state, but each subsequent deploy of this stack will cause it to fail again since the core issue is not resolved.
This was an odd presentation of a permissions issue. As I was reading through some docs, I stumbled upon this section, which discusses changes to tag-based access control.
This lead me start looking into CloudTrail a bit and stumbled upon the exact error that was firing when this deploy happened. It was a little odd because the assumed role granted admin access to CloudFormation, but the last line of this event record caught my eye:
"sourceIPAddress": "cloudformation.amazonaws.com",
"userAgent": "cloudformation.amazonaws.com",
"errorCode": "ValidationException",
"errorMessage": "DP Nodes are OOS, Tags operation is not allowed",
"eventSource": "es.amazonaws.com",
Upon adding es.amazonaws.com to the trust relationship of that role, the deploy fully re-ran successfully.
Hopefully this helps someone else.
Created ECS cluster let's call it tools.
tools cluster have 2 services:
sso-proxy
grafana - opensource
Going through sso-proxy to get grafana.
when trying to enter credentials to grafana i got the following error:
When incognito mode the login working for the first time.
This is the ECS logs:
Please someone can help.
Grafana must receive proper Host header in the request header. I guess your "sso-proxy" doesn't do that. You didn't provide reproducible example (why not, when you want to help), so it is only a guess.
Lazy and insecure workaround will be Grafana downgrade to version 8.3.4-, where CSRF fix for CVE-2022-21703 is not included.
I deleted my KF cluster last night to create a new one (using kubectl cluster command not Kfctl delete), and then when I tied to create a new one, it fails, it does not work with CLI not Console. I found other people have run into this issue before, for example (here and here)
"However, as I said even with CLI my deployment fails, the error from console is:
ailed to apply: (kubeflow.error): Code 500 with message: coordinator Apply failed for gcp: (kubeflow.error): Code 500 with message: gcp apply could not update deployment manager Error could not update storage-kubeflow.yaml; Insert deployment error: googleapi: Error 403: Request had insufficient authentication scopes.
More details:
Reason: insufficientPermissions, Message: Insufficient Permission"
and the error I get from Console is:
"Please enable APIs for your project and try again
Please enable cloud resource manager API: https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/ and iam API: https://console.developers.google.com/apis/api/iam.googleapis.com/"
Note that this error is wrong, all the apis are active already. I'm quite sure this is a bug of KF but not sure how to find a workaround, any thoughts?
With CLI, I'm using my own account which has "owner" privileges.
Thanks
It seems you have an issue with IAM and the installation of Kubeflow, a 3rd party product that itself is not supported by us; nevertheless I went ahead and dig some information about this Machine Learning product.
The main issues (and although it seems you already cover permissions) are permissions, number of projects and some fine grained points.
I was checking and found out the following things that may help
a) Troubleshooting Kubeflow 1
b) Deploying Kubeflow in GKE[2]
c) Kubleflow auto deployer for GKE[3]
There are also some discussion about a mismatch permissions setting in Kubeflow that may be worth reading [4]
Finally there is a group that, also on a best-effort basis due the nature of Kubeflow:"google-kubeflow-support#google.com" that may come in handy.
I trust this information will be useful for you to solve your issue
i just signed up for Google Could Engine and started the most basic container engine quickstart on a brand new project:
https://cloud.google.com/container-engine/docs/quickstart
a few steps in it has me run this command
gcloud container clusters create example-cluster
which errors out:
$ gcloud container clusters create example-cluster
ERROR: (gcloud.container.clusters.create) ResponseError: code=503, message=Project hello-world-161713 is not fully initialized with the default service accounts. Please try again later.
so far, "trying again later" has not helped: it's been doing this every time for the past few hours.
as usual, Google has no obvious way of getting help in any timely manner, and Googling the error turns up nothing useful.
kind of a long shot but i found a link sending me over here on one of their pages (great support guys) so figured i'd give it a shot. thanks in advance.
The Container Engine API needs to be enabled, and unfortunately that error message is not correct (trying again later won't help).
If you visit the Google Container Engine page in the wb console https://console.cloud.google.com/kubernetes/list it should enable Google Container Engine. Make sure you select the project you're using with the quickstart. You can create your cluster from that page too if you'd prefer.
You can also enable the Container Engine API manually here: https://console.cloud.google.com/apis/api/container.googleapis.com/overview
This is a different scenario from other question on this topic. My deployment almost succeeded and I can see the following lines at the end of my log
[datalab].../#015Updating module [datalab]...done.
Jul 25 16:22:36 datalab-deploy-main-20160725-16-19-55 startupscript: Deployed module [datalab] to [https://main-dot-datalab-dot-.appspot.com]
Jul 25 16:22:36 datalab-deploy-main-20160725-16-19-55 startupscript: Step deploy datalab module succeeded.
Jul 25 16:22:36 datalab-deploy-main-20160725-16-19-55 startupscript: Deleting VM instance...
The landing page keeps showing a wait bar indicating the deployment is still in progress. I have tried deploying several times in last couple of days.
About additions described on the landing page -
An App Engine "datalab" module is added. - when I click on the pop-out url "https://datalab-dot-.appspot.com/" it throws an error page with "404 page not found"
A "datalab" Compute Engine network is added. - Under "Compute Engine > Operations" I can see a create instance for datalab deployment with my id and a delete instance operation with *******-ompute#developer.gserviceaccount.com id. not sure what it means.
Datalab branch is added to the git repo- Yes and with all the components.
I think the deployment is partially successful. When I visit the landing page again, the only option I see is to deploy the datalab again and not to start it. Can someone spot the problem ? Appreciate the help.
I read the other posts on this topic and tried to verify my deployment using - "https://console.developers.google.com/apis/api/source/overview?project=" I get the following message-
The API doesn't exist or you don't have permission to access it
You can try looking at the App Engine dashboard here, to verify that there is a "datalab" service deployed.
If that is missing, then you need to redeploy again (or switch to the new locally-run version).
If that is present, then you should also be able to see a "datalab" network here, and a VM instance named something like "gae-datalab-main-..." here. If either of those are missing, then try going back to the App Engine console, deleting the "datalab" service, and redeploying.