our terraform plan is suddenly reporting errors such as the following while it is 'refreshing state':
Error: multiple IAM policies found matching criteria (ARN:arn:aws:iam::aws:policy/ReadOnlyAccess); try different search;
on ../../modules/xxxx/policies.tf line 9, in data "aws_iam_policy" "read_only_access":
9: data "aws_iam_policy" "read_only_access" {
and
Error: no IAM policy found matching criteria (ARN: arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy); try different search
on ../../modules/xxxx/iam.tf line 97, in data "aws_iam_policy" "aws_eks_worker_node":
97: data "aws_iam_policy" "aws_eks_worker_node" {
We recently updated our dev EKS cluster from 1.20 to 1.21. Stage and Live environments are still on 1.20 but they are built from the same module. We didn't see these errors until a day after the upgrade and there were no changes to these reported terraform files. The errors also appear to be somewhat intermittent and random. One plan run will be successful, while the next will include some of these policies that we have defined.
I know this is a shot in the dark with limited information so please ask questions if you have them. I'm just really looking for someone knows what this error means because Google isn't returning anything useful.
We also run terraform version 0.14
Related
I recently updated our CDK code to move our OpenSearch cluster from version 1.3 to 2.3. The cluster itself seems to have upgraded to a healthy state and is still accessible / usable by our application, but CloudFormation failed when attempting to update our domain resource with:
Resource handler returned message: "Resource handler returned message: "Invalid request provided: DP Nodes are OOS, Tags operation is not allowed"
This kicked the stack into UPDATE_ROLLBACK_FAILED, which is not allowed. The cluster cannot be downgraded back to 1.3.
I'm struggling to find any information about this error it's kicking out and not quite sure how to resolve it to unblock the CloudFormation stack.
Things I have tried:
Digging through CloudWatch logs only revealed information pertaining to queries.
Forcing the rollback to occur without Domain resource. This got me back to an UPDATE_COMPLETE state, but each subsequent deploy of this stack will cause it to fail again since the core issue is not resolved.
This was an odd presentation of a permissions issue. As I was reading through some docs, I stumbled upon this section, which discusses changes to tag-based access control.
This lead me start looking into CloudTrail a bit and stumbled upon the exact error that was firing when this deploy happened. It was a little odd because the assumed role granted admin access to CloudFormation, but the last line of this event record caught my eye:
"sourceIPAddress": "cloudformation.amazonaws.com",
"userAgent": "cloudformation.amazonaws.com",
"errorCode": "ValidationException",
"errorMessage": "DP Nodes are OOS, Tags operation is not allowed",
"eventSource": "es.amazonaws.com",
Upon adding es.amazonaws.com to the trust relationship of that role, the deploy fully re-ran successfully.
Hopefully this helps someone else.
I'm using the latest ibm_watson_machine_learning SDK (python)
Until a few days/weeks ago my code was working fine but now I get an error when running
client.repository.store_model(model='./model.tar.gz', meta_props=model_metadata)
Here is some sample code:
https://github.com/IBMDecisionOptimization/oplrunonwml
Exception has occurred: IndexError
list index out of range
File "C:\Temp\oplrunonwml\oprunonwmlv2.py", line 126, in main
model_details = client.repository.store_model(model='./model.tar.gz', meta_props=model_metadata)
File "C:\Temp\oplrunonwml\oprunonwmlv2.py", line 215, in <module>
main(sys.argv[1:])
I get this error while using various different models (OPL/Cplex/Docplex) and they all fail with this error.
What's strange, is that the model is uploaded correctly in the Deployment Space and I can use it without problems in deployment/jobs on the UI or on other scripts.
The code was working fine without any changes a few weeks ago so I assume something's changed on the API side
Update:
I'm using a Cloud Lite account.
I'm also using the latest version of the SDK
client = APIClient(wml_credentials)
print(client.version) # 1.0.29
print(client.version_param) #2020-08-01
I deleted all my IBM services (ObjectStorage,WatsonStudio) and created new ones but I still get the same error.
I would suspect the WML v2 instances deployement.
*** With V2 plan, user need to use updated Python SDK (ibm-watson-machine-learning 1.0.38) ***
If you had a v1 iunstance before and according to your plan, it might have been keeping working withoutmirgation for a while.
May be you reached the end of this compatibility period.
Can you clarify your plan type?
See https://medium.com/#AlainChabrier/migrate-your-python-code-for-do-in-wml-v2-instances-710025796f7
Alain
I deleted my KF cluster last night to create a new one (using kubectl cluster command not Kfctl delete), and then when I tied to create a new one, it fails, it does not work with CLI not Console. I found other people have run into this issue before, for example (here and here)
"However, as I said even with CLI my deployment fails, the error from console is:
ailed to apply: (kubeflow.error): Code 500 with message: coordinator Apply failed for gcp: (kubeflow.error): Code 500 with message: gcp apply could not update deployment manager Error could not update storage-kubeflow.yaml; Insert deployment error: googleapi: Error 403: Request had insufficient authentication scopes.
More details:
Reason: insufficientPermissions, Message: Insufficient Permission"
and the error I get from Console is:
"Please enable APIs for your project and try again
Please enable cloud resource manager API: https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/ and iam API: https://console.developers.google.com/apis/api/iam.googleapis.com/"
Note that this error is wrong, all the apis are active already. I'm quite sure this is a bug of KF but not sure how to find a workaround, any thoughts?
With CLI, I'm using my own account which has "owner" privileges.
Thanks
It seems you have an issue with IAM and the installation of Kubeflow, a 3rd party product that itself is not supported by us; nevertheless I went ahead and dig some information about this Machine Learning product.
The main issues (and although it seems you already cover permissions) are permissions, number of projects and some fine grained points.
I was checking and found out the following things that may help
a) Troubleshooting Kubeflow 1
b) Deploying Kubeflow in GKE[2]
c) Kubleflow auto deployer for GKE[3]
There are also some discussion about a mismatch permissions setting in Kubeflow that may be worth reading [4]
Finally there is a group that, also on a best-effort basis due the nature of Kubeflow:"google-kubeflow-support#google.com" that may come in handy.
I trust this information will be useful for you to solve your issue
We manage our Github org through terraform and are getting the following error against many of our modules.
* module.product_tools.github_team_repository.write: 2 error(s) occurred:
* module.product_tools.github_team_repository.write[1]: github_team_repository.write.1: At least one permission expected from permissions map.
This occurred after we hit an abuse detection mechanism in Github which seemed to mess up our terraform state. I reverted back to an earlier version of the state file however I am now getting the above errors.
Anyone have any ideas about the permissions map?
Thanks!
Sinéad
This is a different scenario from other question on this topic. My deployment almost succeeded and I can see the following lines at the end of my log
[datalab].../#015Updating module [datalab]...done.
Jul 25 16:22:36 datalab-deploy-main-20160725-16-19-55 startupscript: Deployed module [datalab] to [https://main-dot-datalab-dot-.appspot.com]
Jul 25 16:22:36 datalab-deploy-main-20160725-16-19-55 startupscript: Step deploy datalab module succeeded.
Jul 25 16:22:36 datalab-deploy-main-20160725-16-19-55 startupscript: Deleting VM instance...
The landing page keeps showing a wait bar indicating the deployment is still in progress. I have tried deploying several times in last couple of days.
About additions described on the landing page -
An App Engine "datalab" module is added. - when I click on the pop-out url "https://datalab-dot-.appspot.com/" it throws an error page with "404 page not found"
A "datalab" Compute Engine network is added. - Under "Compute Engine > Operations" I can see a create instance for datalab deployment with my id and a delete instance operation with *******-ompute#developer.gserviceaccount.com id. not sure what it means.
Datalab branch is added to the git repo- Yes and with all the components.
I think the deployment is partially successful. When I visit the landing page again, the only option I see is to deploy the datalab again and not to start it. Can someone spot the problem ? Appreciate the help.
I read the other posts on this topic and tried to verify my deployment using - "https://console.developers.google.com/apis/api/source/overview?project=" I get the following message-
The API doesn't exist or you don't have permission to access it
You can try looking at the App Engine dashboard here, to verify that there is a "datalab" service deployed.
If that is missing, then you need to redeploy again (or switch to the new locally-run version).
If that is present, then you should also be able to see a "datalab" network here, and a VM instance named something like "gae-datalab-main-..." here. If either of those are missing, then try going back to the App Engine console, deleting the "datalab" service, and redeploying.