Terraform apply error 'The number of path segments is not divisible by 2' for Azure App Feature Flag - azure-devops

Terraform apply error 'The number of path segments is not divisible by 2' for Azure App Feature Flag
Why am I seeing this error? Hard to find any answer to this anywhere. I am using Terraform v2.93.0
and I also tried 2.90.0 and 2.56.0, and got the same problem. I was adding configs just fine but
as soon as I tried to configure a Feature Flag, it breaks the Terraform project AND
I am forced to rebuild re-init from scratch. Terraform is not able to recover on its own if I remove the config and running plan again.
╷
│ Error: while parsing resource ID: while parsing resource ID:
| The number of path segments is not divisible by 2 in
| "subscriptions/{key}/resourceGroups/my-config-test/providers/Microsoft.AppConfiguration/configurationStores/my-app-configuration/AppConfigurationFeature/.appconfig.featureflag/DEBUG/Label/my-functions-test"
│
│ while parsing resource ID: while parsing resource ID:
| The number of path segments is not divisible by 2 in
│ "subscriptions/{key}/resourceGroups/my-config-test/providers/Microsoft.AppConfiguration/configurationStores/my-app-configuration/AppConfigurationFeature/.appconfig.featureflag/DEBUG/Label/my-functions-test"
╵
╷
│ Error: obtaining auth token for "https://my-app-configuration.azconfig.io": getting authorization token for endpoint https://my-app-configuration.azconfig.io:
| obtaining Authorization Token from the Azure CLI: parsing json result from the Azure CLI: waiting for the Azure CLI: exit status 1: ERROR: The command failed with an unexpected error. Here is the traceback:
│ ERROR: [Errno 2] No such file or directory
WHY is the slash missing from the front of the ID????
And here is the config that breaks it:
resource "azurerm_app_configuration_feature" "my_functions_test_DEBUG" {
configuration_store_id = azurerm_app_configuration.my_app_configuration.id
description = "Debug Flag"
name = "DEBUG"
label = "my-functions-test"
enabled = false
}
When it is healthy, the apply on configs works, and looks like this:
Plan: 4 to add, 0 to change, 0 to destroy.
Do you want to perform these actions in workspace "my-app-config-test"?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
azurerm_resource_group.my_config_rg_test: Creating...
azurerm_resource_group.my_config_rg_test: Creation complete after 0s [id=/subscriptions/{key}/resourceGroups/my-config-test]

Ok, I figured it out. There is a bug: when create a azurerm_app_configuration_key resource, the key can be like so key = "/application/config.EXSTREAM_DOMAIN" BUT when you create a azurerm_app_configuration_feature, you will HOSE your terraform config if you try to set the name field to name = .appconfig.featureflag/DEBUG. Instead, just set the name field to DEBUG. If you don't do that, you have to completely reset your terraform and re-initialize all the resources. Had to learn the hard way. There error message was not helpful but could be updated to be helpful in this respect.

Related

Flink not starting TaskManagers on Kubernetes, job reached global-terminal state

I have deployed a Flink cluster to Kubrnetes and I only see the JobManagers running.
I had Flink running on another Kubernetes cluster where I did a SavePoint using the FlinkDeployment from the Flink Operator. The Savepoint was saved correctly. I then deployed the Flink app to the new Kubernetes cluster and patched the savepointLocationPath in FlinkDeployment.
The Flink pods now log this error
│ WARN org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Ignoring JobGraph submission 'Windchill ESI Post Processing' because the job already reached a globally-terminal state (i.e. FAILED, CANCELED, FINISHED) in a previous execution.
...
│ io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException: Unable to update ConfigMapLock
...
│ Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://10.0.0.1/api/v1/namespaces/post-processing-int2/configmaps/post-processing-cluster-c │
│ onfig-map. Message: Operation cannot be fulfilled on configmaps "post-processing-cluster-config-map": the object has been modified; please apply your changes to the latest version and tr │
│ y again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=configmaps, name=post-processing-cluster-config-map, retryAfterSeconds=null, u │
│ id=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on configmaps "post-processing-cluster-config-map": the object has been modified; please apply your │
│ changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, st │
│ atus=Failure, additionalProperties={}).
The ConfgiMap mentioned in the error is present.
My question is how do I start a new TaskManager now? I have numberOfTaskSlots: 4 set. I tried shelling into a JobManager pod and running bin/taskmanager.sh start but this just started a process in the pod which doesn't seem correct to me. I then stopped it.
I am expecting to see a new TaskManager pod start up. Thank you
The clue was in the first line of the logs
WARN org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Ignoring JobGraph submission 'Windchill ESI Post Processing' because the job already reached a globally-terminal state (i.e. FAILED, CANCELED, FINISHED) in a previous execution
My mistake started with this command
kubectl patch flinkdeployment/<name-of-flink-deployment> --type=merge -p '{"spec": {"job": {"state": "suspended", "upgradeMode": "savepoint"}}}'
The problem was the upgradeMode. That should not have been edited and left as last-state. Last state tells the Flink deployment to start from where it left off using the HA state (which is state stored in Azure Blob Storage in my case). savepoint will put the deployment in a FINISHED state and will not start a new TaskManager upon deployment.
Below is the correct edit
kubectl patch flinkdeployment/<name-of-flink-deployment> --type=merge -p '{"spec": {"job": {"state": "suspended"}}}'

Error: HttpError: Resource not accessible by integration when using merge-schedule-action

I am attempting to use merge-schedule-action to update my webcomic. However the workflow job fails. The description goes like:
/schedule 2022-01-02(which is it)
But I get an error when it runs the job after creating the pull request.
Run gr[2](https://github.com/AHumanIBelieve/ahumanibelieve.github.io/actions/runs/3817569850/jobs/6493904876#step:2:2)m/merge-schedule-action#v2
with:
merge_method: merge
time_zone: UTC
require_statuses_success: false
automerge_fail_label: automerge-fail
env:
GITHUB_TOKEN: ***
Handling pull request opened for https://github.com/AHumanIBelieve/ahumanibelieve.github.io/pull/[5](https://github.com/AHumanIBelieve/ahumanibelieve.github.io/actions/runs/3817569850/jobs/6493904876#step:2:4)
Schedule date found: "2022-01-02"
Error: HttpError: Resource not accessible by integration
Is there something I am missing or is it just my repo? Do I need to give it permissions or something?
Link to pull request: AHumanIBelieve/ahumanibelieve.github.io#5
I have tried copy and pasting the workflow yaml file directly, but it doesn't work.
https://github.com/marketplace/actions/merge-schedule

AWS CDK not deploying API Gateway change (EDGE to REGIONAL)

I'm experimenting with AWS CDK by converting a console-developed application (just API Gateway and Lambdas for now). All is well--I can hit the API's resources and methods and the appropriate lambdas get executed.
I'm trying to understand what triggers a deployment (and what doesn't). For example, if I try to change the API's endpoint type from the default (EDGE) to REGIONAL:
const api = new apigateway.RestApi(this, "cy-max-api", {
restApiName: "CY Max Service",
description: "CDK version of Max AWS demo app.",
endpointConfiguration: [EndpointType.REGIONAL] // <-- add only this line and deploy
});
and deploy (cdk deploy), nothing is deployed (I checked the logs, console says no stack changes). I even tried forcing the deploy (cdk deploy -f)--no joy.
I suspect this is the expected behavior, but would like to understand why this change doesn't trigger a deploy (and what would be necessary to force one).
Update based on response by #balu-vyamajala (thanks for taking the time to test it).
I am using version 1.82.0 of the CDK. Here's the result of cdk diff when the only change is adding the endpointConfiguration line:
Stack CyMaxStack
Resources
[-] AWS::ApiGateway::Deployment CyMaxcymaxapiDeploymentD64E3EA0186ed2bffe1dbc3004a8457d0ce5eb28 destroy
[+] AWS::ApiGateway::Deployment CyMax/cy-max-api/Deployment CyMaxcymaxapiDeploymentD64E3EA0cd62c1e6cd1229987f977199cc5906ea
[~] AWS::ApiGateway::RestApi CyMax/cy-max-api CyMaxcymaxapi48ECF39D
└─ [+] EndpointConfiguration
└─ {}
[~] AWS::ApiGateway::Stage CyMax/cy-max-api/DeploymentStage.prod CyMaxcymaxapiDeploymentStageprod5291AAF0
└─ [~] DeploymentId
└─ [~] .Ref:
├─ [-] CyMaxcymaxapiDeploymentD64E3EA0186ed2bffe1dbc3004a8457d0ce5eb28
└─ [+] CyMaxcymaxapiDeploymentD64E3EA0cd62c1e6cd1229987f977199cc5906ea
and here's what cdk deploy says:
CyMaxStack: deploying...
[0%] start: Publishing 6280a7c7fbc87dd62aeb85e098d6de2f0b644eea442dcbfc67063a56c08ce151:current
[100%] success: Published 6280a7c7fbc87dd62aeb85e098d6de2f0b644eea442dcbfc67063a56c08ce151:current
CyMaxStack: creating CloudFormation changeset...
[█████████████████████████████·····························] (5/10)
✅ CyMaxStack
Outputs:
CyMaxStack.CyMaxcymaxapiEndpoint52D905B0 = https://...my URL...
Stack ARN:
arn:aws:cloudformation:us-west-1:...my ARN...
When I check the console the API has not been updated to REGIONAL. Also, endpointConfiguration is either missing, or {} in cdk.out/tree.json. I never see {REGIONAL} in that file.
I am guessing you are asking about update to AWS::ApiGateway::Deployment which doesn't automatically happen and cdk generates a hash of methods and resources to append to resource name to force deployment.
But in your case, EndpointConfiguration is a property of AWS::ApiGateway::RestApi which is directly referred in AWS::ApiGateway::Deployment. Irrespective of any other changes, it must trigger a new Deployment.
which version of cdk you are using?
I just tested it with 1.80.0, it did trigger a change in three resources AWS::ApiGateway::Deployment, AWS::ApiGateway::Stage and AWS::ApiGateway::RestApi.
Please try cdk synth and observe generated CloudFormation for resource AWS::ApiGateway::RestApi before and after compiling your change

How to reconcile the Terraform State with an existing bucket?

Using Terraform 11.14
My terraform file contains the following resource:
resource "google_storage_bucket" "assets-bucket" {
name = "${local.assets_bucket_name}"
storage_class = "MULTI_REGIONAL"
force_destroy = true
}
And this bucket has already been created (it exists on the infrastructure based on a previous apply)
However the state (remote on gcs) is inconsistent and doesn't seem to include this bucket.
As a result, terraform apply fails with the following error:
google_storage_bucket.assets-bucket: googleapi: Error 409: You already own this bucket. Please select another name., conflict
How can I reconcile the state? (terraform refresh doesn't help)
EDIT
Following #ydaetskcoR's response, I did:
terraform import module.bf-nathan.google_storage_bucket.assets-bucket my-bucket
The output:
module.bf-nathan.google_storage_bucket.assets-bucket: Importing from ID "my-bucket"...
module.bf-nathan.google_storage_bucket.assets-bucket: Import complete! Imported google_storage_bucket (ID: next-assets-bf-nathan-botfront-cloud)
module.bf-nathan.google_storage_bucket.assets-bucket: Refreshing state... (ID: next-assets-bf-nathan-botfront-cloud)
Error: module.bf-nathan.provider.kubernetes: 1:11: unknown variable accessed: var.cluster_ip in:
https://${var.cluster_ip}
The refreshing step doesn't work. I ran the command from the project's root where a terraform.tfvars file exists.
I tried adding -var-file=terraform.tfvars but no luck. Any idea?
You need to import it into the existing state file. You can do this with the terraform import command for any resource that supports it.
Thankfully the google_storage_bucket resource does support it:
Storage buckets can be imported using the name or project/name. If the project is not passed to the import command it will be inferred from the provider block or environment variables. If it cannot be inferred it will be queried from the Compute API (this will fail if the API is not enabled).
e.g.
$ terraform import google_storage_bucket.image-store image-store-bucket
$ terraform import google_storage_bucket.image-store tf-test-project/image-store-bucket

Chef Deployment with irrelevant default symlinks

I am trying to deploy my application code with Chef, which is working for one node and failing on another. I cannot determine why it works for one node and not another when they have the exact same config, but I can at least try to debug the problem on the node that fails.
deploy_revision app_config['deploy_dir'] do
scm_provider Chef::Provider::Git
repo app_config['deploy_repo']
revision app_config['deploy_branch']
if secrets["deploy_key"]
git_ssh_wrapper "#{app_config['deploy_dir']}/git-ssh-wrapper" # For private Git repos
end
enable_submodules true
shallow_clone false
symlink_before_migrate({}) # Symlinks to add before running db migrations
purge_before_symlink [] # Directories to delete before adding symlinks
create_dirs_before_symlink [] # Directories to create before adding symlinks
# symlinks()
action :deploy
restart_command do
service "apache2" do action :restart; end
end
end
This is my recipe for deploying the code. Notice that I have tried disabling symlinking entirely, as Chef always jams its own defaults in. Even with this I get the error:
================================================================================
Error executing action `deploy` on resource 'deploy_revision[/var/www]'
================================================================================
Chef::Exceptions::FileNotFound
------------------------------
Cannot symlink /var/www/shared/config/database.yml to /var/www/releases/7404041cf8859a35de90ae72091bea1628391075/config/database.yml before migrate: No such file or directory - /var/www/shared/config/database.yml or /var/www/releases/7404041cf8859a35de90ae72091bea1628391075/config/database.yml
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/kapture/recipes/api.rb
68:
69: deploy_revision app_config['deploy_dir'] do
70: scm_provider Chef::Provider::Git
71: repo app_config['deploy_repo']
72: revision app_config['deploy_branch']
73: if secrets["deploy_key"]
74: git_ssh_wrapper "#{app_config['deploy_dir']}/git-ssh-wrapper" # For private Git repos
75: end
76: enable_submodules true
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/kapture/recipes/api.rb:69:in `from_file'
deploy_revision("/var/www") do
destination "/var/www/shared/cached-copy"
symlink_before_migrate {"config/database.yml"=>"config/database.yml"}
updated_by_last_action true
restart_command #<Proc:0x00007f40f366e5a0#/var/chef/cache/cookbooks/kapture/recipes/api.rb:82>
repository_cache "cached-copy"
retries 0
keep_releases 5
create_dirs_before_symlink ["tmp", "public", "config"]
updated true
provider Chef::Provider::Deploy::Revision
enable_submodules true
deploy_to "/var/www"
current_path "/var/www/current"
recipe_name "api"
revision "HEAD"
scm_provider Chef::Provider::Git
purge_before_symlink ["log", "tmp/pids", "public/system"]
git_ssh_wrapper "/var/www/git-ssh-wrapper"
remote "origin"
shared_path "/var/www/shared"
cookbook_name "kapture"
symlinks {"log"=>"log", "system"=>"public/system", "pids"=>"tmp/pids"}
action [:deploy]
repo "git#github.com:kapture/api.git"
retry_delay 2
end
[2012-09-24T15:42:07+00:00] ERROR: Running exception handlers
[2012-09-24T15:42:07+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
[2012-09-24T15:42:07+00:00] ERROR: Exception handlers complete
[2012-09-24T15:42:07+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2012-09-24T15:42:07+00:00] FATAL: Chef::Exceptions::FileNotFound: deploy_revision[/var/www] (kapture::api line 69) had an error: Chef::Exceptions::FileNotFound: Cannot symlink /var/www/shared/config/database.yml to /var/www/releases/7404041cf8859a35de90ae72091bea1628391075/config/database.yml before migrate: No such file or directory - /var/www/shared/config/database.yml or /var/www/releases/7404041cf8859a35de90ae72091bea1628391075/config/database.yml
Here you can see it mention database.yml, tmp/, system/ and pids folders, all of which are defaults that Chef likes to set (see related bug)
Question 1
What are these symlinks for and how do I know if I even need any. What sort of things am I symlinking? I will be using migrations, so if they are useful for the migration then I'll need them working.
I have read the documentation many times and it just doesn't explain this is plain English - at least not that I have found.
Question 2
If I do not require them, how can I disable symlinking entirely? Following the examples in that bug report have not helped.
Clear out all the symlink attributes.
deploy_revision("/var/www") do
# ...
symlink_before_migrate.clear
create_dirs_before_symlink.clear
purge_before_symlink.clear
symlinks.clear
end
Make sure the deployment directory in shared has the proper directory structure (/var/www/shared/[log,pids,system,config]) and that all config files necessary for your application are in the config directory.
Your recipe for your application's cookbook should have an array of directory names to create (recursively) so that you won't run into this error again.
The symlinks are there so that while your application code will continue to evolve, you can share the log, pids, and system folder by symlinking shared/log to current/log and so on and so forth..
Chef happens to cache the directory structure - somehow, I haven't dug into that - with this troll application cookbook. It's something in the deploy resource I believe - I never use that - but you can fix it by deleting the directory structure it creates in /var/derp or whatever. Also ensure your tmp directory is setup.
A couple reasons this may be an issue:
File permissions are incorrect
The currently running chef user cannot access the file
You are using the application cookbook in a configuration for rails deploys and your application does not have the same directory structure.
It's definitely caused by Chef caching the state of the deploy somewhere, then reading that state out of it's cache - wherever that is - and then reusing that. I would look at either the application cookbook to see for any persistance, and if you fail at finding it there, look in the deploy resource from chef itself.