Missing rollover_alias index setting in OpenSearch - opensearch

Question

I am trying to setup index rollover in OpenSearch with simple min_doc_count condition, but I am getting "message": "Missing rollover_alias index setting [index=app_logs-000002]" error.
I have a rollover alias called app_logs, and also have the following policy (for demo purpose it is dummy to rollover after 3 documents) attached to indexes:
PUT _plugins/_ism/policies/rollover_policy
{
"policy": {
"description": "Rollover policy",
"default_state": "rollover",
"states": [
{
"name": "rollover",
"actions": [
{
"rollover": {
"min_doc_count": 3
}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"app_logs-*"
]
}
]
}
}
GET _cat/aliases:
app_logs app_logs-000001 - - - false
app_logs app_logs-000002 - - - true
GET _cat/indices:
yellow open app_logs-000002 V4j0gxaYTcqoQZvtd0u2zc 1 1 6 0 4.1kb 4.1kb
yellow open app_logs-000001 AnPjlOq6Q5We411z2q_YpQ 1 1 5 0 18.8kb 18.8kb
...
When doing
GET _opendistro/_ism/explain/app_logs-000002?pretty I get:
{
"app_logs-000002" : {
"index.plugins.index_state_management.policy_id" : "rollover_policy",
"index.opendistro.index_state_management.policy_id" : "rollover_policy",
"index" : "app_logs-000002",
"index_uuid" : "V4j0gxaYTcqoQZvtd0u2zc",
"policy_id" : "rollover_policy",
"policy_seq_no" : -2,
"policy_primary_term" : 0,
"rolled_over" : false,
"index_creation_date" : 1659299029428,
"state" : {
"name" : "rollover",
"start_time" : 1659299410303
},
"action" : {
"name" : "rollover",
"start_time" : 1659424192817,
"index" : 0,
"failed" : true,
"consumed_retries" : 3,
"last_retry_time" : 1659424804833
},
"step" : {
"name" : "attempt_rollover",
"start_time" : 1659424192817,
"step_status" : "failed"
},
"retry_info" : {
"failed" : false,
"consumed_retries" : 0
},
"info" : {
"message" : "Missing rollover_alias index setting [index=app_logs-000002]"
},
"enabled" : false
},
"total_managed_indices" : 1
}
When I do GET app_logs-000002/_settings I get:
{
"app_logs-000002" : {
"settings" : {
"index" : {
"creation_date" : "1659299029428",
"number_of_shards" : "1",
"number_of_replicas" : "1",
"uuid" : "V4j0gxaYTcqoQZvtd0u2zc",
"version" : {
"created" : "136227827"
},
"provided_name" : "app_logs-000002"
}
}
}
}
so yes rollover alias setting is really missing there. But I would expect that this will be added automatically.
When I do GET _template I get:
{
"ism_rollover" : {
"order" : 0,
"index_patterns" : [
"app_logs-*"
],
"settings" : {
"index" : {
"opendistro" : {
"index_state_management" : {
"rollover_alias" : "app_logs"
}
}
}
},
"mappings" : { },
"aliases" : { }
}
}
so rollover_alias is there in template. Why this is not used in a new index from template?
Thanks!

Answer 1

I experienced a similar problem. The issue was that the indices needed to be created after the ism policy and template. I'm not sure if you managed to find a solution but perhaps for those future users this could prove useful.
Some docs:
Very useful sample on setting up a rolling index strategy: https://opensearch.org/docs/latest/im-plugin/ism/policies/#sample-policy-with-ism-template-for-auto-rollover
Official AWS docs on the same topic with some examples: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html.
A great writeup on common errors experienced when implementing a rolling index ISM policy: https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-failed-rollover-index/
In your case it appears that the policy was not correctly applying to your indices which is likely a result of you creating your indices before the policy and template were created. If you want to add a policy to an index see the step 6 of Create an ISM policy in the linked AWS docs above:
POST _plugins/_ism/add/my-index
{
"policy_id": "my-policy-id"
}
Here is how I went about solving this problem using a policy and template:
Implement an ISM policy (as you did above)
Create an ISM template
PUT _template/ism_rollover_app
{
"index_patterns": "app_logs-*",
"settings": {
"index": {
"opendistro.index_state_management.rollover_alias": "app_logs"
}
}
}
Create an initial index called app_logs-00001 (or some variant that matches the regex ^.*-\d+$)
This should hopefully see app_logs-00001 be created from the ism_rollover_app template and have the app_logs index associated with it. This should subsequently fix this missing alias issue.