com.mongodb.CommandFailureException: { "serverUsed" : "localhost:27017" , "createdCollectionAutomatically" : true , "numIndexesBefore" : 1 , "ok" : 0.0 , "errmsg" : "namespace name generated from index name \"NDS.ABCD_pre_import.$importabilityEvaluations.perNameResults.straightImportResults.resultPolContent_NOT_IN_CURRENT_USE.officialPolResultNameContentId\" is too long (127 byte max)" , "code" : 67}
at com.mongodb.CommandResult.getException(CommandResult.java:76)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:131)
at com.mongodb.DBCollectionImpl.createIndex(DBCollectionImpl.java:362)
at com.mongodb.DBCollection.createIndex(DBCollection.java:563)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.createIndex(MongoPersistentEntityIndexCreator.java:136)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.checkForAndCreateIndexes(MongoPersistentEntityIndexCreator.java:129)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.checkForIndexes(MongoPersistentEntityIndexCreator.java:121)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.onApplicationEvent(MongoPersistentEntityIndexCreator.java:105)
at org.springframework.data.mongodb.core.index.MongoMappingEventPublisher.publishEvent(MongoMappingEventPublisher.java:60)
at org.springframework.data.mapping.context.AbstractMappingContext.addPersistentEntity(AbstractMappingContext.java:306)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:180)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:140)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:67)
at org.springframework.data.mongodb.core.MongoTemplate.determineCollectionName(MongoTemplate.java:1881)
at org.springframework.data.mongodb.core.MongoTemplate.determineEntityCollectionName(MongoTemplate.java:1868)
at org.springframework.data.mongodb.core.MongoTemplate.save(MongoTemplate.java:825)
You can pass an index name as parameter to ensureIndex:
db.collection.ensureIndex({"birds.parrots.macaw.blue.id": 1}, {name:"myIndex1"});
db.collection.ensureIndex({"birds.parrots.macaw.blue.id": 1, "field2": 1}, {name:"myIndex1"});
record my case for other refer:
my case: code
indexKeyList = [
("shortLink", pymongo.ASCENDING),
("parsedLink.isParseOk", pymongo.ASCENDING),
("parsedLink.errType", pymongo.ASCENDING),
("parsedGame.realGameName", pymongo.ASCENDING),
("parsedGame.gameTheme", pymongo.ASCENDING),
]
mongoCollectionShortlink.create_index(indexKeyList)
report error:
发生异常: OperationFailure
namespace name generated from index name "shortLink.gameShortLink.$shortLink_1_parsedLink.isParseOk_1_parsedLink.errType_1_parsedGame.realGameName_1_parsedGame.gameTheme_1" is too long (127 byte max), full error: {'ok': 0.0, 'errmsg': 'namespace name generated from index name "shortLink.gameShortLink.$shortLink_1_parsedLink.isParseOk_1_parsedLink.errType_1_parsedGame.realGameName_1_parsedGame.gameTheme_1" is too long (127 byte max)', 'code': 67, 'codeName': 'CannotCreateIndex'}
root cause: use create_index to create multiple index. then multiple index name joined together, cause name too long, exceed 127 byte limit
Solution: should use create_indexes
import pymongo
from pymongo import IndexModel
indexShortLink = IndexModel([("shortLink", pymongo.ASCENDING)], name="shortLink")
indexIsParseOk = IndexModel([("parsedLink.isParseOk", pymongo.ASCENDING)], name="parsedLink_isParseOk")
indexErrType = IndexModel([("parsedLink.errType", pymongo.ASCENDING)], name="parsedLink_errType")
indexRealGameName = IndexModel([("parsedGame.realGameName", pymongo.ASCENDING)], name="parsedGame_realGameName")
indexGameTheme = IndexModel([("parsedGame.gameTheme", pymongo.ASCENDING)], name="parsedGame_gameTheme")
indexModelList = [
indexShortLink,
indexIsParseOk,
indexErrType,
indexRealGameName,
indexGameTheme,
]
mongoCollectionShortlink.create_indexes(indexModelList)
You can not disable indexing as MongoDB will always create an index for _id. Shorten your collection name instead - saves you some typing too
Related
When I am trying to insert the record(json) to mongodb collection. I am getting exception
{ "ok" : 0.0, "errmsg" : "Unsupported OP_QUERY command: insert. The client driver may require an upgrade. For more details see https://dochub.mongodb.org/core/legacy-opcode-removal", "code" : 352, "codeName" : "UnsupportedOpQueryCommand" }
I have MongoDb 6.0 version configured
MongoDb removes Opal_insert operation in new version.How to resolve it ?
We have MongoDB-collection which we want to import to Elasticsearch (for now as a one-off effort). For this end, we have exported the collection with monogexport. It is a huge JSON file with entries like the following:
{
"RefData" : {
"DebtInstrmAttrbts" : {
"NmnlValPerUnit" : "2000",
"IntrstRate" : {
"Fxd" : "3.1415"
},
"MtrtyDt" : "2020-01-01",
"TtlIssdNmnlAmt" : "200000000",
"DebtSnrty" : "SNDB"
},
"TradgVnRltdAttrbts" : {
"IssrReq" : "false",
"Id" : "BMTF",
"FrstTradDt" : "2019-04-01T12:34:56.789"
},
"TechAttrbts" : {
"PblctnPrd" : {
"FrDt" : "2019-04-04"
},
"RlvntCmptntAuthrty" : "GB"
},
"FinInstrmGnlAttrbts" : {
"ClssfctnTp" : "DBFNXX",
"ShrtNm" : "AVGO 3.625 10/16/24 c24 (URegS)",
"FullNm" : "AVGO 3 5/8 10/15/24 BOND",
"NtnlCcy" : "USD",
"Id" : "USU1109MAXXX",
"CmmdtyDerivInd" : "false"
},
"Issr" : "549300WV6GIDOZJTVXXX"
}
We are using the following Logstash configuration file to import this data set into Elasticsearch:
input {
file {
path => "/home/elastic/FIRDS.json"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => json
}
}
filter {
mutate {
remove_field => [ "_id", "path", "host" ]
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "firds"
}
}
All this works fine, the data ends up in the index firds of Elasticsearch, and a GET /firds/_search returns all the entries within the _source field.
We understand that this field is not indexed and thus is not searchable, which we are actually after. We want make all of the entries within the original nested JSON searchable in Elasticsearch.
We assume that we have to adjust the filter {} part of our Logstash configuration, but how? For consistency reasons, it would not be bad to keep the original nested JSON structure, but that is not a must. Flattening would also be an option, so that e.g.
"RefData" : {
"DebtInstrmAttrbts" : {
"NmnlValPerUnit" : "2000" ...
becomes a single key-value pair "RefData.DebtInstrmAttrbts.NmnlValPerUnit" : "2000".
It would be great if we could do that immediately with Logstash, without using an additional Python script operating on the JSON file we exported from MongoDB.
EDIT: Workaround
Our current work-around is to (1) dump the MongoDB database to dump.json and then (2) flatten it with jq using the following expression, and finally (3) manually import it into Elastic
ad (2): This is the flattening step:
jq '. as $in | reduce leaf_paths as $path ({}; . + { ($path | join(".")): $in | getpath($path) }) | del(."_id.$oid") '
-c dump.json > flattened.json
References
Walker Rowe: ElasticSearch Nested Queries: How to Search for Embedded Documents
ElasticSearch search in document and in dynamic nested document
Mapping for Nested JSON document in Elasticsearch
Logstash - import nested JSON into Elasticsearch
Remark for the curious: The shown JSON is a (modified) entry from the Financial Instruments Reference Database System (FIRDS), available from the European Securities and Markets Authority (ESMA) who is an European financial regulatory agency overseeing the capital markets.
My below code is producing an error. If the groupBy is removed it works fine. But I only need to get distinct values for common_id. How can i solve this issue?
MasterAffiliateProductMappingMongo::select('_id', 'our_product_id')
->where('top_deal', '=', 'true')
->orderBy('srp', 'asc')
->groupBy('common_id')
->get();
Error: [MongoDB\Driver\Exception\RuntimeException] Unrecognized
expression '$last'
MongoDocument Example:
{
"_id" : ObjectId("5911af8209ed4456d069b1d1"),
"product_id" : "MOBDRYWXFKNPZVG6",
"our_product_id" : "5948e0dca6bc725adb35af2e",
"mrp" : 0.0,
"srp" : 500.0,
"ID" : "5911af8209ed4456d069b1d1",
"common_id" : ObjectId("5911af8209ed4456d069b1d1"),
"top_deal" : "true"
}
Error Log:
[2017-06-28 12:19:46] lumen.ERROR: exception
'MongoDB\Driver\Exception\RuntimeException' with message 'Unrecognized
expression '$last'' in
/var/www/html/PaymetryService4/vendor/mongodb/mongodb/src/Operation/Aggregate.php:219
Refer to this link https://github.com/jenssegers/laravel-mongodb/issues/1185#issuecomment-321267144 it works. We should remove '_id' from the select field.
I want to select all the data which ts(timestamp) less than specific time
last_record = History.where(report_type: /#{params["report_type"]}/).order_by(ts: 1).only(:ts).last
History.where(:ts.lte => last_record.ts)
It seems this query will take super long time
I don't understand why, is there any quick way to do the sort of query ?
class History
include Mongoid::Document
include Mongoid::Timestamps
include Mongoid::Attributes::Dynamic
field :report_type, type: String
field :symbol, type: String
field :ts, type: Time
end
The query log in console
Started GET "/q/com_disagg/last" for 127.0.0.1 at 2015-01-10 10:36:55 +0800
Processing by QueryController#last as HTML
Parameters: {"report_type"=>"com_disagg"}
MOPED: 127.0.0.1:27017 COMMAND database=admin command={:ismaster=>1} runtime: 0.4290ms
...
MOPED: 127.0.0.1:27017 GET_MORE database=cot_development collection=histories limit=0 cursor_id=44966970901 runtime: 349.9560ms
Have set the timestamp as index, but still extremely slow query
db.system.indexes.find()
{ "v" : 1, "key" : { "ts" : 1 },
"name" : "ts_index",
"ns" : "cot_development.histories" }
We have been having some problems where our mongo connection fails to allow us to get the database names from the DB.
We call the method:
com.mongodb.Mongo.getDatabaseNames()
And 1 out of 100 times, it throws the following exception:
Caused by: com.mongodb.CommandFailureException: { "serverUsed" : "localhost:27017" ,
"errmsg" : "exception: can't open database in a read lock. if db was just closed, consider retrying the query. might otherwise indicate an internal error" , "code" : 15927 , "ok" : 0.0}
at com.mongodb.CommandResult.getException(CommandResult.java:76)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:131)
at com.mongodb.Mongo.getDatabaseNames(Mongo.java:397)
Looking at the mongo code, the database is the internal admin database
public List<String> getDatabaseNames(){
BasicDBObject cmd = new BasicDBObject();
cmd.put("listDatabases", 1);
>>CommandResult res = getDB(ADMIN_DATABASE_NAME).command(cmd, getOptions());
res.throwOnError();
This database is not being deleted, but there are several other databases that could be deleted around this time.
Has anyone else had this problem?