I have around 6 million rows in my mongodb collection and importing into meilisearch using php artisan scout:import 'model' takes forever to finish.
Importing data with limit option php artisan scout:import 'model' -c 10000 gives me the following error.
MongoDB\Exception\InvalidArgumentException
Expected "limit" option to have type "integer" but found "string"
at vendor/mongodb/mongodb/src/Exception/InvalidArgumentException.php:59
55▕
56▕ $expectedType = $typeString;
57▕ }
58▕
➜ 59▕ return new static(sprintf('Expected %s to have type "%s" but found >"%s"', $name, $expectedType, get_debug_type($value)));
60▕ }
61▕ }
62▕
+27 vendor frames
28 artisan:37
Illuminate\Foundation\Console\Kernel::handle()
I also tried exporting the collection as json from mongodb and manual importing into meilisearch using curl -X POST 'http://127.0.0.1:7700/indexs/posts/documents' / --data #/data/posts.json gives the following error.
{"message":"Invalid JSON: invalid type: map, expected a sequence at line 1 column 0","errorCode":"bad_request","errorType":"invalid_request_error","errorLink":"https://docs.meilisearch.com/errors#bad_request"}curl: (3) URL using bad/illegal format or missing URL
Posts.json is the exported json file of mongodb collection using mongoexport command.
How can I import data fast into meilisearch?
Versions
"laravel/scout":"^9.1"
"laravel/framework": "^8.12",
"meilisearch/meilisearch-php": "^0.18.2",
mongodb version : "3.6"
OS
Ubuntu 20.04
After a year of going through a boring CLI to interact with my data with mongo client. I found out the best tool i wish i would have get at first time. MongoDb Compass.
After going through all the features, I found the similarity beetween this tools and PhpMyadmin. My question are.
How can i view all the query i have executed just like Phpmyadmin console.
Is it possible to Export all the query and or Import query to compass just like PhpMyadmin.
Thanks.
Compass is not PhpMyAdmin. PhpMyAdmin allows you to enter and run queries whereas Compass is more like a wrapper around find, and although it won't give you the query to run you can easily build the query yourself.
Take the following example:
I could build this query like so:
db.users.find({ username: 'jim' }, { password: 0 })
.sort({ created_at: -1 })
.skip(2)
.limit(1)
To export the result of this query you can use mongoexport. Sadly you can't use the above query for this but you will have to add a separate argument for each section. You should also note that in the above I exclude password, but with mongoexport you are unable to exclude fields - you can only specify which fields to include.
mongoexport -d test -c users -q '{ "username": "jim" }, { "password": 0 }' --fields='username,created_at' --sort '{ "created_at": -1 }' --skip 2 --limit 1 --out exported_users.json
I am using RESTHeart to access a Mongo database. RESTHeart has a an API that is supposed to create a database, e.g.:
curl -X put http://localhost:8080/db1
Well, I was using a chrome browser-based REST client that happened to do the equivalent of the follow curl call, but I accidentally forgot to nuke the data portion. It contained the JSON {"e":"f"} for data.
curl -X put -H 'Content-Type: application/json' --data-raw '{"e":"f"}' http://localhost:8080/db2`
When I then tried to do a curl get, it returns a value with the key/value pair "e":"f" stuffed in there - which is not what I want.
$ curl http://localhost:8080/db2
... { "_id" : "db2" , "e" : "f" , "_etag" : { "$oid" : "570f90601d956327e8df28c4"} , "_size" : 0 , "_total_pages" : 0 , "_returned" : 0}
Now, using the Mongo shell, I try to find this key/value pair using just about every Mongo shell command. But, I can't find it, nor can I remove it either. In fact, I can create a rather large Mongo database, then do that curl put, and I'm screwed, but it then adds the pair to my nice clean database.
Does anyone know how I can remove that strange key/value pair, either using Mongo shell, or the RESTHeart API - short of nuking the database and recreating it from scratch?! Thanks.
To remove the db property just update the db:
With PATCH:
PATCH /db {"$unset": {"e": null}}
Or with PUT
PUT /db {}
For more info look at the documentation reference sheet and representation format
I am using Robo 3T (formerly RoboMongo) which I connect to a MongoDB. What I need to do is this: There is a collection in that MongoDB. I want to export the data from that collection so that I can save it into a file.
I used the interface to open the data from the collection as text and did a Ctrl + A and pasted into a text file. However, I found that not all data is copied and also that there were many comments in the text data which naturally breaks the JSON.
I am wondering if Robo 3T has a "Export As JSON" facility so that I can do a clean export.
Any pointers are appreciated!
A quick and dirty way: Just write your query as db.getCollection('collection').find({}).toArray() and right click Copy JSON. Paste the data in the editor of your choice.
You can use tojson to convert each record to JSON in a MongoDB shell script.
Run this script in RoboMongo:
var cursor = db.getCollection('foo').find({}, {});
while(cursor.hasNext()) {
print(tojson(cursor.next()))
}
This prints all results as a JSON-like array.
The result is not really JSON! Some types, such as dates and object IDs, are printed as JavaScript function calls, e.g., ISODate("2016-03-03T12:15:49.996Z").
Might not be very efficient for large result sets, but you can limit the query. Alternatively, you can use mongoexport.
Robomongo's shell functionality will solve the problem. In my case I needed couple of columns as CSV format.
var cursor = db.getCollection('Member_details').find({Category: 'CUST'},{CustomerId :1,Name :1,_id:0})
while (cursor.hasNext()) {
var record = cursor.next();
print(record.CustomerID + "," + record.Name)
}
Output : -------
334, Harison
433, Rechard
453, Michel
533, Pal
you say "export to file" as in a spreadsheet? like to a .csv?
IMO this is the EASIEST way to do this in Robo 3T (formerly robomongo):
In the top right of the Robo 3T GUI there is a "View Results in text
mode" button, click it and copy everything
paste everything into this website: https://json-csv.com/
click the download button and now you have it in a spreadsheet.
hope this helps someone, as I wish Robo 3T had export capabilities
There are a few MongoDB GUIs out there, some of them have built-in support for data exporting. You'll find a comprehensive list of MongoDB GUIs at http://mongodb-tools.com
You've asked about exporting the results of your query, and not about exporting entire collections. Give 3T MongoChef MongoDB GUI a try, this tool has support for your specific use case.
Don't run this command on shell, enter this script at a command prompt with your database name, collection name, and file name, all replacing the placeholders..
mongoexport --db (Database name) --collection (Collection Name) --out (File name).json
It works for me.
I don't think robomongo have such a feature.
So you better use mongodb function as mongoexport for a specific Collection.
http://docs.mongodb.org/manual/reference/program/mongoexport/#export-in-json-format
But if you are looking for a backup solution is better to use
mongodump / mongorestore
If you want to use mongoimport, you'll want to export this way:
db.getCollection('tables')
.find({_id: 'q3hrnnoKu2mnCL7kE'})
.forEach(function(x){printjsononeline(x)});
Expanding on Anish's answer, I wanted something I can apply to any query to automatically output all fields vs. having to define them within the print statement. It can probably be simplified but this was something quick & dirty that works great:
var cursor = db.getCollection('foo').find({}, {bar: 1, baz: 1, created_at: 1, updated_at: 1}).sort({created_at: -1, updated_at: -1});
while (cursor.hasNext()) {
var record = cursor.next();
var output = "";
for (var i in record) {
output += record[i] + ",";
};
output = output.substring(0, output.length - 1);
print(output);
}
Using a robomongo shell script:
//on the same db
var cursor = db.collectionname.find();
while (cursor.hasNext()) {
var record = cursor.next();
db.new_collectionname.save(record);
}
Using mongodb's export and import command
You can add the --jsonArray parameter / flag to your mongoexport command, this exports the result as single json array.
Then just specify the --jsonArray flag again when importing.
Or remove the starting and ending array brackets [] in the file, then your modified & exported file will import with the mongoimport command without the --jsonArray flag.
More on Export here: https://docs.mongodb.org/manual/reference/program/mongoexport/#cmdoption--jsonArray
Import here:
https://docs.mongodb.org/manual/reference/program/mongoimport/#cmdoption--jsonArray
Solution:
mongoexport --db test --collection traffic --out traffic.json<br><br>
Where:
database -> mock-server
collection name -> api_defs
output file name -> childChoreRequest.json
An extension to Florian Winter answer for people looking to generate ready to execute query.
drop and insertMany query using cursor:
{
// collection name
var collection_name = 'foo';
// query
var cursor = db.getCollection(collection_name).find({});
// drop collection and insert script
print('db.' + collection_name + '.drop();');
print('db.' + collection_name + '.insertMany([');
// print documents
while(cursor.hasNext()) {
print(tojson(cursor.next()));
if (cursor.hasNext()) // add trailing "," if not last item
print(',');
}
// end script
print(']);');
}
Its output will be like:
db.foo.drop();
db.foo.insertMany([
{
"_id" : ObjectId("abc"),
"name" : "foo"
}
,
{
"_id" : ObjectId("xyz"),
"name" : "bar"
}
]);
I had this same issue, and running script in robomongo (Robo 3T 1.1.1) also doesn't allow to copy values and there was no export option either.
The best way I could achieve this is to use mongoexport, if mongodb is installed on your local, you can use mongoexport to connect to database on any server and extract data
To connect to Data on remote server, and csv output file, run the following mongoexport in your command line
mongoexport --host HOSTNAME --port PORT --username USERNAME --password "PASSWORD" --collection COLLECTION_NAME --db DATABASE_NAME --out OUTPUTFILE.csv --type=csv --fieldFile fields.txt
fieldFile: helps to extract the desired columns, ex:
contents of fields.txt can be just:
userId
to only extract values of the column 'userId'
Data on remote server, json output file:
mongoexport --host HOST_NAME --port PORT --username USERNAME --password "PASSWORD" --collection COLECTION_NAME --db DATABASE_NAME --out OUTPUT.json
this extracts all fields into the json file
data on localhost (mongodb should be running on localhost)
mongoexport --db DATABASE_NAME --collection COLLECTION --out OUTPUT.json
Reference: https://docs.mongodb.com/manual/reference/program/mongoexport/#use
Simple solution:
tostrictjson(db.getCollection(collection_name).find({}))
Note:
Other solutions are fine but might cause errors during import when your collection has types like Date, ObjectId etc...
Happy Hacking :)
I export using Mongodb Compass, you can export to csv or json.
On the menu of Mongo Compass select Collection-> export collection, and you can select the fields to export, and the file to export the result, previously you can specify the query.
Regards
make your search
push button view results in JSON mode
copy te result to word
print the result from word
can't find solution for simple question:
I have file text.js
use somedb
db.somecollection.findOne()
When I run this file in cmd with redirection command from file:
"mongo < text.js"
it's work properly
But when I try this way
"mongo text.js" or "mongo --shell test.js"
I got this error message
MongoDB shell version: 2.2.0
connecting to: test
type "help" for help
Wed Dec 05 16:05:21 SyntaxError: missing ; before statement pathToFile\test.js.js:1
failed to load: pathToFile\test.js.js
It's fail on "use somedb". If I remove this line, it's run without error, but console is clear.
is there any idea, what is this and how to fix?
I'm tying to find sollution for this, to create build tool for Sublime Text 2.
default build file was
{
"cmd": ["mongo","$file"]
}
but in this case I get the error above
PS. right after posting this question I find sollution for SublimeText2:
{
"selector": "source.js",
"shell":true,
"cmd": ["mongo < ${file}"]
}
PSS. right after posting this question I find sollution for SublimeText3:
{
"selector": "source.js",
"shell":true,
"cmd": ["mongo","<", "$file"]
}
this build tool work properly
use dbname is a helper function in the interactive shell which does not work when you are using mongo shell with a JS script file like you are.
There are multiple solutions to this. The best one, IMO is to explicitly pass the DB name along with host and port name to mongo like this:
mongo hostname:27017/dbname mongoscript.js // replace 27017 with your port number
A better way to do this would be to define the DB at the beginning of your script:
mydb=db.getSiblingDB("yourdbname");
mydb.collection.findOne();
etc.
The latter is preferable as it allows you to interact with multiple DBs in the same script if you need to do so.
You can specify the database while starting the mongo client:
mongo somedb text.js
To get the output from the client to stdout just use the printjson function in your script:
printjson(db.somecollection.findOne());
Mongo needs to be invoked from a shell to get that mode, with Ansible you would have this:
- name: mongo using different databases
action: shell /usr/bin/mongo < text.js
Instead of this:
- name: mongo breaking
command: /usr/bin/mongo < text.js
This is what finally worked for me on Windows + Sublime Text 2 + MongoDB 2.6.5
{
"selector": "source.js",
"shell":true,
"cmd": ["mongo","<", "$file"],
"working_dir" : "C:\\MongoDB\\bin"
}