Upload Data to Meteor / Mongo DB - mongodb

I have a Meteor app and would like to upload data (from csv) to a meteor collection.
I have found:
solutions (e.g. Collectionfs) which deal with file uploads
methods for uploading directly to the underlying mongo db from the shell
references to meteor router - but I am using the excellent iron-router, which does not appear to provide this functionality
My requirement is that the app user be able to upload csv data to the app from within the app. I do not need to store the csv file anywhere within the app file structure, I just need to read the csv data to the collection.
It is possible that I cannot figure out how to do this because my terms of reference ('upload data to meteor') are ambiguous or incorrect. Or that I am an idiot.

ChristianF's answer is spot on and I have accepted it as the correct answer. However, it provides even more than I need at this stage, so I am including here the code I have actually used - which is largely taken from Christian's answer and other elements I have found as a result:
HTML UPLOAD BUTTON (I am not including drag and drop at this stage)
<template name="upload">
<input type="file" id="files" name="files[]" multiple />
<output id="list"></output>
</template>
JAVASCRIPT
Template.upload.events({
"change #files": function (e) {
var files = e.target.files || e.dataTransfer.files;
for (var i = 0, file; file = files[i]; i++) {
if (file.type.indexOf("text") == 0) {
var reader = new FileReader();
reader.onloadend = function (e) {
var text = e.target.result;
console.log(text)
var all = $.csv.toObjects(text);
console.log(all)
_.each(all, function (entry) {
Members.insert(entry);
});
}
reader.readAsText(file);
}
}
}
})
NB there is a jquery-csv library for Meteor here: https://github.com/donskifarrell/meteor-jquery-csv

I've solved this problem in the past using this gist of mine, together with this code (using the jquery-csv plugin to parse the csv data). This is done on the client side and is independent of using iron-router or not. It would be fairly straightforward to move the insertion code into a Meteor method, uploading the csv file first and then parsing and inserting the data on the server. I've tried that, too, but didn't see any performance improvement.
$(document).ready(function() {
var dd = new dragAndDrop({
onComplete: function(files) {
for (var i = 0; i < files.length; i++) {
// Only process csv files.
if (!f.type.match('text/csv')) {
continue;
}
var reader = new FileReader();
reader.onloadend = function(event) {
var all = $.csv.toObjects(event.target.result);
// do something with file content
_.each(all, function(entry) {
Items.insert(entry);
});
}
}
}
});
dd.add('upload-div'); // add to an existing div, turning it into a drop container
});
Beware though that if you are inserting a lot of entries, then you are better off turning all reactive rerendering off for a while, until all of them are inserted. Otherwise, both node on the server and the browser tab will get really slow. See my suggested solution here: Meteor's subscription and sync are slow

Related

Get undefined variable trying to update the path to an uploaded file within MongoDB using angular-file-upload and MEAN.js

If someone could help me I would be eternally grateful. I have been slamming my head against a brick wall for weeks trying to get images to upload the way it is demonstrated out of the box with the MEAN.js users module. In the generated users module the file is uploaded into a directory and the path to that file is stored in a field in the mongodb document. I can get the file to upload to where it needs to go using multer and the fileupload function. However, I cannot save the path to the field within the document. I cannot figure out how to avoid getting an 'undefined' variable. I've tried creating a $window service and passing data to it as a global variable and a bunch of other things and I'm totally stuck.
I have commented the code below to demonstrate what is going awry in my server controller changeShoePicture function.
// This is the boilerplate code from the mean.js "users" module.
// I can not create a $window service or global variable to store the
// shoe data below so that I can update the shoe.shoeImageURL field
// in MongoDB with path to the successfully uploaded file.
exports.changeShoePicture = function (req, res) {
var message = null;
var shoe = req.shoe;
var upload = multer(config.uploads.shoeUpload).single('newProfilePicture');
var profileUploadFileFilter = require(path.resolve('./config/lib/multer')).profileUploadFileFilter;
console.log('i am here', shoe); // shoe is defined here.
// Filtering to upload only images. This works and proceeds to the else condition!
upload.fileFilter = profileUploadFileFilter;
upload(req, res, function (uploadError) {
if(uploadError) {
return res.status(400).send({
message: 'Error occurred while uploading profile picture'
});
} else {
//shoe image file is successfully uploaded to the location on the server,
// However the following fails because the shoe variable is undefined.
shoe.shoeImageURL = config.uploads.shoeUpload.dest + req.file.filename;
}
});
To make sure I've got this right:
The upload function is being called on your parameters passed by your route, req and res. You set the shoe var from req.shoe.
What are the chances that upload() is messing with your req?
Drop a console.log(req) in right after you call upload and report back

Meteor: Unique MongoDB URL for different users

I'm very keen to utilize Meteor as the framework for my next project. However, there is a requirement to keep customer data separated into different MongoDB instances for users from different customers.
I have read on this thread that it could be as simple as using this:
var d = new MongoInternals.RemoteCollectionDriver("<mongo url>");
C = new Mongo.Collection("<collection name>", { _driver: d });
However, I was dished this error on my server/server.js. I'm using meteor 0.9.2.2
with meteor-platform 1.1.0.
Exception from sub Ep9DL57K7F2H2hTBz Error: A method named '/documents/insert' is already defined
at packages/ddp/livedata_server.js:1439
at Function._.each._.forEach (packages/underscore/underscore.js:113)
at _.extend.methods (packages/ddp/livedata_server.js:1437)
at Mongo.Collection._defineMutationMethods (packages/mongo/collection.js:888)
at new Mongo.Collection (packages/mongo/collection.js:208)
at Function.Documents.getCollectionByMongoUrl (app/server/models/documents.js:9:30)
at null._handler (app/server/server.js:12:20)
at maybeAuditArgumentChecks (packages/ddp/livedata_server.js:1594)
at _.extend._runHandler (packages/ddp/livedata_server.js:943)
at packages/ddp/livedata_server.js:737
Can anyone be so kind as to enlighten me whether or not I have made a mistake somewhere?
Thanks.
Br,
Ethan
Edit: This is my server.js
Meteor.publish('userDocuments', function () {
// Get company data store's mongo URL here. Simulate by matching domain of user's email.
var user = Meteor.users.findOne({ _id: this.userId });
if (!user || !user.emails) return;
var email = user.emails[0].address;
var mongoUrl = (email.indexOf('#gmail.com') >= 0) ?
'mongodb://localhost:3001/company-a-db' :
'mongodb://localhost:3001/company-b-db';
// Return documents
return Documents.getCollectionByMongoUrl(mongoUrl).find();
});
and this is the server side model.js
Documents = function () { };
var documentCollections = { };
Documents.getCollectionByMongoUrl = function (url) {
if (!(url in documentCollections)) {
var driver = new MongoInternals.RemoteCollectionDriver(url);
documentCollections[url] = new Meteor.Collection("documents", { _driver: driver });
}
return documentCollections[url];
};
Observation: The first attempt to new a Meteor.Collection works fine. I can continue to use that collection multiple times. But when I log out and login as another user from another company (in this example by using an email that is not from #gmail.com), the error above is thrown.
Downloaded meteor's source codes and peeked into mongo package. There is a way to hack around having to declare different collection names on the mongodb server based on Hubert's suggestion.
In the server side model.js, I've made these adaptation:
Documents.getCollectionByMongoUrl = function (userId, url) {
if (!(userId in documentCollections)) {
var driver = new MongoInternals.RemoteCollectionDriver(url);
documentCollections[userId] = new Meteor.Collection("documents" + userId, { _driver: driver });
documentCollections[userId]._connection = driver.open("documents", documentCollections[userId]._connection);
}
return documentCollections[userId];
};
Super hack job here. Be careful when using this!!!!
I believe Meteor distinguish its collections internally by the name you pass to them as the first argument, so when you create the "documents" collection the second time, it tries to override the structure. Hence the error when trying to create the /documents/insert method the second time.
To work around this, you could apply a suffix to your collection name. So instead of:
new Meteor.Collection('documents', { _driver: driver });
you should try:
new Meteor.Collection('documents_' + userId, { _driver: driver })

How do I store large files using Meteor?

I'm building a Meteor (meteorjs) app that needs to store and display PDF files, sometimes as large as 500Mb. GridFS doesn't seem to be integrated yet so I'm wondering if it's worth using Meteor in this case or stick to Rails.
Ideally I would not use S3 - I'd like to keep the files on my server.
UPDATE: it seems it's possible to connect outside of Meteor directly, I don't need PDFs to be automatically moved - and it likely doesn't make sense.
More specifically I'm now looking at:
MongoDB -> ElasticSearch using https://github.com/richardwilly98/elasticsearch-river-mongodb
Using the instructions at https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki
You can use GridFS inside meteor without touching any extra package
var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db; //grab the database object
var GridStore = MongoInternals.NpmModule.GridStore;
WebApp.connectHandlers.use('/someurl', function(req, res) {
var bigFile = new GridStore(db, 'bigfile.iso', 'r') //to read
bigFile.open(function(error, result) {
if (error) return
bigFile.stream(); //stream the file
bigFile.on('error', function(e) {...}) //handle error etc
bigFile.on('end', function() {bigFile.close();}); //close the file when done
bigFile.pipe(res); //pipe the file to res
});
});
However, the current GridStore/mongo (v1.3.x) used by Meteor is a bit dated, the newest verion is 2.x from http://mongodb.github.io/node-mongodb-native/2.0/api-docs/
The v1.x doesnt seem to pipe well so you may need to use the newer version
The second option
var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db; //grab the database object
var GridStore = Npm.require('mongodb').GridStore; //add Npm.depends({mongodb:'2.0.13'}) in your package.js
WebApp.connectHandlers.use('/someurl', function(req, res) {
var bigFile = new GridStore(db, 'bigfile.iso', 'r').stream(true); //the new API doens't require bigFile.open() and will close automatically on end
bigFile.on('error', function(e) {...}); //handle error etc
bigFile.on('end', function() {...});
bigFile.pipe(res); //pipe the file to res
});
In this example, I use the WebApp.connectHandlers, but of course you can use iron: router or something. I tried with a file of 500 MB and it pipes all well. You also need to set the res.writeHead(200) and other stuff such as content-type, etc

Mongoose won't remove embedded documents

I'm scratching my head here, as usual it seems with node projects, and I'm not sure if I'm doing something wrong or if I've run into a bug.
I've got a schema of Server that can have any number of embedded docs called services. I'm running into a problem though where, even though I've successfully removed the individual service from the server object, when I tell it to save it doesn't remove it from the database. The save function is working because it's saving any changes I've made and is also pushing in new embedded docs, it's just not removing one that are already there.
Here is a relatively simplified example of my code:
app.put('/server/:id', function(req, res, next){
app.Server.findOne({_id: req.params.id}, function(err, server) {
server.updated = new Date();
...
for (var num = _.size(req.body.server.services) - 1; num >= 0; num--){
// Is this a new service or an existing one
if (server.services[num]) {
// Is it marked for deletion? If so, delete it
if (req.body.server.services[num].delete == "true") {
server.services[num].remove()
} else { // else, update it
server.services[num].type = req.body.server.services[num].type
...
}
} else {
// It's new, add it
delete req.body.server.services[num]["delete"]
server.services.push(req.body.server.services[num]);
}
}
server.save(function(err){
if (!err) {
req.flash('success', 'Server updated')
} else {
req.flash('error', 'Err, Something broke when we tried to save your server. Sorry!')
console.log(err)
}
res.redirect('/')
});
})
});
So the remove() is actually removing the service. If I do a server.toObject() before the save, it's not there. Any ideas why it wouldn't be removing it from the database when it saves?
Edit: I suppose the version numbers would be helpful. node#0.4.2, mongoose#1.1.5 express#2.0.0rc
I could be wrong, since I've not tested your example, but this sounds like Mongoose isn't detecting that the embedded document is modified.
From the schema types documentation page:
Since it is a schema-less type, you can change the value to anything else you like, but Mongoose loses the ability to auto detect/save those changes. To "tell" Mongoose that the value of a Mixed type has changed, call the .markModified(path) method of the document passing the path to the Mixed type you just changed.
person.anything = { x: [3, 4, { y: "changed" }] };
person.markModified('anything');
person.save(); // anything will now get saved
So you answer might be as simple as using the markModified() function.
I found a way to temporary fix this problem.
What I did is load the embedded documents into an array, splice the one to be deleted and replace the array. Something like this:
var oldusers = dl.users;
oldusers.splice(dl.users.indexOf(req.currentUser.id), 1);
dl.users = oldusers;
dl.save(function(err) {...
I know that depending on the size of the document it will

ADO.NET Data Services - Uploading files

I am trying to write REST web service through which our clients can upload a file on our file server. IS there an example or any useful links which I can refer for any guidance?
I haven't seen many examples of POST operation using ADO.NET data services available.
I've uploaded a file to ADO.NET dataservices using POST although I'm not sure whether it's the recommended approach. The way I went about it is:
On the dataservice I've implemented a service operation called UploadFile (using the WebInvoke attribute so that it caters for POST calls):
[WebInvoke]
public void UploadFile()
{
var request = HttpContext.Current.Request;
for (int i = 0; i < request.Files.Count; i++)
{
var file = request.Files[i];
var inputValues = new byte[file.ContentLength];
using (var requestStream = file.InputStream)
{
requestStream.Read(inputValues, 0, file.ContentLength);
}
File.WriteAllBytes(#"c:\temp\" + file.FileName, inputValues);
}
}
Then on the client side I call the data service using:
var urlString = "http://localhost/TestDataServicePost/CustomDataService.svc/UploadFile";
var webClient = new WebClient();
webClient.UploadFile(urlString, "POST", #"C:\temp\test.txt");
This uses a WebClient to upload the file which places the file data in the HttpRequest.Files collection and sets the content type. If you would prefer to send the contents of the file yourself (eg from an Asp FileUpload control) rather than the webClient reading a file using a path to the file, you can use a WebRequest similar to the way that it's done in this post. Although instead of using
FileStream fileStream = new FileStream(uploadfile,
FileMode.Open, FileAccess.Read);
you could use a byte array that you pass in.
I hope this helps.
I'm not 100% sure how to do this directly to a file server per se, but ADO.Net Data Services definitely support something similar to a database. The code below is how a similar goal of putting a file into a database has been accomplished. Not sure how much that will help, but
var myDocumentRepositoryUri = new Uri("uri here");
var dataContext = new FileRepositoryEntities(myDocumentRepositoryUri);
var myFile = new FileItem();
myfile.Filename = "upload.dat";
myFile.Data = new byte[1000]; // or put whatever file data you want to here
dataContext.AddToFileItem(myFile);
dataContext.SaveChanges();
Note: this code is also using Entity Framework to create a FileItem (representation of a database table as an object) and to save that data.