I'm working on my first node.js / express / mongoose app and I'm facing a problem due to asynchronisation mechanism of node.js. It seems I do not do the thing correctly...
Here is the test route I defined using express:
app.get('/test', function(req, res){
var mod = mongoose.model('MyModel');
mod.find({},function(err, records){
records.forEach(function(record){
console.log('Record found:' + record.id);
// res.send('Thing retrieved:' + record.id);
});
});
});
When I issue a http://localhost/test, I'd like to get the list of records of type 'MyModel' in the response.
The code above is working fine but when it comes to returning this whole list to the client... it does not work (the commented res.send line) and only returned the first record.
I'm very new to node.js so I do not know if it's the good solution to embed several callback functions within the first callback function of app.get . How could I have the whole list returned ?
Any idea ?
What you should be doing is:
mod.find({},function(err, records){
res.writeHead(200, {'Content-Length': body.length});
records.forEach(function(record){
res.write('Thing retrieved:' + record.id);
});
});
Please always check the documentation as well:
http://nodejs.org/docs/v0.3.8/api/http.html#response.write
I missed that you was using express, the send function is part of express and extend's the serverResponse object of node (my bad).
but my answer still applies, express's send function sends the data using ServerResponse.end() so there for the socket get's closed and you cannot send data any more, using the write function uses the native function.
You may also want to call res.end() when the request is fully completed as some item's within express may be affected
Related
I didn't find very much about this topic, so I wonder if it is an easy task to achieve or if it's actually not possible. My problem is that I have a lot of HTTP requests on my server even if a Cloud function is called only once. So I suppose that all the object updating / savings / queries are made by using the REST API. I have so many HTTP requests that several hundred are going timeout, I suppose for the huge traffic that it's generated.
Is there a way to save a ParseObject by executing the query directly to MongoDB? If it's not possible at the moment can you give me some hints if there are already some helper functions to convert a ParseQuery and a ParseObject to the relative in MongoDB so that I can use the MongoDB driver directly?
It's really important for my application to reduce HTTP requests traffic at the moment.
Any idea? Thanks!
EDIT:
Here an example to reproduce the concept:
Make a cloud function:
Parse.Cloud.define('hello', async (req, res) => {
let testClassObject = new Parse.Object('TestClass');
await testClassObject.save(null, {useMasterKey: true});
let query = new Parse.Query('TestClass');
let testClassRecords = await query.find({useMasterKey: true});
return testClassRecords;
});
Make a POST request:
POST http://localhost:1337/parse/functions/hello
Capture HTTP traffic on port 1337 using Wireshark:
You can see that for 1 POST request other 2 are made because of the saving / query code. My goal would be to avoid these two HTTP calls and instead make a DB call directly so that less traffic will go through the whole webserver stack.
Link to the Github question: https://github.com/parse-community/parse-server/issues/6549
The Parse Server directAccess option should do the magic for you. Please make sure you are initializing Parse Server like this:
const api = new ParseServer({
...
directAccess: true
});
...
I'm currently running SailsJS on a Raspberry Pi and all is working well however when I execute a sails.models.nameofmodel.count() when I attempt to respond with the result I end up getting a empty response.
getListCount: function(req,res)
{
var mainsource = req.param("source");
if(mainsource)
{
sails.models.gatherer.find({source: mainsource}).exec(
function(error, found)
{
if(error)
{
return res.serverError("Error in call");
}
else
{
sails.log("Number found "+found.length);
return res.ok({count: found.length});
}
}
);
}
else
{
return res.ok("Error in parameter");
}
},
I am able to see in the logs the number that was found (73689). However when responding I still get an empty response. I am using the default stock ok.js file, however I did stick in additional logging to try to debug and make sure it is going through the correct paths. I was able to confirm that the ok.js was going through this path
if (req.wantsJSON) {
return res.jsonx(data);
}
I also tried adding .populate() to the call before the .exec(), res.status(200) before I sent out a res.send() instead of res.ok(). I've also updated Sails to 11.5 and still getting the same empty response. I've also used a sails.models.gatherer.count() call with the same result.
You can try to add some logging to the beginning of your method to capture the value of mainsource. I do not believe you need to use an explicit return for any response object calls.
If all looks normal there, try to eliminate the model's find method and just evaluate the request parameter and return a simple response:
getListCount: function(req, res) {
var mainsource = req.param("source");
sails.log("Value of mainsource:" + mainsource);
if (mainsource) {
res.send("Hello!");
} else {
res.badRequest("Sorry, missing source.");
}
}
If that does not work, then your model data may not actually be matching on the criteria that you are providing and the problem may lie there; in which case, your response would be null. You mentioned that you do see the resulting count of the query within the log statement. If the res.badRequest is also null, then you may have a problem with the version of express that is installed within sailsjs. You mention that you have 11.5 of sailsjs. I will assume you mean 0.11.5.
This is what is found in package.json of 0.11.5
"express": "^3.21.0",
Check for any possible bugs within the GitHub issues for sailsjs regarding express and response object handling and the above version of express.
It may be worthwhile to perform a clean install using the latest sailsjs version (0.12.0) and see if that fixes your issue.
Another issue may be in how you are handling the response. In this case .exec should execute the query immediately (i.e. a synchronous call) and return the response when complete. So there should be no asynchronous processing there.
If you can show the code that is consuming the response, that would be helpful. I am assuming that there is a view that is showing the response via AJAX or some kind of form POST that is being performed. If that is where you are seeing the null response, then perhaps the problem lies in the view layer rather than the controller/model.
If you are experiencing a true timeout error via HTTP even though your query returns with a result just in time, then you may need to consider using async processing with sailjs. Take a look at this post on using a Promise instead.
I have a large mongoDB database set up and am trying to create a website where a user can use a searchbar to query the database remotely and have the results posted on the site (strictly read-only).
I have experience with databases for data analysis, but have never created a website for querying results.
I'm don't have any experience with web development and don't know what platforms (PHP? node.js?) to use.
Thanks in advance.
There are the following steps to the problem:
Create the front-end, which will consist of HTML, CSS, and Javascript. Beginners often find it easiest to work with jQuery and jQuery UI, because they are well-documented and contain plugins for almost all possible scenarios (they should not, however, be used to create large complex applications!). Bootstrap or Foundation can help you with the HTML / CSS.
Create a (probably) JSON API, which the front-end can communicate with to submit searches and retrieve results. You could use PHP, Python, Ruby, or many other languages to do this. For a simple site like the one you're describing, it's more a matter of preference than anything else.
Translate the search request from the front-end into the MongoDB query APIs, and return the results through the API. You will use a MongoDB client library compatible with whatever language you have chosen.
Depending on your needs, you may be able to eliminate (2) by using an existing REST API for MongoDB.
Note that if you just want to make MongoDB data accessible via searching / charting, then you may be able to avoid coding altogether by leveraging SlamData, an open source project I contribute to. SlamData lets you use Google-style search (or more advanced SQL) to query MongoDB and get the results back in tabular or chart form.
I am doing such in nodejs.
In my server side app I have connection via mognoose like:
var mongoose = require('mongoose');
mongoose.connect('mongodb://yourhost/database');
Next you need to have your model to your database
var YourDBVarName = mongoose.model('collectionName', {
yourfields1 : type,
yourfields2 : type,
yourfields3 : type
...
});
Then I make GET for it
var express = require('express');
var app = express();
app.get('/dblisting', function(req,res){
YourDBVarName.find({ yourfieldsX: 'value'}, function(err, data) {
if(err) {
res.send(err.message);
}
else{
res.send(data);
});
});
Then simply you make some GET with $.ajax to yournodeserver.com/dblisting and in response you recive your collection filtered as in
{ yourfieldsX: 'value'}
Ofcourse you may do just {} so you get all your stored data.
SLee
If you want know about retrieving data from mongoDB, you can use my github https://github.com/parthaindia/CustomMongo .Use getByCondition() method which requires collection name and a Map . The Map can be your queries in form of key value pair,Key being the column name. I use this method when I write search Query for the web development. The java code gives you a Json. Write a Servlet to post your Json to WEB UI.
This is an example to show how to post the retrieved data using Js ,"server_base_url + /server/FetchData" would be your Service URL.The data you has to be appended to a table . Or a span ,depends on what you actually want.The below code appends data
function getdata() {
$.get(server_base_url + "/server/FetchData", {
}).done(function (data) {
$.each(data, function (index, value) {
alert("The Index" + index + "The Value" + value);
$("#11table1").append("<tr><td id='dynamicid1" + index + "'>" + value + "</td></tr>");
});
});
}
This function can be used for defining table
function defineTable() {
$("#mainDivID").text("").append("<div id='contpanel' class='contentpanel'>");
$("#contpanel").append("<div id='rowid11' class='row'>");
$("#rowid11").text("").append("<div id='row11table1' class='col-md-12'>");
$("#row11table1").text("").append('<br /><br /><center><h5 class="lg-title mb5" style="background-color:#428BCA;height:20px;color:#fff;padding-top:4px;"><b>Heading</b></h5></center>');
$("#row11table1").append("<div id='table11id1' class='table-responsive'>");
$("#table11id1").append("<table id='11table1' class='table table table-bordered mb30' >");
$("#11table1").append("<thead><tr><th>Index</th><th>Value</th></tr></thead>");
}
I have been reading a lot lately, and even more experimenting with web Development. There are some things that I simply cant understand, therefore any help is appreciated.
I am not trying to get my homework done for me. I have some holes in my knowledge, that I desire to fill. Please, help me out with your views :)
REST questions:
Reading documentation this is perfectly understandable (NODE.JS / Express) example:
EXAMPLE ONE (get):
app.get('/', function(req, res) {
res.send('please select a collection, e.g., /collections/messages')
})
My explanation: When the root of the server is hit, send thie following message
EXAMPLE TWO (get):
app.get('/collections/:collectionName/:id', function(req, res) {
req.collection.findOne({name: req.collection.id(req.params.id)},
function(e, result){
if (e) return next(e)
res.send(result)
})
})
My explanation: When the url in hit, take id from the URL (that is located in params.id) and make search based on it (that is MongoDB).
EXAMPLE THREE (post):
app.post('/collections/:collectionName', function(req, res) {
req.collection.insert(req.body, {}, function(e, results){
if (e) return next(e)
res.send(results)
})
})
My explanation: When the URL is hit, take the payload(JSON in this case) that is located in req.body, and insert it as a new document.
Questions:
Are example one and two both RESTfull?
I am now totally confused with params.id. I do understand that POST is transmitted in rew.body... what is params.id? Is it containing URL variables, such as :ID?
My explanations... are they correct?
Example three is also REST, regardless of the fact that POST is used?
Example three, '/collections/:collectionName. Why is the ':collectionName' passed in URL, I could have placed it in req.body as a parameter (along with new data) and take it from there? What is the benefit of doing it?
Thank you
An API must be using HATEOAS to be RESTful. On first example, if / is the entry point of your API, the response should contain links for the available collections, not a human readable string like that. That's definitely not RESTful.
Exactly.
They're OK, except that there's nothing in the third example implying it's a JSON body. It should check for a Content-Type header sent by the client.
REST isn't dependent on HTTP. As long as you're using the HTTP methods as they were standardized, it's fine. POST is the method to use for any action that isn't standardized, so it's fine to use POST for anything, if there isn't a method specific for that. For instance, it's not correct to use POST for retrieval, but it's fine to use it for creating a new resource if you don't have the full representation.
POST means the data body is subordinated to the resource at the target URI. If collectionName were in the POST body, this would mean you were POSTing to /collections, which would make more sense to create a new collection, not a new item of a collection.
I've got a node.js application that 'streams' tweets to users. At the moment, it just searches Twitter for a hard-coded string, but I'd like to allow users to configure this in the URL (eg. by visiting /?q=stackoverflow).
At the moment, my code looks a bit like this:
app.get('/', function (req, res) {
// page rendering skipped
io.sockets.on('connection', function (socket) {
twit.stream('user', {track: 'stackoverflow'}, function(stream) {
stream.on('data', function (data) {
socket.volatile.emit('tweet', data);
}
});
});
});
The question is, how do I make it so that each user can see a different stream of tweets simultaneously? At the moment, it works fine in a single browser tab, but it falls over as soon as a second one is opened - and the error is fairly deep down inside socket.io. Am I misusing it?
I haven't fully got my head around socket.io yet, so that could be the issue.
Thanks in advance!
Every time a new request comes in, you are redefining the connection callback with io.sockets.on - you should move that block of code outside of app.get, after your initialization statement of the io object.