NodeJS Node-apn implementation as daemon - iphone

I have a node-apn nodejs script running as a daemon on AmazonWS. The daemon runs fine and the script stays up and comes back when it goes down but I believe I am having a synchronous execution and exiting issue with node.js. When I release the process with process.exit(); even though all console.logs output saying they have sent my messages, they never are received on the phone. I decided to remove the exit and let the process "hang" after execution and all messages were sent successfully. This led me to do the following implementation using an ASYNC function, but the same result seems to be happening. Can anyone provide insight to this? There are no errors being thrown from APN or anywhere else.
function closeDB()
{
connection.end(function(err) {
if (err) {
console.log("ERROR: " + util.inspect(err, false, 5));
process.exit(1);
}
console.log("APNS-PUSH: COMPLETED.");
});
setTimeout(function(){process.exit();}, 50);
} // End of closeDB()
function apnsError(err, notification)
{
console.log(err);
console.log(notification);
closeDB();
}
function async(arg, callback)
{
apnsConnection.sendNotification(arg);
console.log(arg);
setTimeout(function() { callback(1); }, 100);
}
/**
* Our MySQL query callback.
*/
function queryCB(err, results)
{
//error in our all, report and exit
if (err) {
console.log("ERROR: " + util.inspect(err, false, 5));
closeDB();
}
if(results.length == 0)
{
closeDB();
}
var notes = [];
var count = 0;
try {
for( var i = 0; i < results.length; i++ ) {
var myDevice = new apns.Device(results[i]['udid']);
var note = new apns.Notification();
note.expiry = Math.floor(Date.now() / 1000) + 3600; // Expires 1 hour from now.
note.badge = results[i]["notification_count"];
note.sound = "ping.aiff";
note.alert = results[i]["message"];
note.device = myDevice;
connection.query('UPDATE `tbl_notifications` SET `sent`=1 WHERE `id`=' + results[i]["id"] , function(err, results) {
if(err)
{
console.log("ERROR: " + util.inspect(err, false, 5));
}
});
notes.push(note);
}
} catch( err ) {
console.log('error: ' + err)
}
console.log(notes.length);
notes.forEach(function(nNode) {
async(nNode, function(result) {
count++;
if(count == notes.length) {
closeDB();
}
})
});
} // End of queryCB()

I had the same problem where killing the process also killed the open socket connections and didn't allow the notifications to be sent. The solution I came up with isn't an an ideal solution but it will work in your situation as well. I looked into the node-apn code and found that the Connection object inherited from EventEmitter so you can monitor events on the object like so:
var apnsConnection = new apn.Connection(options)
apnsConnection.sendNotification(notification)
apnsConnection.on('transmitted', function(){
console.log("Transmitted")
callback()
})
apnsConnection.on('error', function(){
console.log("Error")
callback()
})
This is monitoring the socket that the notification is sent through so I don't know how accurate it is at determining when a notification has successfully been passed off to Apple's APNS servers but it has worked pretty well for me.

The reason you are seeing this problem is that when you use #pushNotification it buffers the notification inside the module and handles sending it asynchronously.
Listening for "transmitted" is valid and this is emitted when the notification has been written to the socket. However, if your objective is to close the socket after all notifications have been sent then the easiest way to accomplish this is using the connectionTimeout property when creating your connection.
Simply set connectionTimeout to something around 1000 (milliseconds) and assuming you have no other connections open then the process will exit automatically. Or you can set an event listener on the timeout event and call process.exit() from there.

Related

Make promise calls with a fixed timeout

I am currently trying to make it check for a database connection. but it seems like the result, ie connection is pending.
I am looking to impliment a system where i can send an timeout input, where the promise would be rejected due to a fixed timeout.
Something like;
try {
start(timeout: 6000) // 6 secs timeout on promise. default, ie no params: 3sec
} catch(e) {
// failed due to, in this case, timeout, since there is no connection running, and the database is pending their promise.
}
How can i accomplish such timeout?
current running example gives the following:
connected to mongo Promise { <pending> }
starting server code:
const start = async () => {
console.log("connecting to database");
try {
console.log("Attempting to establish connection....");
var result = await mongoose.connect('mongodb://localhost/db',{useNewUrlParser: true});
console.log("connected to mongo",result);
return result;
} catch (error) {
console.log("failed to connect to mongo",error);
}
}
try {
start();
} catch(e) {
console.log("failed to start server",e);
throw new Error("failed to start server",e);
}

Netty Client: request-response in same future

I'm trying to implement a Netty client which will send a TCP request and receive a response in the same Future. The code is below
public void sendMessage(String msg) {
Future<Channel> future = simpleChannelPool.acquire();
future.addListener((FutureListener<Channel>) f -> {
if (f.isSuccess()) {
Channel ch = f.getNow();
ChannelFuture channelFuture = ch.writeAndFlush((msg + " " + now() + lineSeparator()));
channelFuture.addListener(writeFuture -> {
if (writeFuture.isSuccess()) {
System.out.println("Channel write successful");
// how to READ data here?
}
});
// Release back to pool
simpleChannelPool.release(ch);
} else {
System.out.println("Failed to acquire a channel");
}
});
}
The part that I'm missing is how to read the data after successful write, line if (writeFuture.isSuccess()) {.... Basically I need to wait until next read is finished.
Also, given Netty's asynchronous nature, is it a right tool for request/response protocols?
Netty version used: 4.1.39.Final.
The complete code is available here.

How to set username and password in mongdb program? [duplicate]

Working with Nodejs and MongoDB through Node MongoDB native driver. Need to retrieve some documents, and make modification, then save them right back. This is an example:
db.open(function (err, db) {
db.collection('foo', function (err, collection) {
var cursor = collection.find({});
cursor.each(function (err, doc) {
if (doc != null) {
doc.newkey = 'foo'; // Make some changes
db.save(doc); // Update the document
} else {
db.close(); // Closing the connection
}
});
});
});
With asynchronous nature, if the process of updating the document takes longer, then when cursor reaches the end of documents, database connection is closed. Not all updates are saved to the database.
If the db.close() is omitted, all the documents are correctly updated, but the application hangs, never exits.
I saw a post suggesting using a counter to track number of updates, when fall back to zero, then close the db. But am I doing anything wrong here? What is the best way to handle this kind of situation? Does db.close() have to be used to free up resource? Or does a new db connection needs to open?
Here's a potential solution based on the counting approach (I haven't tested it and there's no error trapping, but it should convey the idea).
The basic strategy is: Acquire the count of how many records need to be updated, save each record asynchronously and a callback on success, which will decrement the count and close the DB if the count reaches 0 (when the last update finishes). By using {safe:true} we can ensure that each update is successful.
The mongo server will use one thread per connection, so it's good to either a) close unused connections, or b) pool/reuse them.
db.open(function (err, db) {
db.collection('foo', function (err, collection) {
var cursor = collection.find({});
cursor.count(function(err,count)){
var savesPending = count;
if(count == 0){
db.close();
return;
}
var saveFinished = function(){
savesPending--;
if(savesPending == 0){
db.close();
}
}
cursor.each(function (err, doc) {
if (doc != null) {
doc.newkey = 'foo'; // Make some changes
db.save(doc, {safe:true}, saveFinished);
}
});
})
});
});
It's best to use a pooled connection and then call db.close() in cleanup function at the end of your application's life:
process.on('SIGINT', cleanup);
process.on('SIGTERM', cleanup);
See http://mongodb.github.io/node-mongodb-native/driver-articles/mongoclient.html
A bit old thread, but anyway.
Here an extended example to the answer given by pkopac, since I had to figure out the rest of the details:
const client = new MongoClient(uri);
(async () => await client.connect())();
// use client to work with db
const find = async (dbName, collectionName) => {
try {
const collection = client.db(dbName).collection(collectionName);
const result = await collection.find().toArray()
return result;
} catch (err) {
console.error(err);
}
}
const cleanup = (event) => { // SIGINT is sent for example when you Ctrl+C a running process from the command line.
client.close(); // Close MongodDB Connection when Process ends
process.exit(); // Exit with default success-code '0'.
}
process.on('SIGINT', cleanup);
process.on('SIGTERM', cleanup);
Here is a link to the difference between SIGINT and SIGTERM.
I had to add the process.exit(), otherwise my node web-server didn't exit cleanly when doing Ctrl + C on the running process in command line.
I found that using counter may apply to simple scenario, but may be hard in complicated situations. Here is a solution that I come up by closing the database connection when database connection is idle:
var dbQueryCounter = 0;
var maxDbIdleTime = 5000; //maximum db idle time
var closeIdleDb = function(connection){
var previousCounter = 0;
var checker = setInterval(function(){
if (previousCounter == dbQueryCounter && dbQueryCounter != 0) {
connection.close();
clearInterval(closeIdleDb);
} else {
previousCounter = dbQueryCounter;
}
}, maxDbIdleTime);
};
MongoClient.connect("mongodb://127.0.0.1:27017/testdb", function(err, connection)(
if (err) throw err;
connection.collection("mycollection").find({'a':{'$gt':1}}).toArray(function(err, docs) {
dbQueryCounter ++;
});
//do any db query, and increase the dbQueryCounter
closeIdleDb(connection);
));
This can be a general solution for any database Connections. maxDbIdleTime can be set as the same value as db query timeout or longer.
This is not very elegant, but I can't think of a better way to do this. I use NodeJs to run a script that queries MongoDb and Mysql, and the script hangs there forever if the database connections are not closed properly.
Here's a solution I came up with. It avoids using toArray and it's pretty short and sweet:
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect("mongodb://localhost:27017/mydb", function(err, db) {
let myCollection = db.collection('myCollection');
let query = {}; // fill in your query here
let i = 0;
myCollection.count(query, (err, count) => {
myCollection.find(query).forEach((doc) => {
// do stuff here
if (++i == count) db.close();
});
});
});
I came up with a solution that involves a counter like this. It does not depend on a count() call nor does it wait for a time out. It will close the db after all the documents in each() are exhausted.
var mydb = {}; // initialize the helper object.
mydb.cnt = {}; // init counter to permit multiple db objects.
mydb.open = function(db) // call open to inc the counter.
{
if( !mydb.cnt[db.tag] ) mydb.cnt[db.tag] = 1;
else mydb.cnt[db.tag]++;
};
mydb.close = function(db) // close the db when the cnt reaches 0.
{
mydb.cnt[db.tag]--;
if ( mydb.cnt[db.tag] <= 0 ) {
delete mydb.cnt[db.tag];
return db.close();
}
return null;
};
So that each time you are going to make a call like db.each() or db.save() you would use these methods to ensure the db is ready while working and closed when done.
Example from OP:
foo = db.collection('foo');
mydb.open(db); // *** Add here to init the counter.**
foo.find({},function(err,cursor)
{
if( err ) throw err;
cursor.each(function (err, doc)
{
if( err ) throw err;
if (doc != null) {
doc.newkey = 'foo';
mydb.open(db); // *** Add here to prevent from closing prematurely **
foo.save(doc, function(err,count) {
if( err ) throw err;
mydb.close(db); // *** Add here to close when done. **
});
} else {
mydb.close(db); // *** Close like this instead. **
}
});
});
Now, this assumes that the second to last callback from each makes it through the mydb.open() before the last callback from each goes to mydb.close().... so, of course, let me know if this is an issue.
So: put a mydb.open(db) before a db call and put a mydb.close(db) at the return point of the callback or after the db call (depending on the call type).
Seems to me that this kind of counter should be maintained within the db object but this is my current workaround. Maybe we could create a new object that takes a db in the constructor and wrap the mongodb functions to handle the close better.
Based on the suggestion from #mpobrien above, I've found the async module to be incredibly helpful in this regard. Here's an example pattern that I've come to adopt:
const assert = require('assert');
const async = require('async');
const MongoClient = require('mongodb').MongoClient;
var mongodb;
async.series(
[
// Establish Covalent Analytics MongoDB connection
(callback) => {
MongoClient.connect('mongodb://localhost:27017/test', (err, db) => {
assert.equal(err, null);
mongodb = db;
callback(null);
});
},
// Insert some documents
(callback) => {
mongodb.collection('sandbox').insertMany(
[{a : 1}, {a : 2}, {a : 3}],
(err) => {
assert.equal(err, null);
callback(null);
}
)
},
// Find some documents
(callback) => {
mongodb.collection('sandbox').find({}).toArray(function(err, docs) {
assert.equal(err, null);
console.dir(docs);
callback(null);
});
}
],
() => {
mongodb.close();
}
);
Modern way of doing this without counters, libraries or any custom code:
let MongoClient = require('mongodb').MongoClient;
let url = 'mongodb://yourMongoDBUrl';
let database = 'dbName';
let collection = 'collectionName';
MongoClient.connect(url, { useNewUrlParser: true }, (mongoError, mongoClient) => {
if (mongoError) throw mongoError;
// query as an async stream
let stream = mongoClient.db(database).collection(collection)
.find({}) // your query goes here
.stream({
transform: (readElement) => {
// here you can transform each element before processing it
return readElement;
}
});
// process each element of stream (async)
stream.on('data', (streamElement) => {
// here you process the data
console.log('single element processed', streamElement);
});
// called only when stream has no pending elements to process
stream.once('end', () => {
mongoClient.close().then(r => console.log('db successfully closed'));
});
});
Tested it on version 3.2.7 of mongodb driver but according to link might be valid since version 2.0

Streaming from mongodb in AWS lambda times out

I have a lambda function which connects to a mongodb database and streams some records from the database.
exports.handler = (event, context, callback) => {
let url = event.mongodbUrl;
let collectionName = event.collectionName;
MongoClient.connect(url, (error, db) => {
if (error) {
console.log("Error connecting to mongodb: ${error}");
callback(error);
} else {
console.log("Connected to mongodb");
let events = [];
console.log("Streaming data from mongodb...");
let mongoStream = db.collection(collectionName).find().sort({ _id : -1 }).limit(500).stream();
mongoStream.on("data", data => {
events.push(data);
});
mongoStream.once("end", () => {
console.log("Stream ended");
db.close(() => {
console.log("Database connection closed");
callback(null, "Lambda function succeeded!!");
});
});
}
});
};
When the stream is ended I close the database connection and call the callback function which should end the lambda function. This works locally using node-lambda, but when I try to run it in AWS lambda I get all of the logs, including console.log("Database connection closed"); coming through, but the callback doesn't seem to be called, so the function always times out, despite the last log occurring a few seconds before the time out.
I can force it to end using context.succeed(), but that seems to be deprecated when using node version 4, so I want to avoid using it. How can I stop this function from timing out in AWS lambda?
Add the following line at the beginning of your handler function:
context.callbackWaitsForEmptyEventLoop = false
Try following:
mongoStream.once("end", callback);
This is also calling back with err and result but will not lose the context.

Finagle No asyncronous executing

i have a simple finagle thrift server:
import com.twitter.finagle.Thrift
import scala.concurrent.Future
import com.twitter.util.{ Await, Future }
object Main{
def main(args: Array[String]) {
var count = 0
val myserver = Thrift.serveIface("0.0.0.0:9090", new RealTimeDatabasePageImpressions[com.twitter.util.Future] {
def saveOrUpdate(pageImpression: PageImpressions):
com.twitter.util.Future[Boolean] = {
count += 1
println(count)
com.twitter.util.Future.value(true)
}
}
Await.ready(myserver)
}
}
This server works but i have one big problem: i wrote a thrift nodejs client with a for loop. It executes 10.000 thrift request. But it's not asynchronous. It executes 500 request and stops. After a while, 2 or 3 seconds, 300 more requests will executed. Now the question: Why this happen? Is something wrong with my server or client? I use only the apache thrift generated nodejs code. No wrapper. The function executed 10.000 times. I think the nodejs isn't the problem:
function callFunc(i){
console.log("started executing: " + i);
var connection = thrift.createConnection("IP", 9090, {
transport: transport,
protocol: protocol
});
connection.on('error', function (err) {
console.log(err);
});
// Create a Calculator client with the connection
var client = thrift.createClient(Realtime_pageImpressions, connection);
var rand = Math.random() * (20000 - 1);
var trackId = trackIds[Math.round(Math.random() * 10)];
var values = new PageImpressions({
trackId: trackId,
day: 4,
hour: 4,
minute: 13,
pageId: 'blabla',
uniqueImpressions: Math.random() * (13000 - 1),
sumImpressions: Math.random() * (1000450 - 1)
});
client.saveOrUpdate(values, function (error, message) {
if (message) {
console.log("Successful, got Message: " + message);
} else {
console.log("Error with Message: " + error);
}
});
return true;
}
for(var i = 0; i < 10000; i++){
callFunc(i);
}
Your var count is unsynchronized. This is a very big problem, but, probably, not related to your performance issue.
You are also blocking finagle thread, which is also a big problem, but does not matter in your mock case, because there is no wait time.
Think about it this way. Let's say, you have one cpu (you probably have several, but there are other things going on the machine as well), and you are asking it to execute 10000 operations all at the same time.
How can this work? It will have to execute one of the requests, save the context, the stack, flush all caches, switch to the next request, execute that one ...
500 requests in 2 seconds is 4 milliseconds per request. Does not sound that bad, does it?
Also, have you turned your GC (on both server and client)? If requests are processed in bursts followed by long pauses, that's probably a sign of full GC kicking in