I have a regular TCP socket connection between a Node.js application (using standard net.Socket), and some other application.
Suppose I send a 1MB buffer:
socket.write(new Buffer(1048576));
While this is done in a non-blocking manner, obviously the data is not transmitted instantaneously.
How can I measure the data rate that the underlying system is sending the internal stream buffer, from my Node.js application?
Ultimately I just need to know the average speed of data being sent to the client on the other end over the last several seconds. Is this possible?
try to use callback in write
var net = require('net');
var s = net.createConnection(8888);
s.on('connect', function() {
function test() {
var len = 512*1048576;
var start = +new Date();
var b = new Buffer(len);
b.fill('x');
s.write(b, function() {
console.log(len + ' bytes written, ' + (1000*len/(+new Date() - start)).toString() + ' bytes/sec');
test();
});
}
test();
});
with nc -l 8888 > /dev/null on the other end I get around 350M bytes/sec
Related
I need to convert my ScriptProcessor logic to an AudioWorklet. The ScriptProcessor process is very easy and only a few lines of code, but I'm having a hard time converting it over to an AudioWorklet.
Here is my current process (the stream comes from userMedia):
var context = new AudioContext();
var microphone = context.createMediaStreamSource(stream);
var processor = context.createScriptProcessor(0, 1, 1);
microphone.connect(processor);
processor.connect(context.destination);
and then I send the data to a worker:
processor.onaudioprocess = function (event) {
var array = event.inputBuffer.getChannelData(0);
realTimeWorker.postMessage({cmd: 'encode', buf: array})
};
I just need the AudioWorklet to process the data in the same way the ScriptProcessor does it, and then send the same data to the worker.
How could I convert this process to an AudioWorklet?
i have a simple finagle thrift server:
import com.twitter.finagle.Thrift
import scala.concurrent.Future
import com.twitter.util.{ Await, Future }
object Main{
def main(args: Array[String]) {
var count = 0
val myserver = Thrift.serveIface("0.0.0.0:9090", new RealTimeDatabasePageImpressions[com.twitter.util.Future] {
def saveOrUpdate(pageImpression: PageImpressions):
com.twitter.util.Future[Boolean] = {
count += 1
println(count)
com.twitter.util.Future.value(true)
}
}
Await.ready(myserver)
}
}
This server works but i have one big problem: i wrote a thrift nodejs client with a for loop. It executes 10.000 thrift request. But it's not asynchronous. It executes 500 request and stops. After a while, 2 or 3 seconds, 300 more requests will executed. Now the question: Why this happen? Is something wrong with my server or client? I use only the apache thrift generated nodejs code. No wrapper. The function executed 10.000 times. I think the nodejs isn't the problem:
function callFunc(i){
console.log("started executing: " + i);
var connection = thrift.createConnection("IP", 9090, {
transport: transport,
protocol: protocol
});
connection.on('error', function (err) {
console.log(err);
});
// Create a Calculator client with the connection
var client = thrift.createClient(Realtime_pageImpressions, connection);
var rand = Math.random() * (20000 - 1);
var trackId = trackIds[Math.round(Math.random() * 10)];
var values = new PageImpressions({
trackId: trackId,
day: 4,
hour: 4,
minute: 13,
pageId: 'blabla',
uniqueImpressions: Math.random() * (13000 - 1),
sumImpressions: Math.random() * (1000450 - 1)
});
client.saveOrUpdate(values, function (error, message) {
if (message) {
console.log("Successful, got Message: " + message);
} else {
console.log("Error with Message: " + error);
}
});
return true;
}
for(var i = 0; i < 10000; i++){
callFunc(i);
}
Your var count is unsynchronized. This is a very big problem, but, probably, not related to your performance issue.
You are also blocking finagle thread, which is also a big problem, but does not matter in your mock case, because there is no wait time.
Think about it this way. Let's say, you have one cpu (you probably have several, but there are other things going on the machine as well), and you are asking it to execute 10000 operations all at the same time.
How can this work? It will have to execute one of the requests, save the context, the stack, flush all caches, switch to the next request, execute that one ...
500 requests in 2 seconds is 4 milliseconds per request. Does not sound that bad, does it?
Also, have you turned your GC (on both server and client)? If requests are processed in bursts followed by long pauses, that's probably a sign of full GC kicking in
I need to send several commands over telnet to a server. If I try to send them without a time delay between every command, the server freaks out:
var net = require('net');
var conn = net.createConnection(8888, 'localhost');
conn.on('connect', function() {
conn.write(command_1);
conn.write(command_2);
conn.write(command_3);
//...
conn.write(command_n);
})
I guess the server needs some time to respond to command n before I send it command n+1. One way is to write something to the log and fake a "wait":
var net = require('net');
var conn = net.createConnection(8888, 'localhost');
conn.on('connect', function() {
console.log('connected to server');
console.log('I'm about to send command #1');
conn.write(command_1);
console.log('I'm about to send command #2');
conn.write(command_2);
console.log('I'm about to send command #3');
conn.write(command_3);
//...
console.log('I'm about to send command #n');
conn.write(command_n);
})
It might also be the fact that conn.write() is asynchronous, and putting one command after another doesn't guranty the correct order??
Anyway, what is the correct pattern to assure correct order and enough time between two consecutive commands, for the server to respond?
First things first: if this is truly a telnet server, then you should do something with the telnet handshaking (where terminal options are negotiated between the peers, this is the binary data you can see when opening the socket).
If you don't want to get into that (it will depend on your needs), you can ignore the negotiation and go straight to business, but you will have to read this data and ignore it yourself.
Now, in your code, you're sending the data as soon as the server accepts the connection. This may be the cause of your troubles. You're not supposed to "wait" for the response, the response will get to you asynchronously thanks to nodejs :) So you just need to send the commands as soon as you get the "right" response from the server (this is actually useful, because you can see if there were any errors, etc).
I've tried this code (based on yours) against a device I've got at hand that has a telnet server. It will do a login and then a logout. See how the events are dispatched according to the sever's response:
var net = require('net');
var conn = net.createConnection(23, '1.1.1.1');
var commands = [ "logout\n" ];
var i = 0;
conn.setEncoding('ascii');
conn.on('connect', function() {
conn.on('login', function () {
conn.write('myUsername\n');
});
conn.on('password', function () {
conn.write('myPassword\n');
});
conn.on('prompt', function () {
conn.write(commands[i]);
i++;
});
conn.on('data', function(data) {
console.log("got: " + data + "\n");
if (data.indexOf("login") != -1) {
conn.emit('login');
}
if (data.indexOf("password") != -1) {
conn.emit('password');
}
if (data.indexOf(">#") != -1) {
conn.emit('prompt');
}
});
});
See how the commands are in an array, where you can iteratively send them (the prompt event will trigger the next command). So the right response from the server is the next prompt. When the server sends (in this case) the string ># another command is sent.
Hope it helps :)
The order of writes is guaranteed. However:
You must subscribe to data event. conn.on('data', function(data)
{}) will do.
You must check return values of writes - if a write
fails, you must wait for 'drain' event. So you should check if any
write really fails and if it does then fix the problem. If it
doesn't - then you can leave current dirty solution as is.
You
must check if your server supports request piplining (sending
multiple requests without waiting for responses). If it doesn't -
you must not send next request before receiving a data event after
the previous one.
You must ensure that the commands you send are real telnet commands - telnet expects a \0 byte after \r\n (see the RFC), so certain servers may freak if \0 is not present.
So:
var net = require('net');
var conn = net.createConnection(8888, 'localhost');
conn.on('connect', function() {
console.log(conn.write(command_1) &&
conn.write(command_2) &&
conn.write(command_3) &&
//...
conn.write(command_n))
})
conn.on('data', function () {})
If it writes false - then you must wait for 'drain'. If it writes true - you must implement waiting. I discourage the event-based solution and suggest to look at using async or Step NPM modules instead.
1- Which one is better to use for streaming video ? TCP or UDP socket and why?
2- While streaming live, audio and video are coming from the server separately, so how can i make sure that the video i display and the audio I play on the device are in sync?
I wrote a voice chat application a while ago and TCP was out of the question, UDP multicasting is really the only way to go if you're looking for near-realtime data flow. There's two main issues with streaming stuff over UDP though:
The dropped packets. In the case of audio, it's a pretty easy fix. Usually the dropped packets won't make an audible difference (the packets are decompressed individually). However, when dealing with video, especially if the video is compressed (it usually is), figuring out a proper transfer protocol that ensures network robustness is a daunting task to say the least, especially if you're doing this from scratch. Video frames are split up in various packets. Figuring out what to do when these packets are missing is tough.
Synchronization between audio and video. This is a very tough problem and I suggest reading up on protocols such as RTSP (Real-Time Streaming Protocol). This is not an easy task, but here's some introductory info: http://www.cs.columbia.edu/~hgs/rtsp/ - sometimes it's done by sending separate sync packets (some protocols send these over TCP) that tell the player how the sound should match up with the video.
I would do UDP. However it depends on what you want. UDP will drop packets rather than wait (TCP). The trade off is whether you want a stable, but sometimes slow and costly, or one that is efficient, but sometimes may not get delivered. The choice is yours when it comes to how you want to implement it and how you are using it.
Today even youtube streams over HTTP ... here is a nodejs app which streams a file to the browser client ... use as a starting point to live stream video with audio nicely in sync
// usage
// do following on server side (your laptop running nodejs)
// node this_file.js
//
// then once above is running point your browser at
// http://localhost:8888
//
// of course your browser could be on your mobile or own custom app
var http = require('http'),
fs = require('fs'),
util = require('util');
var path = "/path/to/audio/or/video/file/local/to/server/cool.mp4"; // put any audio or video file here
var port = 8888;
var host = "localhost";
http.createServer(function (req, res) {
var stat = fs.statSync(path);
var total = stat.size;
if (req.headers.range) { // meaning client (browser) has moved the forward/back slider
// which has sent this request back to this server logic ... cool
var range = req.headers.range;
var parts = range.replace(/bytes=/, "").split("-");
var partialstart = parts[0];
var partialend = parts[1];
var start = parseInt(partialstart, 10);
var end = partialend ? parseInt(partialend, 10) : total-1;
var chunksize = (end-start)+1;
console.log('RANGE: ' + start + ' - ' + end + ' = ' + chunksize);
var file = fs.createReadStream(path, {start: start, end: end});
res.writeHead(206, { 'Content-Range': 'bytes ' + start + '-' + end + '/' + total, 'Accept-Ranges': 'bytes', 'Content-Length': chunksize, 'Content-Type': 'video/mp4' });
file.pipe(res);
} else {
console.log('ALL: ' + total);
res.writeHead(200, { 'Content-Length': total, 'Content-Type': 'video/mp4' });
fs.createReadStream(path).pipe(res);
}
}).listen(port, host);
console.log("Server running at http://" + host + ":" + port + "/");
I want to realize a simple client-server connection using Nodejs.
But I've encountered with the following problem.
Consider the code
server.js:
var net = require('net'),
sys = require('sys');
net.createServer(onConnection).listen(8124);
function onConnection(socket) {
socket.setNoDelay(true);
socket.addListener("connect", function () {
sys.puts('client connected: ' + this.remoteAddress);
});
socket.addListener("data", function (data) {
sys.puts("message: \n" + data + "\n - end of msg.");
});
socket.addListener("end", function () {
sys.puts('end of connection');
this.end();
});
}
sys.puts('Server running at 127.0.0.1:8124');
client.js:
var net = require('net'),
sys = require('sys');
var stream = net.createConnection(8124);
stream.addListener("connect", function(){
sys.puts('connected');
stream.write('a');
stream.flush();
stream.write('b');
stream.flush();
});
stream.addListener("data", function(data){
sys.puts("Message: \n" + data + "\n - end of msg.");
});
When I run client.js I sometimes get only one message 'ab' instead of two messages 'a' and 'b'.
Is there some 'right method' to deal with that?
TCP is a stream protocol. Single write on one end of the pipe can result in multiple "reads" on the other end, and the other way around. You have to either explicitly tell the other side how many bytes you are sending by including the length in the message; or provide easily recognizable message delimiters. In any case you need to read in a loop.
use socket.write return value and callback as documented here https://nodejs.org/api/net.html#net_socket_write_data_encoding_callback to know when the data is completly flushed to the kernel . Wait for that to happen and after that call the second write. that way you make sure of the ordering. Regarding the problem of logic separartion of "a" and "b" you might want to design/implement that "protocol yourself, it's not a responsiblity of the (low-level) socket API.
FYI, if you are able to structure to be a line-based text protocol, you can use readline. See:
https://nodejs.org/api/readline.html#example-tiny-cli
https://nodejs.org/api/readline.html#example-read-file-stream-line-by-line