Receiving data in node.js - sockets

I am having a java program send data to me over a specific socket to my node.js application. I want to be able to obtain all of the data, which is information from a SQlite database, and send it off to something else.
I've found something like the following can work but it seems to be unreliable as data is missing and sometimes it doesn't even show up.
stream.addListener('data', function(data){
buffer.write(data.toString());
});
on a side note, I need the socket to stay open so I can't call the "end" event.
I really don't have any attachment to stream.addListener so i can use something else if it works how i want. Basically what i'm asking is, What is the most effective way to obtain data from a socket using node.js?
P.S. thank you for your time

The data event is not guaranteed to have all the data sent to it in one go. You'll need to build up a buffer over multiple events and watch for delimiters of some kind (newlines, null characters, whatever you feel). Here's an example from a project where I'm parsing data from IRC (converted from CoffeeScript); parseData is the event handler for the data event (e.g. socket.on('data', this.parseData);):
IrcConnection.prototype.parseData = function(data) {
var line, lines, i;
data = data.replace("\r\n", "\n");
this.buffer += data;
lines = this.buffer.split("\n");
this.buffer = "";
/* Put the last line back in the buffer if it was incomplete */
if (lines[lines.length - 1] !== '') {
this.buffer = lines[lines.length - 1];
}
/* Remove the final \n or incomplete line from the array */
lines = lines.splice(0, lines.length - 1);
for (i = 0; i < lines.length; i++) {
line = lines[i];
this.emit('raw', line);
}
};

Related

osip_message_to_str fails on a message I created using the message/parser functions

I'm trying to receive an invite message and then reply with 100 trying and/or 180 ringing to the same client. I'd like to use only the parser from libosip2.
So when I try to osip_message_to_str so I have a buffer to send back, it always fails with -2.
I tried cloning all the fields I thought it would make sense. But still having the same issue.
If I try to osip_message_to_str on the message I received, it works well.
parser_init();
osip_message_t *request = received_buffer(buffer); // the message is received properly from the buffer
osip_message_t *response;
int i = osip_message_init(&response);
if (i != 0) {
fprintf(stderr, "cannot allocate\n");
return -1;
}
osip_message_set_version(response, strdup("SIP/2.0"));
osip_message_set_status_code(response, 100);
osip_from_clone(request->from, &response->from);
osip_to_clone(request->to, &response->to); /* include the tag! */
osip_call_id_clone(request->call_id, &response->call_id);
osip_contact_t *contact = nullptr;
osip_message_get_contact(response, 0, &contact);
osip_uri_clone(osip_contact_get_url(contact), &response->req_uri);
osip_cseq_clone(request->cseq, &(response->cseq));
char *dest = NULL;
size_t length = 0;
i = osip_message_to_str(response, &dest, &length);
if (i != 0) {
fprintf(stderr, "resp cannot get printable message %d\n", i);
return -1;
}
fprintf(stdout, "message:\n%s\n", dest);
I expect to be able to print a response message.
From libosip2, file osip_port.h, the error return code -2 means bad parameter:
#define OSIP_BADPARAMETER -2
The first line of an answer is something like this: "SIP/2.0 100 Trying".
In your code, you are setting correctly both "SIP/2.0" and "100". However, you forgot the reason phrase. For "100", obviously, the string should be "Trying". Thus, a complete first line of a response should be done with:
osip_message_set_version(response, osip_strdup("SIP/2.0"));
osip_message_set_status_code(response, 100);
//ADD THIS
osip_message_set_reason_phrase (answer, osip_strdup("Trying");
The above will fix the first error, but there looks to be more. You are using "osip_message_get_contact" to retrieve a contact from the response. But there is none. In order to set a contact, you need to search for your IP address, port number, and parameters you want to add. Something like this is advised:
osip_message_set_contact (response, "<sip:192.168.1.10:5678;ob>");
The above API will parse the string as a Contact header and add it to the response.
To make it clear (as you have used it), "response->req_uri" is empty for a response. It means "Request-URI" which is only for request.
If you wish a complete response, you will also need to copy all the "Via" headers:
{
osip_list_iterator_t it;
osip_via_t *via = (osip_via_t *) osip_list_get_first (&request->vias, &it);
while (via != NULL) {
osip_via_t *via2;
i = osip_via_clone (via, &via2);
if (i != 0) {
osip_message_free (response);
return i;
}
osip_list_add (&response->vias, via2, -1);
via = (osip_via_t *) osip_list_get_next (&it);
}
}
NOTE: use osip_strdup instead of strdup for any osip allocation to make your code more portable.
osip_message_to_str should work then!
For more precise code, feel free to have a look at my exosip2 code here. It will certainly help you for the next question you will have!

How to differentiate as abbreviation for voice response in dialog flow

I have integrated my dialogflow agent with google assistant. There is a welcome intent that will ask you to choose any of the option
Choose any of the sports
1. NBA
2. NHL
3. FIH
It reads the response with ever individual words(as an abbreviation). But when I produce the same in response from webhook, it is not reading the response with individual words(or not considering the response as abbreviation) and reads together. How can I achieve this? Am I missing something in the response?
You likely want to make sure you're sending back SSML in your response, rather than sending back text and letting it convert it to speech, and specifically marking the abbreviations using the <say-as> tag and telling it to interpret the contents as characters.
So you might send it back as something like:
<speak>
Are you interested in learning more about
the <say-as interpret-as="characters">NBA</say-as>,
the <say-as interpret-as="characters">NHL</say-as>
or the <say-as interpret-as="characters">FIH</say-as>?
</speak>
The little pronunciation differences with and without SSML are serious problems. I stick in a speak /speak for everything. Also a unique number I like and a test hook to have speech 'count' or not so there is a way to test things. Also a hook so an intent is triggered for 'repeat that please' :
Point is to use sayUsual for everything ordinary.
// Mostly SSML start char kit as globals
const startSp = "<speak>", endSp = "</speak>";
// Handle "Can you repeat that ?" well
var vfSpokenByMe = "";
// VF near globals what was said, etc
var repeatPossible = {}; repeatPossible.vf = ""; repeatPossible.n = 0;
// An answer from this app to the human in text
function absorbMachineVf( intentNumber, aKind, aStatement )
{
// Numbers reserved for 'repeats'
if( intentNumber > 9000 ) { return; }
// Machine to say this, a number for intents too
repeatPossible.vf = aStatement; repeatPossible.n = intentNumber;
}
// Usual way to say a thing
function sayUsual( n, speechAgent, somethingToSay )
{
// Work with an answer of any sort
absorbMachineVf( n, 'usual', somethingToSay );
// Sometimes we are just pretending, so
if( !testingNow )
{ speechAgent.add( startSp + somethingToSay + endSp ); }
// Make what we said as an answer available 'for sure' to rest of code
vfSpokenByMe = somethingToSay; // Even in simulation
}

Protractor - Create a txt file as report with the "Expect..." result

I'm trying to create a report for my scenario, I want to execute some validations and add the retults in a string, then, write this string in a TXT file (for each validation I would like to add the result and execute again till the last item), something like this:
it ("Perform the loop to search for different strings", function()
{
browser.waitForAngularEnabled(false);
browser.get("http://WebSite.es");
//strings[] contains 57 strings inside the json file
for (var i = 0; i == jsonfile.strings.length ; ++i)
{
var valuetoInput = json.Strings[i];
var writeInFile;
browser.wait;
httpGet("http://website.es/search/offers/list/"+valuetoInput+"?page=1&pages=3&limit=20").then(function(result) {
writeInFile = writeInFile + "Validation for String: "+ json.Strings[i] + " Results is: " + expect(result.statusCode).toBe(200) + "\n";
});
if (i == jsonfile.strings.length)
{
console.log("Executions finished");
var fs = require('fs');
var outputFilename = "Output.txt";
fs.writeFile(outputFilename, "Validation of Get requests with each string:\n " + writeInFile, function(err) {
if(err)
{
console.log(err);
}
else {
console.log("File saved to " + outputFilename);
}
});
}
};
});
But when I check my file I only get the first row writen in the way I want and nothing else, could you please let me know what am I doing wrong?
*The validation works properly in the screen for each of string in my file used as data base
**I'm a newbie with protractor
Thank you a lot!!
writeFile documentation
Asynchronously writes data to a file, replacing the file if it already
exists
You are overwriting the file every time, which is why it only has 1 line.
The easiest way would probably (my opinion) be appendFile. It writes to a file without overwriting existing data and will also create the file if it doesnt exist in the first place.
You could also re-read that log file, store that data in a variable, and re-write to that file with the old AND new data included in it. You could also create a writeStream etc.
There are quite a few ways to go about it and plenty of other answers
on SO specifically on those functions that can provide more info.
Node.js Write a line into a .txt file
Node.js read and write file lines
Final note, if you are using Jasmine you can also create a custom jasmine reporter. They have methods that contain exactly what you want (status Pass/Fail, actual vs expected values etc) and it's fairly easy to set up with Protractor

Sim900 only echos back the commands- No response

I'm using Atmega32 and sim900 for a project. I keep sending "AT" command and wait for the "OK" response, but all I am getting is AT\r\n. I've checked and rechecked wiring and my baud rate, but still getting no where. Whatever I send to sim900 I only get echoed back of the same transmitted string.
Can anyone help me please? I'd really appreciate it.
I'm posting my code here:
int sim900_init(void)
{
while(1)
{
sim_command("AT");
_delay_ms(2000);
}
return 0;
}
void usart_init(void)
{
/************ENABLE USART***************/
UBRRH=(uint8_t)(MYUBRR>>8);
UBRRL=(uint8_t)(MYUBRR); //set baud rate
UCSRB=(1<<TXEN)|(1<<RXEN); //enable receiver and transmitter
UCSRC=(1<<UCSZ0)|(1<<UCSZ1)|(1<<URSEL); // 8bit data format
UCSRB |= (1 << RXCIE); // Enable the USART Recieve Complete interrupt (USART_RXC)
/***************FLUSH ALL PRIVIOUS ACTIVITIES********************/
flush_usart();
/*********APPOINT POINTERS TO ARRAYS********************/
command=commandArray; // Assigning the pointer to array
response=responseArray; //Assigning the pointer to array
/*****************ENABLE INTRUPT***************************/
sei(); //Enabling intrupts for receving characters
}
void flush_usart(void)
{
response_full=FALSE; //We have not yet received the
}
void transmit_char(unsigned char value)
{
while (!( UCSRA & (1<<UDRE))); // wait while register is free
UDR = value;
}
void sim_command(char *cmd)
{
int j=0;
strcpy(command,cmd);
while(*(cmd+j)!='\0')
{
transmit_char(*(cmd+j));
j++;
}
transmit_char(0x0D); // \r // after all the at commands we should send \r\n so, we send it here after the string
transmit_char(0x0A); // \n
}
unsigned char recieve_char(void)
{
char temp;
while(!(UCSRA) & (1<<RXC)); // wait while data is being received
temp=UDR;
LCDdata(lcdchar,temp);
return temp;
}
void recive_sim900_response(void)
{
static int i=0;
char temp;
temp=recieve_char();
if(temp!='\n' && temp!='\r') // We dont want \r \n that will be send from Sim so we dont store them
*(response+i)=temp;
if(i==8) //when null char is sent means the string is finished- so we have full response
{ //we use them later in WaitForResponse function. we wait until the full response is received
i=0;
response_full=TRUE;
}
else
i++;
}
You were the only one who had exactly the same problem as I.
Somehow the library from gsmlib.org worked but entering AT commands directly using the Arduino serial monitor using the Arduino as a bridge or just an FTDI didn't.
The reason is that apparently the SIM900 expects commands to end with an '\r' character. I found this by trying GTKTerm which worked.
If typing "AT" and pressing enter in GTKTerm what is actually sent is "AT" followed by twice '\r' (0x0d) and one 0x0a
By default GSM module is in echo back ON mode. And you need to change your command.
sim_command("AT");
you need Enter=CR/LF after Command so modify your code like this and give a try
sim_command("AT\r");
And in case you want to turn off echo back of command you send then you should send this command once you have OK response back for AT command.
sim_command("ATE0\r"); //Echo back OFF
sim_command("ATE1\r"); //Echo back ON

NodeJS: What is the proper way to handling TCP socket streams ? Which delimiter should I use?

From what I understood here, "V8 has a generational garbage collector. Moves objects aound randomly. Node can’t get a pointer to raw string data to write to socket." so I shouldn't store data that comes from a TCP stream in a string, specially if that string becomes bigger than Math.pow(2,16) bytes. (hope I'm right till now..)
What is then the best way to handle all the data that's comming from a TCP socket ? So far I've been trying to use _:_:_ as a delimiter because I think it's somehow unique and won't mess around other things.
A sample of the data that would come would be something_:_:_maybe a large text_:_:_ maybe tons of lines_:_:_more and more data
This is what I tried to do:
net = require('net');
var server = net.createServer(function (socket) {
socket.on('connect',function() {
console.log('someone connected');
buf = new Buffer(Math.pow(2,16)); //new buffer with size 2^16
socket.on('data',function(data) {
if (data.toString().search('_:_:_') === -1) { // If there's no separator in the data that just arrived...
buf.write(data.toString()); // ... write it on the buffer. it's part of another message that will come.
} else { // if there is a separator in the data that arrived
parts = data.toString().split('_:_:_'); // the first part is the end of a previous message, the last part is the start of a message to be completed in the future. Parts between separators are independent messages
if (parts.length == 2) {
msg = buf.toString('utf-8',0,4) + parts[0];
console.log('MSG: '+ msg);
buf = (new Buffer(Math.pow(2,16))).write(parts[1]);
} else {
msg = buf.toString() + parts[0];
for (var i = 1; i <= parts.length -1; i++) {
if (i !== parts.length-1) {
msg = parts[i];
console.log('MSG: '+msg);
} else {
buf.write(parts[i]);
}
}
}
}
});
});
});
server.listen(9999);
Whenever I try to console.log('MSG' + msg), it will print out the whole buffer, so it's useless to see if something worked.
How can I handle this data the proper way ? Would the lazy module work, even if this data is not line oriented ? Is there some other module to handle streams that are not line oriented ?
It has indeed been said that there's extra work going on because Node has to take that buffer and then push it into v8/cast it to a string. However, doing a toString() on the buffer isn't any better. There's no good solution to this right now, as far as I know, especially if your end goal is to get a string and fool around with it. Its one of the things Ryan mentioned # nodeconf as an area where work needs to be done.
As for delimiter, you can choose whatever you want. A lot of binary protocols choose to include a fixed header, such that you can put things in a normal structure, which a lot of times includes a length. In this way, you slice apart a known header and get information about the rest of the data without having to iterate over the entire buffer. With a scheme like that, one can use a tool like:
node-buffer - https://github.com/substack/node-binary
node-ctype - https://github.com/rmustacc/node-ctype
As an aside, buffers can be accessed via array syntax, and they can also be sliced apart with .slice().
Lastly, check here: https://github.com/joyent/node/wiki/modules -- find a module that parses a simple tcp protocol and seems to do it well, and read some code.
You should use the new stream2 api. http://nodejs.org/api/stream.html
Here are some very useful examples: https://github.com/substack/stream-handbook
https://github.com/lvgithub/stick