I have a very simple proto:
syntax = "proto2";
message TestMessage {
optional int32 val = 1;
optional string msg = 2; // I set max size to 40 in options, so TestMessage_size is defined.
}
...and I have the following code in my main loop for an arduino program:
TestMessage test_msg = TestMessage_init_zero;
test_msg.val = 123;
// Print message length.
size_t msg_length;
bool get_msg_length = pb_get_encoded_size(&msg_length, TestMessage_fields, &test_msg);
Serial.println(msg_length);
// Encode and print message.
uint8_t testbuffer[TestMessage_size];
pb_ostream_t teststream = pb_ostream_from_buffer(testbuffer, sizeof(testbuffer));
bool teststatus = pb_encode(&teststream, TestMessage_fields, &test_msg);
if (!teststatus) {
Serial.println("Failed to encode test message.");
return;
}
Serial.print("Message: ");
Serial.println(teststream.bytes_written);
for(size_t i = 0; i < teststream.bytes_written; i++){
Serial.print(testbuffer[i], OCT);
}
Serial.println("testbuffer flushed");
For some reason I can print test_msg.val and it will show 123 but when I try to encode it (following examples like this one) it always is empty / has size 0.
Is this a configuration issue with nanopb? I wonder if the encode method requires something that I am not using?
For optional fields, you also have to set the has_field:
TestMessage test_msg = TestMessage_init_zero;
test_msg.has_val = true;
test_msg.val = 123;
That's because otherwise there is no way to know if the optional field has been set or not. C++ handles this via setter methods, but C doesn't have those.
So I am working on a project where we are making a small compiler program but before I can move on to the other parts I am having troubles with getting the lexical analyzer to output anything after '\BEGIN' afterwards I debugged it and it seems the value is stuck in a loop where the condition is saying the next character is always a newline. Is it because I haven't added the pattern matching yet to the defined tokens?
Here is the code
import java.util
//import com.sun.javafx.fxml.expression.Expression.Parser.Token
/*Lexical analyzer will be responsible for the following:
- finds the lexemes
- Checks each given character determining the tokens
* */
class MyLexicalAnalyzer extends LexicalAnalyzer {
//Array full of the keywords
//val SpecialCharacters = List(']', '#', '*', '+', '\\', '[', '(',')', "![", '=')
val TEXT = "[a-z] | _ | 0-9 | [A-Z]:"
private var sourceLine: String = null
private val lexeme: Array[Char] = new Array[Char](999)
private var nextChar: Char = 0
private var lexLength: Int = 0
private var position: Int = 0
private val lexems: util.List[String] = new util.ArrayList[String]
def start(line: String): Unit = {
initializeLexems()
sourceLine = line
position = 0
getChar()
getNextToken()
}
// A helper method to determine if the current character is a space.
private def isSpace(c: Char) = c == ' '
//Defined and intialized tokens
def initializeLexems(): Any = {
lexems.add("\\BEGIN")
lexems.add("\\END")
lexems.add("\\PARAB")
lexems.add("\\DEF[")
lexems.add("\\USE[")
lexems.add("\\PARAE")
lexems.add("\\TITLE[")
lexems.add("]")
lexems.add("[")
lexems.add("\\")
lexems.add("(")
lexems.add(")")
lexems.add("![")
lexems.add("=")
lexems.add("+")
lexems.add("#")
}
//val pattern = new regex("''").r
def getNextToken() ={
lexLength = 0
// Ignore spaces and add the first character to the token
getNonBlank()
addChar()
getChar()
// Continue gathering characters for the token
while ( {
(nextChar != '\n') && (nextChar != ' ')
}) {
addChar()
getChar()
}
// Convert the gathered character array token into a String
val newToken: String = new String(lexeme)
if (lookup(newToken.substring(0, lexLength)))
MyCompiler.setCurrentToken(newToken.substring(0,lexLength))
}
// A helper method to get the next non-blank character.
private def getNonBlank(): Unit = {
while ( {
isSpace(nextChar)
}) getChar()
}
/*
Method of function that adds the current character to the token
after checking to make sure that length of the token isn't too
long, a lexical error in this case.
*/
def addChar(){
if (lexLength <= 998) {
lexeme({
lexLength += 1; lexLength - 1
}) = nextChar
lexeme(lexLength) = 0
}
else
System.out.println("LEXICAL ERROR - The found lexeme is too long!")
if (!isSpace(nextChar))
while ( {
!isSpace(nextChar)
})
getChar()
lexLength = 0
getNonBlank()
addChar()
}
//Reading from the file its obtaining the tokens
def getChar() {
if (position < sourceLine.length)
nextChar = sourceLine.charAt ( {
position += 1;
position - 1
})
else nextChar = '\n'
def lookup(candidateToken: String): Boolean ={
if (!(lexems.contains(candidateToken))) {
System.out.println("LEXICAL ERROR - '" + candidateToken + "' is not recognized.")
return false
}
return true
}
}
else nextChar = '\n'<- this is where the condition goes after rendering the first character '\BEGIN' then just keeps outputting in the debug console as listed below.
This is what the debug console it outputting after '\BEGIN' is read through
Can anyone please let me know why that is? This happens after I keep stepping into it many times as well.
Here is the driver class that uses the lexical analyzer
import scala.io.Source
object MyCompiler {
//check the arguments
//check file extensions
//initialization
//get first token
//call start state
var currentToken : String = ""
def main(args: Array[String]): Unit = {
val filename = args(0)
//check if an input file provided
if(args.length == 0) {
//usage error
println("USAGE ERROR: Must provide an input file. ")
System.exit(0)
}
if(!checkFileExtension(args(0))) {
println("USAGE ERROR: Extension name is invalid make sure its .gtx ")
System.exit(0)
}
val Scanner = new MyLexicalAnalyzer
val Parser = new MySyntaxAnalyzer
//getCurrentToken(Scanner.getNextToken())
//Parser.gittex()
for (line <- Source.fromFile(filename).getLines()){
Scanner.start(line)
println()
}
//.......
//If it gets here, it is compiled
//post processing
}
//checks the file extension if valid and ends with .gtx
def checkFileExtension(filename : String) : Boolean = filename.endsWith(".gtx")
def getCurrentToken() : String = this.currentToken
def setCurrentToken(t : String ) : Unit = this.currentToken = t
}
The code is operating as it is supposed to. The first line contains only the string \BEGIN so the lexical analyser is treating the end of the first line as an '\n' as shown in this method:
def getChar() {
if (position < sourceLine.length)
nextChar = sourceLine.charAt ( {
position += 1;
position - 1
})
else nextChar = '\n'
However, the comment directly above that method does not describe what the method actually does. This could be a hint as to where your confusion lies. If the comment says it should read from the file, but it is not reading from the file, maybe that's what you've forgotten to implement.
I am trying to send some data from Matlab to arduino. It is a matrix of [1 by 2]. My plan is to convert this two numbers in to a string and send to the arduino. However in the serial monitor, I can not read any value coming from matlab.
This is my matlab code,
val_a = matt(n,:);
A = [val_a];
asd = A(1:1);
asb = A(:,2);
strA = num2str(asd);
strB = num2str(asb);
comma = ',';
endVal = '#';
theString = strcat(strA,comma,endVal);
obj1 = instrfind('Type', 'serial', 'Port', 'COM19', 'Tag', '');
if isempty(obj1)
obj1 = serial('COM19');
else
fclose(obj1);
obj1 = obj1(1);
end
fopen(obj1);
fprintf(obj1,theString)
fclose(obj1);
delete(obj1);
A = [];
And this is the serial event of arduino
bool gotalfa = false;
bool event = false;
void serialEvent() {
while (Serial.available())
{
char inChar = (char)Serial.read();
event = true;
if (inChar == , && !gotalfa)
{
alfa = inputString;
inputString = "";
gotalfa = true;
event = false;
}
if (inChar == '#' && gotalfa)
{
theta = inputString;
gotalfa = false;
inputString = "";
Serial.print("alfa ");
Serial.print(alfa);
Serial.print("theta ");
Serial.println(theta);
//some program....
event = false;
}
if(event)
{
inputString += inChar;
}}}
Do I have to change anything in my matlab/arduino code. Any helpful tip is greatly appreciated.
Thank you in advance
Just a tip, hopefully it points you to the answer.
I would use a terminal program like putty or teraterm to isolate the root cause of your error.
i.e. Run putty/teraterm to send certain string to your serial port of your arduino and see if it does returns the expected strings.
http://www.putty.org/
https://ttssh2.osdn.jp/index.html.en
Also, consider using the ReadStringUntil in Arduino.
https://www.arduino.cc/en/Serial/ReadStringUntil
Hope it helps.
I've searched and found a few posts relating to postgres csv imports, but nothing that solves my current problem.
I use the postgres copy command all the time to bring data from hetergeneous data sources into our system. Currently struggling with a 100-million row .csv file, comma-quote delimited. Issue is with rows like so:
009098,0981098094,"something","something else",""this one, well, is a problem"", "another thing"
Fields enclosed in double-quotes with embedded commas. The fields are not correctly parsed and I get the error:
"ERROR: extra data after last expected column"
Usually when this arises I deal with the offending rows ad hoc, but this file is so huge I'm hoping for some more general way to defend against it. Asking for a revised data format is not a possibility.
copy mytable from '/path/to/file.csv' csv header quote '"'
That's malformed CSV. You double a double quote to embed a double quote inside a quote field; for example:
"where","is ""pancakes""","house?"
has three values:
where
is "pancakes"
house?
The row you're having trouble with has stray doubled double quotes:
009098,0981098094,"something","something else",""this one, well, is a problem"", "another thing"
^^ ^^
I don't think there is anything that COPY can do about this as the correct version is ambiguous: should it be "this one, well, is a problem" or should it be """this one, well, is a problem"""?
I think you'll have to fix it by hand. A quick sed one-liner should be able to do the job if you can uniquely identify the broken row.
For reference purposes, the closest thing I've seen to a CSV standard is RFC 4180 and section two has this to say:
5. Each field may or may not be enclosed in double quotes (however
some programs, such as Microsoft Excel, do not use double quotes
at all). If fields are not enclosed with double quotes, then
double quotes may not appear inside the fields. For example:
"aaa","bbb","ccc" CRLF
zzz,yyy,xxx
[...]
7. If double-quotes are used to enclose fields, then a double-quote
appearing inside a field must be escaped by preceding it with
another double quote. For example:
"aaa","b""bb","ccc"
Here is code based on the CSV code from The Practice of Programming by Kernighan and Plauger that has been adapted to deal with your weird malformed CSV data. (It wasn't all that hard to do; I already had the main code working and packaged, so I just had to add the CSV output functions and to modify the advquoted() function to handle the weird format in this question.
csv2.h
/*
#(#)File: $RCSfile: csv2.h,v $
#(#)Version: $Revision: 2.1 $
#(#)Last changed: $Date: 2012/11/01 22:23:07 $
#(#)Purpose: Scanner for Comma Separated Variable (CSV) Data
#(#)Author: J Leffler
#(#)Origin: Kernighan & Pike, 'The Practice of Programming'
*/
/*TABSTOP=4*/
#ifndef CSV2_H
#define CSV2_H
#ifdef __cplusplus
extern "C" {
#endif
#ifdef MAIN_PROGRAM
#ifndef lint
/* Prevent over-aggressive optimizers from eliminating ID string */
const char jlss_id_csv2_h[] = "#(#)$Id: csv2.h,v 2.1 2012/11/01 22:23:07 jleffler Exp $";
#endif /* lint */
#endif /* MAIN_PROGRAM */
#include <stdio.h>
extern char *csvgetline(FILE *ifp); /* Read next input line */
extern char *csvgetfield(size_t n); /* Return field n */
extern size_t csvnfield(void); /* Return number of fields */
extern void csvreset(void); /* Release space used by CSV */
extern int csvputfield(FILE *ofp, const char *field);
extern int csvputline(FILE *ofp, char **fields, int nfields);
extern void csvseteol(const char *eol);
#ifdef __cplusplus
}
#endif
#endif /* CSV2_H */
csv2.c
/*
#(#)File: $RCSfile: csv2.c,v $
#(#)Version: $Revision: 2.1 $
#(#)Last changed: $Date: 2012/11/01 22:23:07 $
#(#)Purpose: Scanner for Comma Separated Variable (CSV) Data
#(#)Modification: Deal with specific malformed CSV
#(#)Author: J Leffler
#(#)Origin: Kernighan & Pike, 'The Practice of Programming'
*/
/*TABSTOP=4*/
#ifndef lint
/* Prevent over-aggressive optimizers from eliminating ID string */
const char jlss_id_csv2_c[] = "#(#)$Id: csv2.c,v 2.1 2012/11/01 22:23:07 jleffler Exp $";
#endif /* lint */
/*
** See RFC 4180 (http://www.ietf.org/rfc/rfc4180.txt).
**
** Specific malformed CSV - see SO 13183644 (http://stackoverflow.com/questions/13183644).
** Data contains malformed CSV fields like: OK,""this is a problem"",OK
** Two (but not three) field quotes at the start extract as "this is a problem" (with the quotes).
*/
#include "csv2.h"
#include <stdlib.h>
#include <string.h>
enum { NOMEM = -2 };
static char *line = 0; /* Input line */
static char *sline = 0; /* Split line */
static size_t maxline = 0; /* Size of line[] and sline[] */
static char **field = 0; /* Field pointers */
static size_t maxfield = 0; /* Size of field[] */
static size_t nfield = 0; /* Number of fields */
static char fieldsep[]= ","; /* Field separator characters */
static char fieldquote = '"'; /* Quote character */
static char eolstr[8] = "\n";
void csvreset(void)
{
free(line);
free(sline);
free(field);
line = 0;
sline = 0;
field = 0;
maxline = maxfield = nfield = 0;
}
static int endofline(FILE *ifp, int c)
{
int eol = (c == '\r' || c == '\n');
if (c == '\r')
{
c = getc(ifp);
if (c != '\n' && c != EOF)
ungetc(c, ifp);
}
return(eol);
}
/* Modified to deal with specific malformed CSV */
static char *advquoted(char *p)
{
size_t i;
size_t j;
if (p[0] == fieldquote && (p[1] != *fieldsep && p[1] != fieldquote))
{
/* Malformed CSV: ""some stuff"" --> "some stuff" */
/* Find "\"\"," or "\"\"\0" to mark end of field */
/* If we don't find it, drop through to 'regular' case */
char *eof = strstr(&p[2], "\"\"");
if (eof != 0 && (eof[2] == *fieldsep || eof[2] == '\0'))
{
p[eof + 1 - p] = '\0';
return(eof + 2);
}
}
for (i = j = 0; p[j] != '\0'; i++, j++)
{
if (p[j] == fieldquote && p[++j] != fieldquote)
{
size_t k = strcspn(p+j, fieldsep);
memmove(p+i, p+j, k); // 1 -> i fixing transcription error
i += k;
j += k;
break;
}
p[i] = p[j];
}
p[i] = '\0';
return(p + j);
}
static int split(void)
{
char *p;
char **newf;
char *sepp;
int sepc;
nfield = 0;
if (line[0] == '\0')
return(0);
strcpy(sline, line);
p = sline;
do
{
if (nfield >= maxfield)
{
maxfield *= 2;
newf = (char **)realloc(field, maxfield * sizeof(field[0]));
if (newf == 0)
return NOMEM;
field = newf;
}
if (*p == fieldquote)
sepp = advquoted(++p);
else
sepp = p + strcspn(p, fieldsep);
sepc = sepp[0];
sepp[0] = '\0';
field[nfield++] = p;
p = sepp + 1;
} while (sepc == ',');
return(nfield);
}
char *csvgetline(FILE *ifp)
{
size_t i;
int c;
if (line == NULL)
{
/* Allocate on first call */
maxline = maxfield = 1;
line = (char *)malloc(maxline); /*=C++=*/
sline = (char *)malloc(maxline); /*=C++-*/
field = (char **)malloc(maxfield*sizeof(field[0])); /*=C++=*/
if (line == NULL || sline == NULL || field == NULL)
{
csvreset();
return(NULL); /* out of memory */
}
}
for (i = 0; (c = getc(ifp)) != EOF && !endofline(ifp, c); i++)
{
if (i >= maxline - 1)
{
char *newl;
char *news;
maxline *= 2;
newl = (char *)realloc(line, maxline); /*=C++=*/
news = (char *)realloc(sline, maxline); /*=C++-*/
if (newl == NULL || news == NULL)
{
csvreset();
return(NULL); /* out of memory */
}
line = newl;
sline = news;
}
line[i] = c;
}
line[i] = '\0';
if (split() == NOMEM)
{
csvreset();
return(NULL);
}
return((c == EOF && i == 0) ? NULL : line);
}
char *csvgetfield(size_t n)
{
if (n >= nfield)
return(0);
return(field[n]);
}
size_t csvnfield(void)
{
return(nfield);
}
int csvputfield(FILE *ofp, const char *ofield)
{
const char escapes[] = "\",\r\n";
if (strpbrk(ofield, escapes) != 0)
{
size_t len = strlen(ofield) + 2;
const char *pos = ofield;
while ((pos = strchr(pos, '"')) != 0)
{
len++;
pos++;
}
char *space = malloc(len+1);
if (space == 0)
return EOF;
char *cpy = space;
pos = ofield;
*cpy++ = '"';
char c;
while ((c = *pos++) != '\0')
{
if (c == '"')
*cpy++ = c;
*cpy++ = c;
}
*cpy++ = '"';
*cpy = '\0';
int rc = fputs(space, ofp);
free(space);
return rc;
}
else
return fputs(ofield, ofp);
}
int csvputline(FILE *ofp, char **fields, int nfields)
{
for (int i = 0; i < nfields; i++)
{
if (i > 0)
putc(',', ofp);
if (csvputfield(ofp, fields[i]) == EOF)
return EOF;
}
return(fputs(eolstr, ofp));
}
void csvseteol(const char *eol)
{
size_t nbytes = strlen(eol);
if (nbytes >= sizeof(eolstr))
nbytes = sizeof(eolstr) - 1;
memmove(eolstr, eol, nbytes);
eolstr[nbytes] = '\0';
}
#ifdef TEST
int main(void)
{
char *in_line;
while ((in_line = csvgetline(stdin)) != 0)
{
size_t n = csvnfield();
char *fields[n]; /* C99 VLA */
printf("line = '%s'\n", in_line);
for (size_t i = 0; i < n; i++)
{
printf("field[%zu] = '%s'\n", i, csvgetfield(i));
printf("field[%zu] = [", i);
csvputfield(stdout, csvgetfield(i));
fputs("]\n", stdout);
fields[i] = csvgetfield(i);
}
printf("fields[0..%zu] = ", n-1);
csvputline(stdout, fields, n);
}
return(0);
}
#endif /* TEST */
Compile the code with -DTEST to create a program with the example main() function. You need a C99 compiler; the code in main() uses a VLA (variable length array). You could avoid that with dynamic memory allocation or with pessimistic (overkill) memory allocation (an array of a few thousand pointers isn't going to kill most systems these days, but few CSV files will have a few thousand fields per line).
Example Data
Based closely on the data in the question.
009098,0981098094,"something","something else",""this one, well, is a problem"", "another thing"
123458,1234561007,"anything","nothing else",""this one, well, is a problem"","dohicky
503458,1234598094,"nothing","everything else","""this one, well, it isn't a problem""","abelone"
610078,1236100794,"everything","anything else","this ""isn't a problem"", he said.","Orcas Rule"
Example Output
line = '009098,0981098094,"something","something else",""this one, well, is a problem"", "another thing"'
field[0] = '009098'
field[0] = [009098]
field[1] = '0981098094'
field[1] = [0981098094]
field[2] = 'something'
field[2] = [something]
field[3] = 'something else'
field[3] = [something else]
field[4] = '"this one, well, is a problem"'
field[4] = ["""this one, well, is a problem"""]
field[5] = ' "another thing"'
field[5] = [" ""another thing"""]
fields[0..5] = 009098,0981098094,something,something else,"""this one, well, is a problem"""," ""another thing"""
line = '123458,1234561007,"anything","nothing else",""this one, well, is a problem"","dohicky'
field[0] = '123458'
field[0] = [123458]
field[1] = '1234561007'
field[1] = [1234561007]
field[2] = 'anything'
field[2] = [anything]
field[3] = 'nothing else'
field[3] = [nothing else]
field[4] = '"this one, well, is a problem"'
field[4] = ["""this one, well, is a problem"""]
field[5] = 'dohicky'
field[5] = [dohicky]
fields[0..5] = 123458,1234561007,anything,nothing else,"""this one, well, is a problem""",dohicky
line = '503458,1234598094,"nothing","everything else","""this one, well, it isn't a problem""","abelone"'
field[0] = '503458'
field[0] = [503458]
field[1] = '1234598094'
field[1] = [1234598094]
field[2] = 'nothing'
field[2] = [nothing]
field[3] = 'everything else'
field[3] = [everything else]
field[4] = '"this one, well, it isn't a problem"'
field[4] = ["""this one, well, it isn't a problem"""]
field[5] = 'abelone'
field[5] = [abelone]
fields[0..5] = 503458,1234598094,nothing,everything else,"""this one, well, it isn't a problem""",abelone
line = '610078,1236100794,"everything","anything else","this ""isn't a problem"", he said.","Orcas Rule"'
field[0] = '610078'
field[0] = [610078]
field[1] = '1236100794'
field[1] = [1236100794]
field[2] = 'everything'
field[2] = [everything]
field[3] = 'anything else'
field[3] = [anything else]
field[4] = 'this "isn't a problem", he said.'
field[4] = ["this ""isn't a problem"", he said."]
field[5] = 'Orcas Rule'
field[5] = [Orcas Rule]
fields[0..5] = 610078,1236100794,everything,anything else,"this ""isn't a problem"", he said.",Orcas Rule
The fields are printed twice, once to test the field extraction, once to test the field printing. You'd simplify the output by removing the printing except for csvputline() to convert your file from malformed CSV to properly formed CSV.
In a given sentence i want to split into 10 character string. The last word should not be incomplete in the string. Splitting should be done based on space or , or .
For example:
this is ram.he works at mcity.
now the substring of 10 chars is,
this is ra.
but the output should be,
this is.
Last word should not be incomplete
You can use a regular expression that checks that the character after the match is not a word character:
string input = "this is ram.he";
Match match = Regex.Match(input, #"^.{0,10}(?!\w)");
string result;
if (match.Success)
{
result = match.Value;
}
else
{
result = string.Empty;
}
Result:
this is
An alternative approach is to build the string up token by token until adding another token would exceed the character limit:
StringBuilder sb = new StringBuilder();
foreach (Match match in Regex.Matches(input, #"\w+|\W+"))
{
if (sb.Length + match.Value.Length > 10) { break; }
sb.Append(match.Value);
}
string result = sb.ToString();
Not sure if this is the sort of thing you were looking for. Note that this could be done a lot cleaner, but should get you started ... (may want to use StringBuilder instead of String).
char[] delimiterChars = { ',', '.',' ' };
string s = "this is ram.he works at mcity.";
string step1 = s.Substring(0, 10); // Get first 10 chars
string[] step2a = step1.Split(delimiterChars); // Get words
string[] step2b = s.Split(delimiterChars); // Get words
string sFinal = "";
for (int i = 0; i < step2a.Count()-1; i++) // copy count-1 words
{
if (i == 0)
{
sFinal = step2a[i];
}
else
{
sFinal = sFinal + " " + step2a[i];
}
}
// Check if last word is a complete word.
if (step2a[step2a.Count() - 1] == step2b[step2a.Count() - 1])
{
sFinal = sFinal + " " + step2b[step2a.Count() - 1] + ".";
}
else
{
sFinal = sFinal + ".";
}