I have been trying to use automatic replace function to place entries starting with italics on their own lines, so for example
lähteä60*F【动】
1. 离开, 出发, 走掉líkāi, chūfā, zǒu diào(poistua). Vieraat lähtevät.客人走了kèrén zǒule.Juna lähtee raiteelta kaksi. 火车两点离站huǒchē liǎng diǎn lí zhàn.
would turn into
1. 离开, 出发, 走掉líkāi, chūfā, zǒu diào(poistua).
Vieraat lähtevät.客人走了kèrén zǒule.
Juna lähtee raiteelta kaksi. 火车两点离站huǒchē liǎng diǎn lí zhàn.
However, for some reason unknown to me, the replace function (cf. screenshot) breaks up the lines like this:
lähteä60*F【动】
1. 离开, 出发, 走掉líkāi, chūfā, zǒu diào(poistua).
Vieraat lähtev
ä
t.客人走了kèrén zǒule.
Juna lähtee raiteelta kaksi. 火车两点离站huǒchē liǎng diǎn lí zhàn.
(so the first line "Vieraat lähtevät.客人走了kèrén zǒule." is broken up.)
As far as I can tell, it should be all italics, so I have no idea why it breaks up and no way to examine what's the problem. Trying to save to different formats doesn't seem to help either. There's thousands of pages of this stuff, so the automatic function is really required.
A small sample file of the stuff can be downloaded from here:
http://shakki.info/test.docx.
Some screenshots of the problem:
Related
I am using deep speech for speech to text. Up to 0.8.1, when I ran transcriptions like:
byte_encoding = subprocess.check_output(
"deepspeech --model deepspeech-0.8.1-models.pbmm --scorer deepspeech-0.8.1-models.scorer --audio audio/2830-3980-0043.wav", shell=True)
transcription = byte_encoding.decode("utf-8").rstrip("\n")
I would get back results that were pretty good. But since 0.8.2, where the scorer argument was removed, my results are just rife with misspellings that make me think I am now getting a character level model where I used to get a word-level model. The errors are in a direction that looks like the model isn't correctly specified somehow.
Now I when I call:
byte_encoding = subprocess.check_output(
['deepspeech', '--model', 'deepspeech-0.8.2-models.pbmm', '--audio', myfile])
transcription = byte_encoding.decode("utf-8").rstrip("\n")
I now see errors like
endless -> "endules"
service -> "servic"
legacy -> "legaci"
earning -> "erting"
before -> "befir"
I'm not 100% that it is related to removing the scorer from the API, but it is one thing I see changing between releases, and the documentation suggested accuracy improvements in particular.
Short: The scorer matches letter output from the audio to actual words. You shouldn't leave it out.
Long: If you leave out the scorer argument, you won't be able to detect real world sentences as it matches the output from the acoustic model to words and word combinations present in the textual language model that is part of the scorer. And bear in mind that each scorer has specific lm_alpha and lm_beta values that make the search even more accurate.
The 0.8.2 version should be able to take the scorer argument. Otherwise update to 0.9.0, which has it as well. Maybe your environment is changed in a way. I would start in a new dir and venv.
Assuming you are using Python, you could add this to your code:
ds.enableExternalScorer(args.scorer)
ds.setScorerAlphaBeta(args.lm_alpha, args.lm_beta)
And check the example script.
I'm trying to test the model (model.bin) i've made with fastText on a test file (test.txt). In this test file, i have 135 labelised data. I'm expecting from fastText to test my model on this number of example, but instead, it only test it over 1 example. Where does come from this problem ?
I've already tried to do such a thing with another model and another testing file and all worked nicely.
this is how I test my model. model_baby.bin is the model, and test.data.txt is my testing file.
./fasttext test model_baby.bin test.data.txt
N 1
P#1 1
R#1 0.0164
Number of examples: 1
And here is an extract from my testing file
__label__4.0 I love the fact you can hide your stuff. Only down is that the straps to hold it at midpoint and bottom could be better designed for your car. It's got plenty of room which is great. __label__5.0 This hid our ipad wonderfully. Especially for those quick stops where we all had jump out and use the restroom. It zipped, folded and held all our stuff for the kids in the back seat. __label__3.0
As i have more than 1 labelised example in my testing file, I expect the output "Number of examples: " to be at least more than 1 but the actual one is "1"
From the official documentation (https://fasttext.cc/docs/en/supervised-tutorial.html): Each line of the text file contains a list of labels, followed by the corresponding document. All the labels start by the __label__ prefix, which is how fastText recognize what is a label or what is a word.
I don't understand very much your extract. I think it should be like this:
__label__4.0 I love the fact you can hide your stuff. Only down is that the straps to hold it at midpoint and bottom could be better designed for your car. It's got plenty of room which is great.
__label__5.0 This hid our ipad wonderfully. Especially for those quick stops where we all had jump out and use the restroom. It zipped, folded and held all our stuff for the kids in the back seat.
__label__3.0 ...
I'm quite new to ELK and Grok-filtering, and I'm struggling with parsing this particular pattern in my grok filter.
I've used the grok debugger to try and solve this, but although I like the tool, I just get confused by the custom patterns.
Eventually, I hope to parse lots of log files sent by filebeat to logstash, then send the parsed logs to elasticsearch and display with kibana or some similar visualization tool.
The lines that I need to parse follow the following pattern:
1310 2017-01-01 16:48:54 [325:51] [326:49] [359:57] Some log info text
The first four digits is a log type identifier, and will be used for grouping. I've called the field "LogLineID".
The date is formatted YYYY-MM-DD HH:MM:SS, and is parsed ok. I called the field "LogDate".
But now the problem begins. Within the square brackets, I have counters, formatted as MM:SS if you like. I cannot for the life of me find a way to sort these out, but I need to compare these times, hence I want to store them as minutes and seconds, not just numbers.
The first is a counter "TimeSpent",
the second is a counter "TimeStarted" and
the third is a counter "TimeSinceDown".
Then, last, comes the info text, which I've managed to grok with simply applying %{GREEDYDATA:LogInfo}.
I notice that the amount of minutes could be far higher than the standard 60 minutes within an hour, so I may be barking up the wrong tree here trying to parse it with date patterns such as TIMESTAMP_ISO8601, but then, I don't really know how else to do this.
So, I came this far:
%{NUMBER:LogLineID} %{TIMESTAMP_ISO8601:LogDate}
and were as mentioned able to (by cutting away the square bracket parts) to parse the log info text with
%{GREEDYDATA:LogInfo}
to create a field LogInfo.
But that's were I'm stuck. Could someone please help me figure out the rest?
Massive thanks in advance.
PS! I also found %{NUMBER:duration}, but it could as far as I could tell only parse timestamps with dot, not colon..
grok regex expression can help you solve the problem.
but first I wanna make sure that do you mean [325:51] [326:49] [359:57] are the three component that you wanna to fetch? And it will returns the result like :
TimeSpent: 325:51
TimeStarted: 326:49
TimeSinceDown: 359:57
were i get the point , you can use my ways in on of the following suggestions:
define your own custom pattern files and add the pattern in your file.
just use the expression in filter part of logstash conf file
hope it will helps you
Ah, there was a space.. Actually, I was misleading myself and everybody in my question, as it was not actually that log line that was causing problems. I just took the first one, not realizing where the problem really were, but the one causing problems had a space within the brackets as such: [ 42:31]. There are also some parts where there are two spaces, so the way I managed to solve this was to include a %{SPACE} between the \[ and the %{NUMBER}:
%{NUMBER:LogLineID} %{TIMESTAMP_ISO8601:LogDate} \[%{SPACE}%{NUMBER:TimeSpentMinutes}\:%{NUMBER:TimeSpentSeconds}\] \[%{SPACE}%{NUMBER:TimeStartedMinutes}\:%{NUMBER:TimeStartedSeconds}\] \[%{SPACE}%{NUMBER:TimeSinceDownMinutes}\:%{NUMBER:TimeSinceDownSeconds}\] %{GREEDYDATA:LogText}
I still haven't solved the merging of minutes and seconds, but this I can also handle in a later stage.
Thanks to Lin Don for showing an interest in my problem, and sorry for not replying sooner.
Hope the solution will help others (or even myself) if their stuck on the same kind of problem.
Note to myself: Read the logs more carefully before grok'ing.. :)
Among many distressing graphics changes to r2014b, the Plot Browser now only displays a certain number of lines per plot (looks like the limit is 50). Any number of plots above this limit are not displayed in the Plot Browser - it just says "and 78 more..."
Is there anyway to remove the limit? I want to see all my lines in the plot browser.
Unfortunately the answer is currenty:
No you cannot remove this limit
This was already reported to mathworks a while ago, and here is the reply:
I am writing in reference to your Technical Support Case #01143663
regarding 'plot browser with "and xxxx more...." indication'.
Really interesting question (and even a bit surprising). Basically
this limitation has been introduced with MATLAB 2014b!
Our developers are aware of that, and they are working to solve it. An
enhancement/bug request has been already submitted, and I am going to
add this case to the list. However, I cannot guarantee you a release
date.
If you think this limitation is crucial for your work, I would
strongly suggest you to contact your account manager, which will have
a bit more influence on the developers than a simple engineer :).
Of course, if there is anything else I can do for you, please let me
know
So it seems that you will have to deal with this limitation if you keep using 2014b.
This was also an issue for me; in particular I wanted to see the DisplayName property of the graphics object in the list. I used a workaround where I created a callback function, so that when clicking on a data point, the DisplayName would be shown. This can be helpful if you have a plot of many lines and want to see the DisplayName of a particular one. You would first need to set the DisplayName property of the graphics objects for this to work, as it's empty by default. You could also use this to show other properties, such as Color or LineStyle, that are shown in the plot browser:
%Based on
%http://www.mathworks.com/help/matlab/ref/datacursormode.html
%'fig_h' is the figure handle
dcm_obj = datacursormode(fig_h);
set(dcm_obj,'UpdateFcn',#myupdatefcn)
And then include this function as a separate file on the Matlab path, or paste into the function you're currently writing, and include an extra 'end' at the end of that function:
function name = myupdatefcn(empt,event_obj)
% Customizes text of data tips
tar = get(event_obj,'Target');
name = get(tar,'DisplayName');
end
I writing editor with netbeans7 and ANTLR4
I have line in my.g4 file
Label : {(getCharPositionInLine()==0)}? ID;
That works well for static files, but while editing getCharPositionInLine() returns 0 often in other places.
How get a real position in lexer?
updated
I created example with this problem
https://github.com/daimor/SimpleANTLR
Your error is likely in the way you are constructing your lexer and/or input stream (i.e. code that you have not shown here). The predicate you describe above will work as expected for an ANTLR 4 lexer.
Also, if getCharPositionInLine()==0, then exactly one of the following conditions is also true:
_input.index()==0
_input.LA(-1)=='\n'