I am using below command to start the training of deepspeech model
%cd /content/DeepSpeech
!python3 DeepSpeech.py \
--drop_source_layers 2 --scorer /content/DeepSpeech/data/lm/kenlm-nigerian.scorer\
--train_cudnn True --early_stop True --es_epochs 6 --n_hidden 2048 --epochs 5 \
--export_dir /content/models/ --checkpoint_dir /content/model_checkpoints/ \
--train_files /content/train.csv --dev_files /content/dev.csv --test_files /content/test.csv \
--learning_rate 0.0001 --train_batch_size 64 --test_batch_size 32 --dev_batch_size 32 --export_file_name 'he_model_5' \
--max_to_keep 3
I keep getting following error again and again.
(0) Invalid argument: Not enough time for target transition sequence (required: 28, available: 24)0You can turn this error into a warning by using the flag ignore_longer_outputs_than_inputs
(1) Invalid argument: Not enough time for target transition sequence (required: 28, available: 24)0You can turn this error into a warning by using the flag ignore_longer_outputs_than_inputs
Following worked for me
Go to
DeepSpeech/training/deepspeech_training/train.py
Now look for following particular line (Normally in 240-250)
total_loss = tfv1.nn.ctc_loss(labels=batch_y, inputs=logits, sequence_length=batch_seq_len)
Change it to as following
total_loss = tfv1.nn.ctc_loss(labels=batch_y, inputs=logits, sequence_length=batch_seq_len, )
I am following the tutorial in the documentation (https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html) and have been stuck on the "Step 4: Rule parameter" exercise. I would like to access a float from my config file using a wildcard in my params directive.
I seem to be getting the same error whenever I run snakemake -np in the command line:
InputFunctionException in line 46 of /mnt/c/Users/Matt/Desktop/snakemake-tutorial/Snakefile:
Error:
AttributeError: 'Wildcards' object has no attribute 'sample'
Wildcards:
Traceback:
File "/mnt/c/Users/Matt/Desktop/snakemake-tutorial/Snakefile", line 14, in get_bcftools_call_priors
This is my code so far
import time
configfile: "config.yaml"
rule all:
input:
"plots/quals.svg"
def get_bwa_map_input_fastqs(wildcards):
print(wildcards.__dict__, 1, time.time()) #I have this print as a check
return config["samples"][wildcards.sample]
def get_bcftools_call_priors(wildcards):
print(wildcards.__dict__, 2, time.time()) #I have this print as a check
return config["prior_mutation_rates"][wildcards.sample]
rule bwa_map:
input:
"data/genome.fa",
get_bwa_map_input_fastqs
#lambda wildcards: config["samples"][wildcards.sample]
output:
"mapped_reads/{sample}.bam"
params:
rg=r"#RG\tID:{sample}\tSM:{sample}"
threads: 2
shell:
"bwa mem -R '{params.rg}' -t {threads} {input} | samtools view -Sb - > {output}"
rule samtools_sort:
input:
"mapped_reads/{sample}.bam"
output:
"sorted_reads/{sample}.bam"
shell:
"samtools sort -T sorted_reads/{wildcards.sample} "
"-O bam {input} > {output}"
rule samtools_index:
input:
"sorted_reads/{sample}.bam"
output:
"sorted_reads/{sample}.bam.bai"
shell:
"samtools index {input}"
rule bcftools_call:
input:
fa="data/genome.fa",
bam=expand("sorted_reads/{sample}.bam", sample=config["samples"]),
bai=expand("sorted_reads/{sample}.bam.bai", sample=config["samples"])
#prior=get_bcftools_call_priors
params:
prior=get_bcftools_call_priors
output:
"calls/all.vcf"
shell:
"samtools mpileup -g -f {input.fa} {input.bam} | "
"bcftools call -P {params.prior} -mv - > {output}"
rule plot_quals:
input:
"calls/all.vcf"
output:
"plots/quals.svg"
script:
"scripts/plot-quals.py"
and here is my config.yaml
samples:
A: data/samples/A.fastq
#B: data/samples/B.fastq
#C: data/samples/C.fastq
prior_mutation_rates:
A: 1.0e-4
#B: 1.0e-6
I don't understand why my input function call in bcftools_call says that the wildcards object is empty of attributes, yet an almost identical function call in bwa_map has the attribute sample that I want. From the documentation it seems like the wildcards would be propogated before anything is run, so why is it missing?
This is the full output of the commandline call snakemake -np:
{'_names': {'sample': (0, None)}, '_allowed_overrides': ['index', 'sort'], 'index': functools.partial(<function Namedlist._used_attribute at 0x7f91b1a58f70>, _name='index'), 'sort': functools.partial(<function Namedlist._used_attribute at 0x7f91b1a58f70>, _name='sort'), 'sample': 'A'} 1 1628877061.8831172
Job stats:
job count min threads max threads
-------------- ------- ------------- -------------
all 1 1 1
bcftools_call 1 1 1
bwa_map 1 1 1
plot_quals 1 1 1
samtools_index 1 1 1
samtools_sort 1 1 1
total 6 1 1
[Fri Aug 13 10:51:01 2021]
rule bwa_map:
input: data/genome.fa, data/samples/A.fastq
output: mapped_reads/A.bam
jobid: 4
wildcards: sample=A
resources: tmpdir=/tmp
bwa mem -R '#RG\tID:A\tSM:A' -t 1 data/genome.fa data/samples/A.fastq | samtools view -Sb - > mapped_reads/A.bam
[Fri Aug 13 10:51:01 2021]
rule samtools_sort:
input: mapped_reads/A.bam
output: sorted_reads/A.bam
jobid: 3
wildcards: sample=A
resources: tmpdir=/tmp
samtools sort -T sorted_reads/A -O bam mapped_reads/A.bam > sorted_reads/A.bam
[Fri Aug 13 10:51:01 2021]
rule samtools_index:
input: sorted_reads/A.bam
output: sorted_reads/A.bam.bai
jobid: 5
wildcards: sample=A
resources: tmpdir=/tmp
samtools index sorted_reads/A.bam
[Fri Aug 13 10:51:01 2021]
rule bcftools_call:
input: data/genome.fa, sorted_reads/A.bam, sorted_reads/A.bam.bai
output: calls/all.vcf
jobid: 2
resources: tmpdir=/tmp
{'_names': {}, '_allowed_overrides': ['index', 'sort'], 'index': functools.partial(<function Namedlist._used_attribute at 0x7f91b1a58f70>, _name='index'), 'sort': functools.partial(<function Namedlist._used_attribute at 0x7f91b1a58f70>, _name='sort')} 2 1628877061.927639
InputFunctionException in line 46 of /mnt/c/Users/Matt/Desktop/snakemake-tutorial/Snakefile:
Error:
AttributeError: 'Wildcards' object has no attribute 'sample'
Wildcards:
Traceback:
File "/mnt/c/Users/Matt/Desktop/snakemake-tutorial/Snakefile", line 14, in get_bcftools_call_priors
If anyone knows what is going wrong I would really appreciate an explaination. Also if there is a better way of getting information out of the config.yaml into the different directives, I would gladly appreciate those tips.
Edit:
I have searched around the internet quite a bit, but have yet to understand this issue.
Wildcards for each rule are based on that rule's output file(s). The rule bcftools_call has one output file (calls/all.vcf), which has no wildcards. Because of this, when get_bcftools_call_priors is called, it throws an exception when it tries to access the unset wildcards.sample attribute.
You should probably set a global prior_mutation_rate in your config file and then access that in the bcftools_call rule:
rule bcftools_call:
...
params:
prior=config["prior_mutation_rate"],
I came across an odd case, related to operator precendence, I guess. Consider this test program:
use strict;
use warnings;
use Test::More;
my $fn = 'dummy';
ok( ! -e $fn, 'file does not exists' );
ok( not -e $fn, 'file does not exists' );
done_testing();
The output is:
ok 1 - file does not exists
not ok 2
# Failed test at ./p.pl line 10.
1..2
# Looks like you failed 1 test of 2.
The question is: Why does the second test fail? ($fn is assumed known to be non-existent)
See also: List Operator Precedence in Perl.
After reading perlop, my guess is that at least five operators could be involved here:
Terms and List Operators (Leftward)
List Operators (Rightward)
Named Unary Operators
Logical Not
Comma Operator
perl -MO=Deparse shows that your code is interpreted as:
use Test::More;
use warnings;
use strict;
my $fn = 'dummy';
ok !(-e $fn), 'file does not exists';
ok !(-e $fn, 'file does not exists');
done_testing();
-e $fn is false.
But 'file does not exists' is essentially true.
So, the list (-e $fn, 'file does not exists') is true.
Therefore, !(...) is false, and the test fails.
Why does the second test fail?
Because Perl's parser handles ! and not differently. You can see this in Perl's grammar, which is defined in perly.y in the Perl source.
The rule for ! kicks in as soon as the parser encounters a ! followed by a term:
| '!' term /* !$x */
{ $$ = newUNOP(OP_NOT, 0, scalar($2)); }
On the other hand, the rule for not only kicks in when the parser encounters a not followed by a list expression (a list of terms joined by commas*):
| NOTOP listexpr /* not $foo */
{ $$ = newUNOP(OP_NOT, 0, scalar($2)); }
Note that the action for both rules is the same: add a new unary opcode of type OP_NOT to the parse tree. The operand is the second argument (term or listexpr) in scalar context.
* Or a single term, but this has very low precedence.
Tracing the parse
You can see the above rules in action by compiling perl with -DDEBUGGING and running with -Dpv, which turns on debug flags for tokenizing and parsing.
Here's what the parser does with !:
$ perl -Dpv -e'ok(! -e "foo", "bar")'
...
Next token is token '(' (0x1966e98)
Shifting token '(', Entering state 185
Reading a token:
Next token is token '!' (0x1966e98)
Shifting token '!', Entering state 49
Reading a token:
Next token is token UNIOP (0x110)
Shifting token UNIOP, Entering state 39
Reading a token:
Next token is token THING (0x1966e58)
Shifting token THING, Entering state 25
index: 2 3 4 5 6 7 8 9
state: 8 15 103 68 185 49 39 25
token: #1 remember stmtseq amper '(' '!' UNIOP THING
value: 0 22 (Nullop) rv2cv 26635928 26635928 272 const
Reducing stack by rule 184 (line 961), THING -> term
Entering state 128
Reading a token:
Next token is token ',' (0x1966e58)
index: 2 3 4 5 6 7 8 9
state: 8 15 103 68 185 49 39 128
token: #1 remember stmtseq amper '(' '!' UNIOP term
value: 0 22 (Nullop) rv2cv 26635928 26635928 272 const
Reducing stack by rule 199 (line 999), UNIOP term -> term
Entering state 150
Next token is token ',' (0x1966e58)
index: 1 2 3 4 5 6 7 8
state: 1 8 15 103 68 185 49 150
token: GRAMPROG #1 remember stmtseq amper '(' '!' term
value: 0 0 22 (Nullop) rv2cv 26635928 26635928 ftis
Reducing stack by rule 148 (line 829), '!' term -> termunop
Entering state 62
index: 1 2 3 4 5 6 7
state: 1 8 15 103 68 185 62
token: GRAMPROG #1 remember stmtseq amper '(' termunop
value: 0 0 22 (Nullop) rv2cv 26635928 not
...
In other words, the parser reads in
( ! -e "foo"
reduces -e "foo" to a term, and then adds a logical negation opcode to the parse tree. The operand is -e "foo" in scalar context.
Here's what the parser does with not:
$ perl -Dpv -e'ok(not -e "foo", "bar")'
...
Reading a token:
Next token is token '(' (0x26afed8)
Shifting token '(', Entering state 185
Reading a token:
Next token is token NOTOP (0x26afed8)
Shifting token NOTOP, Entering state 48
Reading a token:
Next token is token UNIOP (0x110)
Shifting token UNIOP, Entering state 39
Reading a token:
Next token is token THING (0x26afe98)
Shifting token THING, Entering state 25
index: 2 3 4 5 6 7 8 9
state: 8 15 103 68 185 48 39 25
token: #1 remember stmtseq amper '(' NOTOP UNIOP THING
value: 0 22 (Nullop) rv2cv 40566488 40566488 272 const
Reducing stack by rule 184 (line 961), THING -> term
Entering state 128
Reading a token:
Next token is token ',' (0x26afe98)
index: 2 3 4 5 6 7 8 9
state: 8 15 103 68 185 48 39 128
token: #1 remember stmtseq amper '(' NOTOP UNIOP term
value: 0 22 (Nullop) rv2cv 40566488 40566488 272 const
Reducing stack by rule 199 (line 999), UNIOP term -> term
Entering state 65
Next token is token ',' (0x26afe98)
index: 1 2 3 4 5 6 7 8
state: 1 8 15 103 68 185 48 65
token: GRAMPROG #1 remember stmtseq amper '(' NOTOP term
value: 0 0 22 (Nullop) rv2cv 40566488 40566488 ftis
Reducing stack by rule 105 (line 683), term -> listexpr
Entering state 149
Next token is token ',' (0x26afe98)
Shifting token ',', Entering state 162
Reading a token:
Next token is token THING (0x26afdd8)
Shifting token THING, Entering state 25
index: 3 4 5 6 7 8 9 10
state: 15 103 68 185 48 149 162 25
token: remember stmtseq amper '(' NOTOP listexpr ',' THING
value: 22 (Nullop) rv2cv 40566488 40566488 ftis 40566424 const
Reducing stack by rule 184 (line 961), THING -> term
Entering state 249
Reading a token:
Next token is token ')' (0x26afdd8)
index: 3 4 5 6 7 8 9 10
state: 15 103 68 185 48 149 162 249
token: remember stmtseq amper '(' NOTOP listexpr ',' term
value: 22 (Nullop) rv2cv 40566488 40566488 ftis 40566424 const
Reducing stack by rule 104 (line 678), listexpr ',' term -> listexpr
Entering state 149
Next token is token ')' (0x26afdd8)
index: 1 2 3 4 5 6 7 8
state: 1 8 15 103 68 185 48 149
token: GRAMPROG #1 remember stmtseq amper '(' NOTOP listexpr
value: 0 0 22 (Nullop) rv2cv 40566488 40566488 list
Reducing stack by rule 196 (line 993), NOTOP listexpr -> term
Entering state 65
Next token is token ')' (0x26afdd8)
index: 1 2 3 4 5 6 7
state: 1 8 15 103 68 185 65
token: GRAMPROG #1 remember stmtseq amper '(' term
value: 0 0 22 (Nullop) rv2cv 40566488 not
...
In other words, the parser reads in
( not -e "foo"
reduces -e "foo" to a term, reads in
, "bar"
reduces term, "bar" to a listexpr, and then adds a logical negation opcode to the parse tree. The operand is -e "foo", "bar" in scalar context.
So, even though the opcodes for the two logical negations are the same, their operands are different. You can see this by inspecting the generated parse trees:
$ perl -MO=Concise,-tree -e'ok(! -e "foo", "bar")'
<a>leave[1 ref]-+-<1>enter
|-<2>nextstate(main 1 -e:1)
`-<9>entersub[t1]---ex-list-+-<3>pushmark
|-<6>not---<5>ftis---<4>const(PV "foo")
|-<7>const(PV "bar")
`-ex-rv2cv---<8>gv(*ok)
-e syntax OK
$ perl -MO=Concise,-tree -e'ok(not -e "foo", "bar")'
<c>leave[1 ref]-+-<1>enter
|-<2>nextstate(main 1 -e:1)
`-<b>entersub[t1]---ex-list-+-<3>pushmark
|-<9>not---<8>list-+-<4>pushmark
| |-<6>ftis---<5>const(PV "foo")
| `-<7>const(PV "bar")
`-ex-rv2cv---<a>gv(*ok)
-e syntax OK
With !, the negation acts on the file test:
|-<6>not---<5>ftis
While with not, the negation acts on a list:
|-<9>not---<8>list
You can also dump the parse tree as Perl code using B::Deparse, which shows the same thing in a different format:
$ perl -MO=Deparse,-p -e'ok(! -e "foo", "bar")'
ok((!(-e 'foo')), 'bar');
-e syntax OK
$ perl -MO=Deparse,-p -e'ok(not -e "foo", "bar")'
ok((!((-e 'foo'), 'bar')));
-e syntax OK
With !, the negation acts on the file test:
!(-e 'foo')
While with not, the negation acts on a list:
!((-e 'foo'), 'bar')
And as toolic explained, a list in scalar context evaluates to the last item in the list, giving
ok( ! 'bar' );
where ! 'bar' is falsey.
After rereading the perlop documentation, here is what I believe is going on:
ok( not -e $fn, 'file does not exists' );
Perl parses this statement from left to right. The first thing it encounters is a function call (also called a list operator, if the function is builtin or uses prototypes and operates on lists). The function call ok( ... ). is a described as a TERM in the documentation:
A TERM has the highest precedence in Perl. They include variables,
quote and quote-like operators, any expression in parentheses, and any
function whose arguments are parenthesized.
A list operator (not accurately defined in the perlop page, but briefly mentioned in the perlsub page) is also regarded as a TERM if followed by parenthesis. The perlop says:
If any list operator (print(), etc.) or any unary operator (chdir(),
etc.) is followed by a left parenthesis as the next token, the
operator and arguments within parentheses are taken to be of highest
precedence, just like a normal function call.
Now the parser continues with the expression not -e $fn, 'file does not exists'. That is, it must resolve the arguments to the ok function.
The first thing it encounters here is the not operator. The documentation says:
Unary "not" returns the logical negation of the expression to its
right. It's the equivalent of "!" except for the very low precedence.
Then it must determine "the expression to its right". Here, the parser finds the file test operator -e. The documentation says:
Regarding precedence, the filetest operators, like -f , -M , etc. are
treated like named unary operators, but they don't follow this
functional parenthesis rule. That means, for example, that
-f($file).".bak" is equivalent to -f "$file.bak" .
and
The various named unary operators are treated as functions with one
argument, with optional parentheses.
Now the unary operators (without a following parenthesis) have higher precendence than the not operator, so the parser continues, trying to determine the argument of the -e operator. It now encounters a new TERM, (we are now considering this expression: $fn, 'file does not exists' ). The TERM is $fn and since TERMs have the highest precedence, it is evaluated immediately. Then it continues to the comma operator. Since the comma operator has lower precedence than the filetest operator, and the filetest operator is unary (only takes a single argument), the parser decides it is finished with the argument of the filetest operator and evaluates -e $fn. Then it proceeds with the comma:
Binary "," is the comma operator. In scalar context it evaluates its
left argument, throws that value away, then evaluates its right
argument and returns that value. This is just like C's comma operator.
In list context, it's just the list argument separator, and inserts
both its arguments into the list. These arguments are also evaluated
from left to right.
Since the comma operator has higher precedence than the not operator the parser finds it is still not finished with the argument of not. Instead it discovers that the argument of not is a list (due to the comma operator), it has already evaluated the left argument of the comma operator, -e $fn, and discards that value, and proceeds with the right argument of the comma operator which is the string 'file does not exists'. This is evaluated, and the parser then finds the closing parenthesis ), which means that the argument of not is the latter string. Negating a nonempty string is false.
And finally, the parser finds that the argument to the ok function is false, and runs ok( 0 ).
To relate directly to perlop, all you need to notice is that ! is above , is above not in the table:
right ! ~ \ and unary + and -
...
left , =>
...
right not
That is, the comma binds things together "more tightly" than not does, but "less tightly" than ! does.
Is it possible to read a znode with a space in it through Zookeeper CLI?
I've 2 values under regions ('us-west 1' and 'us-east')
[zk: localhost:2181(CONNECTED) 11] get /regions/
us-west 1 us-east
I can read 'us-east'.
[zk: localhost:2181(CONNECTED) 11] get /regions/us-east
null
cZxid = 0xa
ctime = Tue Jul 10 12:41:49 IST 2012
mZxid = 0xa
mtime = Tue Jul 10 12:41:49 IST 2012
pZxid = 0x1b
cversion = 9
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 9
But not 'us-west 1'
[zk: localhost:2181(CONNECTED) 11] get /regions/us-west 1
Node does not exist: /regions/us-west
I tried options like '%20', '\ ' , '+' etc.. for the space, but nothing worked.
Please try zookeepercli, as follows:
$ zookeepercli --servers srv1,srv2,srv3 -c create "/demo_only 1" "the value"
$ zookeepercli --servers srv1,srv2,srv3 -c get "/demo_only 1"
the value
zookeepercli is free and open source.
Disclaimer: I'm author of this tool.
It looks like you will not be able to do that from the ZK command-line client. The Zookeeper Java client (which is the one you are using, probably) separates commands (e.g. get) from their parameters (e.g. /regions/us-west 1 in your case) by parsing whitespace characters, as you can see in the code provided with the client (e.g. zookeeper-3.3.5\src\java\main\org\apache\zookeeper\ZooKeeperMain.java):
public boolean parseCommand( String cmdstring ) {
String[] args = cmdstring.split(" ");
if (args.length == 0){
return false;
}
command = args[0];
cmdArgs = Arrays.asList(args);
return true;
}
Since they are splitting by a " ", unless you discover a way to overcome that split call by some sort of escaping when calling the get command, you won't be able to retrieve those nodes using this client. The code above interprets your 1 in a get call as if it were the watch parameter, as you can see in the get command syntax:
get path [watch]
I recommend you to use a different character, like '_' for example, instead of the whitespace on the znodes naming. If that is not an option, you will either need to modify the ZK Java client yourself, or use another client.
There's an open issue in JIRA for this feature, but a workaround is to pass a command on the command-line instead of using the interactive console:
$ zkCli.sh -c get "/regions/us-west 1"
After setting db to some arbitrary value, i am not able to perform any operation in shell.
Is it know bug or expected behavior?
> use tutorial
switched to db tutorial
> db
tutorial
> db = 5
5
> db
5
> show dbs
Fri Mar 23 17:18:40 TypeError: db.getMongo is not a function shell/utils.js:1235
>
> use tutorial
Fri Mar 23 17:18:55 TypeError: db.getMongo is not a function shell/utils.js:1167
> db = 'tutorial'
tutorial
> show dbs
Fri Mar 23 17:19:38 TypeError: db.getMongo is not a function shell/utils.js:1235
The Mongo Interactive Shell is a Javascript Shell, and hence it obeys all laws of a Javascript Shell. You are overriding the db variable that got initialized during startup.
> a = db
SocialStreams
> db = "Hello"
Hello
> db.help()
Fri Mar 23 12:08:13 TypeError: db.help is not a function (shell):1
> db = a
SocialStreams
> db.help()
DB methods:
db.addUser(username, password[, readOnly=false])
db.auth(username, password)
db.cloneDatabase(fromhost)
...