Currently I run prove like this:
prove -v -r .
Is there a way to exclude files using regex? I could not see that in perldoc prove or the output of prove -H.
I usually do something like this:
$ find t/ -name '*.t' ! -name '*timeout*' | xargs prove -l
(skips slow timeout tests)
I managed to do this by using a sub-directory :
$ tree t/
t/
├── 01_tests.t
├── 02_tests.t
├── 03_tests.t
└── ignored
└── 01_ignored.t
Then I do this to execute the normal tests:
$ prove -v t/*.t
t/01_tests.t ........ ok
t/02_tests.t ........ ok
t/03_tests.t ........ ok
Result: PASS
And this to execute the ignored tests in another context:
$ prove -v t/ignored/
t/ignored/01_ignored.t ........ ok
Result: PASS
Simple and efficient solution if your tests are not already stored in different sub-directories.
For making it easier to use, and document how to run the tests, I usually put those in a Makefile.
Related
I am using Debian Linux. I'm a newbie. I'll do my best to ask in the simplest way I know.
I have a pretty deep tree of directories on a drive that contain thousands of .tif files and .txt files. I'd like to recursively find (list) all .txt files that do not have a matching .tif file (basename). The .tif files and .txt files are also located in separate directories throughout the tree.
In simple form it could look like this...
directory1: hf-770.tif, hf-771.tif, hf-772.tif
directory2: hf-770.txt, hf-771.txt, hf-771.txt, hr-001.txt, tb-789.txt
I need to find (list) hr-001.txt and tb-789.txt as they do not have a matching .tif file. Again the directory tree is quite deep with multiple sub-directories throughout.
I researched and experimented with variations of the following commands but cannot seem to make it work. Thank you so much.
find -name "*.tif" -name "*.txt" | ls -1 | sed 's/\([^.]*\).*/\1/' | uniq
You can write a shell script for this:
#!/bin/bash
set -ue
while IFS= read -r -d '' txt
do
tif=$(basename "$txt" | sed s/\.txt$/.tif/)
found=$(find . -name "$tif")
if [ -z "$found" ]
then
echo "$txt has no tif"
fi
done < <(find . -name \*.txt -print0)
This has a loop over all .txt files it finds in the current directory or below. For each found file, it replaces the .txt extension with .tif, then tries to find that file. If it cannot find it (returned text is empty), it prints the .txt file name.
robert#saaz:$ tree
.
├── bar
│ └── a.txt
├── foo
│ ├── a.tif
│ ├── b.tif
│ ├── c.tif
│ └── d.txt
└── txt-without-tif
2 directories, 6 files
robert#saaz:$ bash txt-without-tif
./foo/d.txt has no tif
I am using MobaXterm portable.
I found a strange setup, summarized here.
External commands in /bin work fine. E.g., with /bin/ssh.exe I can ssh ok.
Internal commands are
"redirected" to busybox, as
$ which cat
/bin/cat
$ ll /bin/cat
lrwxrwxrwx 1 USER001 UsersGrp 16 Jul 24 07:42 /bin/cat -> /bin/busybox.exe
at the same time aliased to files that apparently do not exist.
$ type cat
cat is aliased to `/bin/cat.exe'
These aliases apparently take precedence over files in PATH, so the commands do not work.
$ cat myfile
bash: /bin/cat.exe: No such file or directory
If I unalias, cat does not look for /bin/cat.exe but for /bin/busybox.exe, and everything is "back to normal".
$ unalias cat
$ cat myfile
Hello world
...
How can I get normal behaviour (either without aliases or with the presence of the alias targets)?
I mean not to write my own unaliases in .bashrc, this shouldn´t be needed.
Moreover, perhaps I would be breaking something.
Why would MobaXterm setup things like this?
PS: In the initial state, even ls does not work, for the same reason.
But ll works, since
$ type ll
ll is aliased to `_bbf ls -l'
$ type _bbf
_bbf is a function
...
How can I get normal behaviour?
Workarounds:
unaliasing by hand, so /bin/busybox.exe is actually used.
Below I add a script for that.
Copying .exe files from the temporary root dir when it is available, so the external versions are used.
Why would MobaXterm setup things like this?
When not using a Persistent root (/) directory, this is obtained
$ which cat
/bin/cat
$ ll /bin/cat
-rwxr-xr-x 1 RY16205 UsersGrp 49703 jul. 28 07:12 /bin/cat
$ type cat
cat is aliased to `/bin/cat.exe'
$ ll /bin/cat.exe
-rwxr-xr-x 1 USER001 UsersGrp 49703 jul. 28 07:12 /bin/cat.exe
$ cat myfile
Hello world
...
$ unalias cat
$ type cat
cat is hashed (/bin/cat)
$ cat myfile
Hello world
...
So any of the two cats work (internal busybox and external versions; I do not know if they are exactly the same).
This is because /bin points to C:\Users\user001\AppData\Local\Temp\Mxt108\bin and cat.exe is there.
But when using a Persistent root (/) directory, /bin points to <Persistent root (/) directory\bin, and cat.exe is not created there.
The former temporary root dir is removed as soon as MXT is closed.
So this is probably a configuration error from MobaXterm.
If so, the only option seems a workaround, as above.
Script for unaliasing:
#!/bin/bash
export ROOTDIR_WIN=$(cygpath -wl /)
if [[ ${ROOTDIR_WIN##*\\} == "Mxt108" ]] ; then
# Not using a Persistent root dir. Do not need to unalias.
echo "Not using a Persistent root dir. Do not need to unalias."
else
# Using a Persistent root dir. Need to unalias.
exe_linux_list="bash busybox chmod cygstart cygtermd cygwin-console-helper dircolors dwm_w32 echo grep ls MoTTY ssh ssh-pageant test twm_w32 wc xkbcomp_w32 XWin_MobaX"
for exe_linux in ${exe_linux_list} ; do
if [[ $(type -t ${exe_linux}) == "alias" ]] ; then
#type ${exe_linux}
unalias ${exe_linux}
fi
done
fi
On my MobaXterm system, /etc/profile sources /etc/baseprofile which includes aliases for all of these sorts of things, i.e.
alias "cat"="_bbf cat"
and checking that from my command prompt yields what I would expect:
$ type cat
cat is aliased to `_bbf cat'
Have you changed your system somehow so that /etc/baseprofile is not being sourced? Or have you changed /etc/baseprofile?
It also appears that you've installed the regular GNU Coreutils package, as I don't have a /bin/cat.exe.
$ ll /bin/cat.exe
ls: /bin/cat.exe: No such file or directory
Perhaps that's where your problem started but the _bbf function is supposed to handle that. Which again leads me to the belief that you've changed /etc/baseprofile somehow.
At most time, it is cool. This error maybe caused by wrong path match of cat.exe.
As for me, when I run git log, the same error message comes out. It is due to PATH variable. There are two dirs and both of them contain git.exe. One of them is half-done with a small size. And Mobaxterm choose it. :D
I confirm this by run which git and it will give the actual path out.
I fix it by
alias git='/drives/C/Program\ Files/Git/mingw64/bin/git.exe'
The following is my dirs.
├─cmd
│ git-gui.exe
│ git-lfs.exe
│ git.exe # oops
│ gitk.exe
│ start-ssh-agent.cmd
│ start-ssh-pageant.cmd
├─mingw64
│ ├─bin
│ │ git-upload-pack.exe
│ │ git.exe # the right one
This is my tree
├── test
│ ├── dir1
│ └── dir2
│ ├── file
│ └── file2
└── test2
└── dir2
├── file
└── file2
I use diff: diff -r test/ test2/
Only in test: dir1
So the only difference is that there is an empty directory (dir1) in in test/ which does not exist in test2/.
I want to ignore empty directories as a difference. So I want in this case that diff tells me that the content of test/ is the same as the content of test2/
How can I achieve this?
I found an way of doing it but i'm not really happy with it.
diff can be told to exclude files matching a pattern. Sadly the pattern only works on filename, so my solution may exclude more directories (and files too) than expected.
Here it is, fwiw :
diff $(find test -empty -type d -exec sh -c 'echo -n "-x $(basename $1) "' _ {} \;) -r test/ test2/
I added the following command
find test -empty -type d -exec sh -c 'echo -n "-x $(basename $1) "' _ {} \;)
which outputs the basename of every empty dir preceding by diff's exclude option -x. With your example tree, it would output : -x dir1
My objective is to get the number of test methods in a package/folder. I'm able to do that by executing
py.test <folder> --collect-only|grep collected
This shows the test count as
collected 104 items
However this counts the parameterized test multiple times,e.g. if a method has two sets of parameter single test will be counted 2 times.
Is there any way to tell pytest to count them as single?
If your tests in pytest use custom test collection method or use parametrization, regular grepping won't be helpful. Regular grepping can only give you test function and test classes. If you are interested in that number, other answers here should work fine for you. If you want to know the total number of collected tests, follow along.
You should run pytest --collect-only for target test directory and run grep on the output.
One possible solution is this:
pytest --collect-only | grep "<Function\|<Class" -c
It will return the number of lines which has <Function or <Class in the output of pytest --collect-only. Since all collected tests have either of these words, it will give correct number of tests.
Another hacky way to find the number of tests is to use -k switch. It searches for expression and run the tests which match the expression.
pytest -k "not test and not Test"
This will give you number of all tests. What it does is collect all the tests and tries to find the test which does not have test in the test name. Since all tests has test in the name, all the tests would be deselected and you would get your total number of tests. This method works with parametrization.
Loosely based on my other answer.
Count all tests
One-liner:
$ pytest --collect-only -q | head -n -2 | wc -l
Explanation
--collect-only combined with -q outputs one test per line, with a trailing info line. Example:
$ pytest --collect-only -q
test_eggs.py::test_bacon[1]
test_eggs.py::test_bacon[2]
test_eggs.py::test_bacon[3]
test_spam.py::test_foo
test_spam.py::test_bar
no tests ran in 0.00 seconds
The rest is just routine: head -n -2 strips the info line, wc -l counts the lines.
Applying further filtering works as usual, e.g.
$ pytest --collect-only -q -k "fizz" | head -n -2 | wc -l
will count only tests containing fizz in name,
$ pytest --collect-only -q buzz/ fuzz/ | head -n -2 | wc -l
will count only tests inside buzz and fuzz directories etc.
Count tests per test module
If you want to get the info about how many tests are in each module, use --collect-only combined with -qq:
$ pytest --collect-only -qq
test_eggs.py: 3
test_spam.py: 2
Count unique tests (test parametrizations counting as single test)
What the OP initially requested. This is a modification of the above command that strips the parametrization from test names and removes duplicate lines before counting:
$ pytest --collect-only -q | head -n -2 | sed 's/\[.*\]$//' | sort | uniq | wc -l
How about
find . -type f -name 'test*.py' -exec grep -e 'def test_' '{}' \; | wc -l
or
ag 'def test_' | wc -l
One more solution:
egrep -e 'def test_' -r ./ | wc -l
Hope it can help.
I suppose I could compare the number of files in the source directory to the number of files in the target directory as cp progresses, or perhaps do it with folder size instead? I tried to find examples, but all bash progress bars seem to be written for copying single files. I want to copy a bunch of files (or a directory, if the former is not possible).
You can also use rsync instead of cp like this:
rsync -Pa source destination
Which will give you a progress bar and estimated time of completion. Very handy.
To show a progress bar while doing a recursive copy of files & folders & subfolders (including links and file attributes), you can use gcp (easily installed in Ubuntu and Debian by running "sudo apt-get install gcp"):
gcp -rf SRC DEST
Here is the typical output while copying a large folder of files:
Copying 1.33 GiB 73% |##################### | 230.19 M/s ETA: 00:00:07
Notice that it shows just one progress bar for the whole operation, whereas if you want a single progress bar per file, you can use rsync:
rsync -ah --progress SRC DEST
You may have a look at the tool vcp. Thats a simple copy tool with two progress bars: One for the current file, and one for overall.
EDIT
Here is the link to the sources: http://members.iinet.net.au/~lynx/vcp/
Manpage can be found here: http://linux.die.net/man/1/vcp
Most distributions have a package for it.
Here another solution: Use the tool bar
You could invoke it like this:
#!/bin/bash
filesize=$(du -sb ${1} | awk '{ print $1 }')
tar -cf - -C ${1} ./ | bar --size ${filesize} | tar -xf - -C ${2}
You have to go the way over tar, and it will be inaccurate on small files. Also you must take care that the target directory exists. But it is a way.
My preferred option is Advanced Copy, as it uses the original cp source files.
$ wget http://ftp.gnu.org/gnu/coreutils/coreutils-8.21.tar.xz
$ tar xvJf coreutils-8.21.tar.xz
$ cd coreutils-8.21/
$ wget --no-check-certificate wget https://raw.githubusercontent.com/jarun/advcpmv/master/advcpmv-0.8-8.32.patch
$ patch -p1 -i advcpmv-0.8-8.32.patch
$ ./configure
$ make
The new programs are now located in src/cp and src/mv. You may choose to replace your existing commands:
$ sudo cp src/cp /usr/local/bin/cp
$ sudo cp src/mv /usr/local/bin/mv
Then you can use cp as usual, or specify -g to show the progress bar:
$ cp -g src dest
A simple unix way is to go to the destination directory and do watch -n 5 du -s . Perhaps make it more pretty by showing as a bar . This can help in environments where you have just the standard unix utils and no scope of installing additional files . du-sh is the key , watch is to just do every 5 seconds.
Pros : Works on any unix system Cons : No Progress Bar
To add another option, you can use cpv. It uses pv to imitate the usage of cp.
It works like pv but you can use it to recursively copy directories
You can get it here
There's a tool pv to do this exact thing: http://www.ivarch.com/programs/pv.shtml
There's a ubuntu version in apt
How about something like
find . -type f | pv -s $(find . -type f | wc -c) | xargs -i cp {} --parents /DEST/$(dirname {})
It finds all the files in the current directory, pipes that through PV while giving PV an estimated size so the progress meter works and then piping that to a CP command with the --parents flag so the DEST path matches the SRC path.
One problem I have yet to overcome is that if you issue this command
find /home/user/test -type f | pv -s $(find . -type f | wc -c) | xargs -i cp {} --parents /www/test/$(dirname {})
the destination path becomes /www/test/home/user/test/....FILES... and I am unsure how to tell the command to get rid of the '/home/user/test' part. That why I have to run it from inside the SRC directory.
Check the source code for progress_bar in the below git repository of mine
https://github.com/Kiran-Bose/supreme
Also try custom bash script package supreme to verify how progress bar work with cp and mv comands
Functionality overview
(1)Open Apps
----Firefox
----Calculator
----Settings
(2)Manage Files
----Search
----Navigate
----Quick access
|----Select File(s)
|----Inverse Selection
|----Make directory
|----Make file
|----Open
|----Copy
|----Move
|----Delete
|----Rename
|----Send to Device
|----Properties
(3)Manage Phone
----Move/Copy from phone
----Move/Copy to phone
----Sync folders
(4)Manage USB
----Move/Copy from USB
----Move/Copy to USB
There is command progress, https://github.com/Xfennec/progress, coreutils progress viewer.
Just run progress in another terminal to see the copy/move progress. For continuous monitoring use -M flag.