How does one extract a unified-diff style patch subset? - diff

Every time I want to take a subset of a patch, I'm forced to write a script to only extract the indices that I want.
e.g. I have a patch that applies to sub directories
'yay' and 'foo'.
Is there a way to create a new patch or apply only a subset of a patch? i.e. create a new patch from the existing patch that only takes all indices that are under sub directory 'yay'. Or all indices that are not under sub directory 'foo'
If I have a patch like ( excuse the below pseudo-patch):
Index : foo/bar
yada
yada
- asdf
+ jkl
yada
yada
Index : foo/bah
blah
blah
- 28
+ 29
blah
blah
blah
Index : yay/team
go
huskies
- happy happy
+ joy joy
cougars
suck
How can I extract or apply only the 'yay' subdirectory like:
Index : yay/team
go
huskies
- happy happy
+ joy joy
cougars
suck
I know if I script up a solution I'll be re-inventing the wheel...

Take a look at the filterdiff utility, which is part of patchutils.
For example, if you have the following patch:
$ cat example.patch
diff -Naur orig/a/bar new/a/bar
--- orig/a/bar 2009-12-02 12:41:38.353745751 -0800
+++ new/a/bar 2009-12-02 12:42:17.845745951 -0800
## -1,3 +1,3 ##
4
-5
+e
6
diff -Naur orig/a/foo new/a/foo
--- orig/a/foo 2009-12-02 12:41:32.845745768 -0800
+++ new/a/foo 2009-12-02 12:42:25.697995617 -0800
## -1,3 +1,3 ##
1
2
-3
+c
diff -Naur orig/b/baz new/b/baz
--- orig/b/baz 2009-12-02 12:41:42.993745756 -0800
+++ new/b/baz 2009-12-02 12:42:37.585745735 -0800
## -1,3 +1,3 ##
-7
+z
8
9
Then you can run the following command to extract the patch for only things in the a directory like this:
$ cat example.patch | filterdiff -i 'new/a/*'
--- orig/a/bar 2009-12-02 12:41:38.353745751 -0800
+++ new/a/bar 2009-12-02 12:42:17.845745951 -0800
## -1,3 +1,3 ##
4
-5
+e
6
--- orig/a/foo 2009-12-02 12:41:32.845745768 -0800
+++ new/a/foo 2009-12-02 12:42:25.697995617 -0800
## -1,3 +1,3 ##
1
2
-3
+c

Here's my quick and dirty Perl solution.
perl -ne '#a = split /^Index :/m, join "", <>; END { for(#a) {print "Index :", $_ if (m, yay/team,)}}' < foo.patch

In response to sigjuice's request in the comments, I'm posting my script solution. It isn't 100% bullet proof, and I'll probably use filterdiff instead.
base_usage_str=r'''
python %prog index_regex patch_file
description:
Extracts all indices from a patch-file matching 'index_regex'
e.g.
python %prog '^evc_lib' p.patch > evc_lib_p.patch
Will extract all indices which begin with evc_lib.
-or-
python %prog '^(?!evc_lib)' p.patch > not_evc_lib_p.patch
Will extract all indices which do *not* begin with evc_lib.
authors:
Ross Rogers, 2009.04.02
'''
import re,os,sys
from optparse import OptionParser
def main():
parser = OptionParser(usage=base_usage_str)
(options, args) = parser.parse_args(args=sys.argv[1:])
if len(args) != 2:
parser.print_help()
if len(args) == 0:
sys.exit(0)
else:
sys.exit(1)
(index_regex,patch_file) = args
sys.stderr.write('Extracting patches for indices found by regex:%s\n'%index_regex)
#print 'user_regex',index_regex
user_index_match_regex = re.compile(index_regex)
# Index: verification/ring_td_cte/tests/mmio_wr_td_target.e
# --- sw/cfg/foo.xml 2009-04-30 17:59:11 -07:00
# +++ sw/cfg/foo.xml 2009-05-11 09:26:58 -07:00
index_cre = re.compile(r'''(?:(?<=^)|(?<=\n))(--- (?:.*\n){2,}?(?![ #\+\-]))''')
patch_file = open(patch_file,'r')
all_patch_sets = index_cre.findall(patch_file.read())
patch_file.close()
for file_edit in all_patch_sets:
# extract index subset
index_path = re.compile('\+\+\+ (?P<index>[\w_\-/\.]+)').search(file_edit).group('index').strip()
if user_index_match_regex.search(index_path):
sys.stderr.write("Index regex matched index: "+index_path+"\n")
print file_edit,
if __name__ == '__main__':
main()

Related

Why are my wildcard attributes not being filled in Snakemake?

I am following the tutorial in the documentation (https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html) and have been stuck on the "Step 4: Rule parameter" exercise. I would like to access a float from my config file using a wildcard in my params directive.
I seem to be getting the same error whenever I run snakemake -np in the command line:
InputFunctionException in line 46 of /mnt/c/Users/Matt/Desktop/snakemake-tutorial/Snakefile:
Error:
AttributeError: 'Wildcards' object has no attribute 'sample'
Wildcards:
Traceback:
File "/mnt/c/Users/Matt/Desktop/snakemake-tutorial/Snakefile", line 14, in get_bcftools_call_priors
This is my code so far
import time
configfile: "config.yaml"
rule all:
input:
"plots/quals.svg"
def get_bwa_map_input_fastqs(wildcards):
print(wildcards.__dict__, 1, time.time()) #I have this print as a check
return config["samples"][wildcards.sample]
def get_bcftools_call_priors(wildcards):
print(wildcards.__dict__, 2, time.time()) #I have this print as a check
return config["prior_mutation_rates"][wildcards.sample]
rule bwa_map:
input:
"data/genome.fa",
get_bwa_map_input_fastqs
#lambda wildcards: config["samples"][wildcards.sample]
output:
"mapped_reads/{sample}.bam"
params:
rg=r"#RG\tID:{sample}\tSM:{sample}"
threads: 2
shell:
"bwa mem -R '{params.rg}' -t {threads} {input} | samtools view -Sb - > {output}"
rule samtools_sort:
input:
"mapped_reads/{sample}.bam"
output:
"sorted_reads/{sample}.bam"
shell:
"samtools sort -T sorted_reads/{wildcards.sample} "
"-O bam {input} > {output}"
rule samtools_index:
input:
"sorted_reads/{sample}.bam"
output:
"sorted_reads/{sample}.bam.bai"
shell:
"samtools index {input}"
rule bcftools_call:
input:
fa="data/genome.fa",
bam=expand("sorted_reads/{sample}.bam", sample=config["samples"]),
bai=expand("sorted_reads/{sample}.bam.bai", sample=config["samples"])
#prior=get_bcftools_call_priors
params:
prior=get_bcftools_call_priors
output:
"calls/all.vcf"
shell:
"samtools mpileup -g -f {input.fa} {input.bam} | "
"bcftools call -P {params.prior} -mv - > {output}"
rule plot_quals:
input:
"calls/all.vcf"
output:
"plots/quals.svg"
script:
"scripts/plot-quals.py"
and here is my config.yaml
samples:
A: data/samples/A.fastq
#B: data/samples/B.fastq
#C: data/samples/C.fastq
prior_mutation_rates:
A: 1.0e-4
#B: 1.0e-6
I don't understand why my input function call in bcftools_call says that the wildcards object is empty of attributes, yet an almost identical function call in bwa_map has the attribute sample that I want. From the documentation it seems like the wildcards would be propogated before anything is run, so why is it missing?
This is the full output of the commandline call snakemake -np:
{'_names': {'sample': (0, None)}, '_allowed_overrides': ['index', 'sort'], 'index': functools.partial(<function Namedlist._used_attribute at 0x7f91b1a58f70>, _name='index'), 'sort': functools.partial(<function Namedlist._used_attribute at 0x7f91b1a58f70>, _name='sort'), 'sample': 'A'} 1 1628877061.8831172
Job stats:
job count min threads max threads
-------------- ------- ------------- -------------
all 1 1 1
bcftools_call 1 1 1
bwa_map 1 1 1
plot_quals 1 1 1
samtools_index 1 1 1
samtools_sort 1 1 1
total 6 1 1
[Fri Aug 13 10:51:01 2021]
rule bwa_map:
input: data/genome.fa, data/samples/A.fastq
output: mapped_reads/A.bam
jobid: 4
wildcards: sample=A
resources: tmpdir=/tmp
bwa mem -R '#RG\tID:A\tSM:A' -t 1 data/genome.fa data/samples/A.fastq | samtools view -Sb - > mapped_reads/A.bam
[Fri Aug 13 10:51:01 2021]
rule samtools_sort:
input: mapped_reads/A.bam
output: sorted_reads/A.bam
jobid: 3
wildcards: sample=A
resources: tmpdir=/tmp
samtools sort -T sorted_reads/A -O bam mapped_reads/A.bam > sorted_reads/A.bam
[Fri Aug 13 10:51:01 2021]
rule samtools_index:
input: sorted_reads/A.bam
output: sorted_reads/A.bam.bai
jobid: 5
wildcards: sample=A
resources: tmpdir=/tmp
samtools index sorted_reads/A.bam
[Fri Aug 13 10:51:01 2021]
rule bcftools_call:
input: data/genome.fa, sorted_reads/A.bam, sorted_reads/A.bam.bai
output: calls/all.vcf
jobid: 2
resources: tmpdir=/tmp
{'_names': {}, '_allowed_overrides': ['index', 'sort'], 'index': functools.partial(<function Namedlist._used_attribute at 0x7f91b1a58f70>, _name='index'), 'sort': functools.partial(<function Namedlist._used_attribute at 0x7f91b1a58f70>, _name='sort')} 2 1628877061.927639
InputFunctionException in line 46 of /mnt/c/Users/Matt/Desktop/snakemake-tutorial/Snakefile:
Error:
AttributeError: 'Wildcards' object has no attribute 'sample'
Wildcards:
Traceback:
File "/mnt/c/Users/Matt/Desktop/snakemake-tutorial/Snakefile", line 14, in get_bcftools_call_priors
If anyone knows what is going wrong I would really appreciate an explaination. Also if there is a better way of getting information out of the config.yaml into the different directives, I would gladly appreciate those tips.
Edit:
I have searched around the internet quite a bit, but have yet to understand this issue.
Wildcards for each rule are based on that rule's output file(s). The rule bcftools_call has one output file (calls/all.vcf), which has no wildcards. Because of this, when get_bcftools_call_priors is called, it throws an exception when it tries to access the unset wildcards.sample attribute.
You should probably set a global prior_mutation_rate in your config file and then access that in the bcftools_call rule:
rule bcftools_call:
...
params:
prior=config["prior_mutation_rate"],

Can I patch the patch?

In our CI we are patching files, the problem appears when we make changes in files which we patch, can I applay commit changes to patch file?
example:
text.txt
A
B
C
D
patch.patch
+++ b/text2.txt
## -1,4 +1,4 ##
A
B
-C
+X
D
new.txt:
Y
Y
B
C
D
diff text.txt new.txt > text_to_new.diff
diff --git a/text.txt b/new.txt
index 8422d40..4780582 100644
--- a/text.txt
+++ b/new.txt
## -1,4 +1,5 ##
-A
+Y
+Y
B
C
D
Can I update patch.patch with text_to_new.diff?

what is the good way to edit the device tree ? and where is it ? (meta-sunxi)

I have currently build a Yocto core-minimal-images (with meta-sunxi) for an orange pi zero board(a cheap chinese board that i use for my studies)
https://github.com/linux-sunxi/meta-sunxi
And it succesfully boot on my board,but in the /dev directory i didnt have acces to the SPI NOR memory. After some search on the orange pi wiki i find that i need some line to my device tree : https://linux-sunxi.org/Orange_Pi_Zero#Installing_from_linux
&spi0 {
status = "okay";
flash: m25p80#0 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "winbond,w25q128";
reg = <0>;
spi-max-frequency = <40000000>;
};
};
But i dont really understand how to proceed...because i don't find which files i need to edit ? and maybe this not a good idee ? i think its better to create a .bbappend recipes no ?
the information that i have gather by searching in the meta-sunxi directories:
in conf/orange-pi-zero/KERNEL_DEVICETREE = "sun8i-h2-plus-orangepi-zero.dtb"
but there is no "sun8i-h2-plus-orangepi-zero.dts" file in the meta-sunxi directories ?
"sun8i-h2-plus-orangepi-zero.dtb"file is present in /build/tmp/deploy/images/orange-pi-zero/ so i don't really know how it is generated ? is it only download by yocto ? ( no device tree compilation ? )
by serachin on the net i was able to find sun8i-h2-plus-orangepi-zero.dts
at: https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts
and it contains theses interesting lines :
&spi0 {
/* Disable SPI NOR by default: it optional on Orange Pi Zero boards */
status = "disabled";
flash#0 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "mxicy,mx25l1606e", "winbond,w25q128";
reg = <0>;
spi-max-frequency = <40000000>;
};
};
So maybe someone is able to give some advice to add SPI NOR support on my board ? what is the best way ? make some .bbappend ? or create my own meta by copying "meta-sunxi" and edit it ? and then which files i need to edit ?
thanks in advance for your time
Pierre.
Compiling the image with Yocto with the meta BSP layer pulls the kernel (checksout into tmp/work-shared/<MACHINE>/kernel-source/) and compiles it and you get the final output image which you can flash from tmp/deploy/images/<MACHINE>/. But in your case the mainline kernel doesn't enabled the SPI by default, so you need to enable it in the Linux Kernel source code.
If you have the Yocto build setup already, then you can edit the Device Tree and prepare the patch. You can move into tmp/work-shared/orange-pi-zero/kernel-source/ and edit the kernel source code and change
status = "okay";
and prepare the git patch using usually sequence,
git add arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts
git commit -s -m "Enable SPI by default"
git format-patch HEAD~
Then you append this patch in two ways.
Edit the recipes-kernel/linux/linux-mainline_git.bb and add your patch file into SRC_URI. Copy the patch file into recipes-kernel/linux/linux-mainline
If you don't want to edit the meta-sunxi layer, create linux-mainline_%.bbappend in your meta layer and do the same.
The below patch can be directly applied to meta-sunxi to fix this case. You can find the same here.
From 3a1a3515d33facdf8ec9ab9735fb9244c65521be Mon Sep 17 00:00:00 2001
From: Parthiban Nallathambi <parthiban#linumiz.com>
Date: Sat, 10 Nov 2018 12:20:41 +0100
Subject: [PATCH] orange pi zero: Add SPI support by default
Signed-off-by: Parthiban Nallathambi <parthiban#linumiz.com>
---
...rm-dts-enable-SPI-for-orange-pi-zero.patch | 26 +++++++++++++++++++
recipes-kernel/linux/linux-mainline_git.bb | 1 +
2 files changed, 27 insertions(+)
create mode 100644 recipes-kernel/linux/linux-mainline/0001-arm-dts-enable-SPI-for-orange-pi-zero.patch
diff --git a/recipes-kernel/linux/linux-mainline/0001-arm-dts-enable-SPI-for-orange-pi-zero.patch b/recipes-kernel/linux/linux-mainline/0001-arm-dts-enable-SPI-for-orange-pi-zero.patch
new file mode 100644
index 0000000..e6d7933
--- /dev/null
+++ b/recipes-kernel/linux/linux-mainline/0001-arm-dts-enable-SPI-for-orange-pi-zero.patch
## -0,0 +1,26 ##
+From 1676d9767686404211c769de40e6aa55642b63d5 Mon Sep 17 00:00:00 2001
+From: Parthiban Nallathambi <parthiban#linumiz.com>
+Date: Sat, 10 Nov 2018 12:16:36 +0100
+Subject: [PATCH] arm: dts: enable SPI for orange pi zero
+
+Signed-off-by: Parthiban Nallathambi <parthiban#linumiz.com>
+---
+ arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts b/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts
+index 0bc031fe4c56..0036065da81c 100644
+--- a/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts
++++ b/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts
+## -144,7 +144,7 ##
+
+ &spi0 {
+ /* Disable SPI NOR by default: it optional on Orange Pi Zero boards */
+- status = "disabled";
++ status = "okay";
+
+ flash#0 {
+ #address-cells = <1>;
+--
+2.17.2
+
diff --git a/recipes-kernel/linux/linux-mainline_git.bb b/recipes-kernel/linux/linux-mainline_git.bb
index 5b8e321..9b2bcbe 100644
--- a/recipes-kernel/linux/linux-mainline_git.bb
+++ b/recipes-kernel/linux/linux-mainline_git.bb
## -27,5 +27,6 ## SRCREV_pn-${PN} = "b04e217704b7f879c6b91222b066983a44a7a09f"
SRC_URI = "git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git;protocol=git;branch=master \
file://defconfig \
+ file://0001-arm-dts-enable-SPI-for-orange-pi-zero.patch \
"
S = "${WORKDIR}/git"
--
2.17.2

Suppress dates in diff -urN

I am using diff -urN to compare two directories and create a patch, something like:
diff -urN ../resources-original/ogc ogc > ../../../src/main/patches/ogc.patch
The patch contains dates in --- and +++ lines:
--- ../resources-original/ogc/wms/1.3.0/capabilities_1_3_0.xsd Sun Jul 22 03:59:38 2012
+++ ogc/wms/1.3.0/capabilities_1_3_0.xsd Sun Jun 12 19:59:29 2016
Is there a possibility to suppress these dates?
Maybe something like
awk '/^(---|\+\+\+)/ { sub(" " $(NF-5) " " $(NF-4) " " $(NF-3) " " $(NF-2) " "$(NF-1) " " $NF, " Thu Jan 1 00:00:00 1970") }1'

Systemtap %M printf format only returns one character

I'm trying to print the data received on a socket - the contents of ubuf on the return of sys_recv. I cant get the %M format specifier to work properly. Can someone please explain how to use it properly. Thanks
stap -L 'kernel.function("sys_recv#net/socket.c")'
kernel.function("sys_recv#net/socket.c:1800") $fd:int $ubuf:void* $size:size_t $flags:unsigned int
using this probe:
[laris#kakitis stap]$ cat socket-recv.stp
#! /usr/bin/env stap
probe kernel.function("sys_recv#net/socket.c").return {
if (pid() == target())
printf ("%s fd %d size %d ubuf %p %10M \n ", ppfunc(),$fd,$size,$ubuf,$ubuf)
}
From my reading of the man page the format %10M should return 10 bytes from the location pointed to by $ubuf:void but I only get 1. Adjusting the parameter 10 shifts the one character output rather than showing more or less memory
[root#kakitis stap]# stap -x 16796 socket-recv.stp
sys_recv fd 13 size 64071 ubuf 0x86ceca0 70
sys_recv fd 13 size 62679 ubuf 0x86cf210 50
Changing 10 to 2 gives this
[root#kakitis stap]# stap -x 16796 socket-recv.stp
sys_recv fd 13 size 64071 ubuf 0x86ceca0 70
sys_recv fd 13 size 62679 ubuf 0x86cf210 50
System particulars are:
[laris#kakitis stap]$ stap --version
Systemtap translator/driver (version 2.1/0.154, rpm 2.1-2.fc17)
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBRPM LIBSQLITE3 NSS TR1_UNORDERED_MAP NLS
[laris#kakitis stap]$ uname -a
Linux kakitis 3.4.33 #1 SMP Tue Jan 7 14:15:58 EST 2014 i686 i686 i386 GNU/Linux
[laris#kakitis stap]$ cat /etc/redhat-release
Fedora release 17 (Beefy Miracle)
Don't confuse the output-width and precision parameters for printf(). The following will do what you meant:
printf ("%33.10M", $pointer)
to print 10 bytes (20 hex characters) in a 33-character-wide output field. One or both numbers can be replaced by *, so that the respective widths are passed as parameters before the $pointer. The upstream man page has been updated with an example.