Today I found out, that just showing a diff in btrfs is extremely complicated.
While in ZFS, it's only zfs diff, in btrfs, one has to use either btrfs subv find-new <SNAPNAME> <last-gen> and this find-new does never show me files I created with touch... or empty directories I created...
or one has to use btrfs send --no-data <SNAP1> <SNAP2> | btrfs recv dump
No what I did to test this: I created an empty directory, called 'blah2' and this is what the "diff" shows me:
[andreas#archlinux data]$ ./bdiff .snaps/Fotos-20220910c .snaps/Fotos-20220910b
At subvol .snaps/Fotos-20220910b
snapshot ./Fotos-20220910b uuid=5b07b5f0-ba94-5a47-b586-6d36805b0c9e transid=6783 parent_uuid=1f25f808-28a4-f34d-98e0-fcb54acf4a8a parent_transid=6788
utimes ./Fotos-20220910b/ atime=2022-09-10T16:10:50+0200 mtime=2022-09-10T16:08:07+0200 ctime=2022-09-10T16:08:07+0200
rmdir ./Fotos-20220910b/blah2
utimes ./Fotos-20220910b/ atime=2022-09-10T16:10:50+0200 mtime=2022-09-10T16:08:07+0200 ctime=2022-09-10T16:08:07+0200
[andreas#archlinux data]$ ./bdiff .snaps/Fotos-20220910b .snaps/Fotos-20220910c
At subvol .snaps/Fotos-20220910c
snapshot ./Fotos-20220910c uuid=1f25f808-28a4-f34d-98e0-fcb54acf4a8a transid=6788 parent_uuid=5b07b5f0-ba94-5a47-b586-6d36805b0c9e parent_transid=6783
utimes ./Fotos-20220910c/ atime=2022-09-10T16:10:50+0200 mtime=2022-09-10T16:16:55+0200 ctime=2022-09-10T16:16:55+0200
mkdir ./Fotos-20220910c/o4139-6788-0
rename ./Fotos-20220910c/o4139-6788-0 dest=./Fotos-20220910c/blah2
utimes ./Fotos-20220910c/ atime=2022-09-10T16:10:50+0200 mtime=2022-09-10T16:16:55+0200 ctime=2022-09-10T16:16:55+0200
chown ./Fotos-20220910c/blah2 gid=1000 uid=1000
chmod ./Fotos-20220910c/blah2 mode=755
utimes ./Fotos-20220910c/blah2 atime=2022-09-10T16:16:55+0200 mtime=2022-09-10T16:16:55+0200 ctime=2022-09-10T16:16:55+0200
[andreas#archlinux data]$
Why the hell does it not only report a mkdir blah2, but a mkdir o4139-6788-0 with a following rename? But the otherway, it just reports one rmdir?
Related
I tried to build the Android SDK Addon System Image of halogenOS 13 (AOSP 13, nothing was changed in the AOSP source code yet).
The steps to build are as usual:
source build/envsetup.sh
lunch aosp_sdk_phone_x86_64-eng
m sdk_addon
At 100%, the build fails with following error:
[100% 12483/12483] Packaging SDK Addon System-Image: out/host/linux-x86/sdk_addon/custom-eng.simao--
FAILED: out/host/linux-x86/sdk_addon/custom-eng.simao--img.zip
/bin/bash -c "(cp -R out/target/product/emulator_x86_64/data out/host/linux-x86/obj/SDK_ADDON/custom_intermediates/custom-eng.simao--img/images/x86_64/data ) && (out/host/linux-x86/bin/soong_zip -o out/host/linux-x86/sdk_addon/custom-eng.simao--img.zip -C out/host/linux-x86/obj/SDK_ADDON/custom_intermediates/custom-eng.simao--img/images/ -D out/host/linux-x86/obj/SDK_ADDON/custom_intermediates/custom-eng.simao--img/images/x86_64 )"
cp: bad 'out/target/product/emulator_x86_64/data': No such file or directory
13:17:13 ninja failed with: exit status 1
If I just mkdir out/target/product/emulator_x86_64/data, of course, that just solves the build error but the SDK addon does not actually boot in the emulator due to encryption issues with the userdata partition so I think this is related. This makes me guess that in the data directory there should be some files in there but for some reason the are not created.
EDIT:
What's really odd here is that the file device/generic/goldfish/vendor.mk explicitly adds some data files to PRODUCT_COPY_FILES, notably:
PRODUCT_COPY_FILES += \
device/generic/goldfish/data/etc/dtb.img:dtb.img \
device/generic/goldfish/emulator-info.txt:data/misc/emulator/version.txt \
device/generic/goldfish/data/etc/apns-conf.xml:data/misc/apns/apns-conf.xml \
device/generic/goldfish/radio/RadioConfig/radioconfig.xml:data/misc/emulator/config/radioconfig.xml \
device/generic/goldfish/data/etc/iccprofile_for_sim0.xml:data/misc/modem_simulator/iccprofile_for_sim0.xml \
If I manually build them using, for example, m out/target/product/emulator_x86_64/data/misc/emulator/version.txt, the file is created at the correct location, as expected. Which leads me to wonder when entries PRODUCT_COPY_FILES are considered targets to be built and when they aren't.
EDIT2:
I got the emulator to boot but the data directory is still not being created. (Creating manually or building one target in the data dir is a workaorund).
I am saving a table which is enumerated and to be splayed on my hdb. After which we load the hdb directory
There was a corruption which caused
'2022.01.01T00:00:01.000 part
(.Q.L) error"
https://code.kx.com/q/ref/dotq/#ql-load/
This is due to load, and I tried corrupting partition to replicate:
Corrupted .d file
Corrupted splayed column(s)
Removed entire partition
Most of the above cases are handled in our exception.
".\2022.01.1\tbl. OS reports: No such file or directory"
but I couldn't replicate the use case where .Q.l happened nor find why it happened from my logs.
Can someone please suggest what kind of corruption could have caused the part error during load.
One possible cause would be a partition folder without correct read permissions:
$ mkdir -p badHDB/2001.01.01/tab1
$ mkdir -p badHDB/20011.01.01/tab2
$ mkdir -p badHDB/2002.01.01
$ chmod 000 badHDB/2002.01.01
Running it we see the error:
q badHDB
'part
[2] (.Q.L)
[0] \l badHDB
You could write a small function to try to narrow down the issues:
// https://code.kx.com/q/ref/system/#capture-stderr-output
q)tmp:first system"mktemp"
q)tab:flip `part`date`osError`files`error!flip {d:1_string x;{y:string y;(y;"D"$y),{r:system x;$[0~"J"$last r;(0b;-1_r;"");(1b;();first r)]} "ls ",x,"/",y," > ",tmp," 2>&1;echo $? >> ",tmp,";cat ",tmp}[d] each key x} `:badHDB
This would result in:
part date osError files error
-----------------------------------------------------------------------------------------------------------
"2001.01.01" 2001.01.01 0 ,"tab1" ""
"20011.01.01" 0 ,"tab2" ""
"2002.01.01" 2002.01.01 1 () "ls: cannot open directory 'badHDB/2002.01.01': Permission denied"
For a larger HDB filter down to partitions with issues:
select from tab where or[null date;osError]
I am trying to create a TPM-based unlock script using tpm2-tools with instructions from Tevora Secure boot tpm2. I have set up the key, loaded it with cryptsetup luksAddKey secret.bin, then tested it using tpm2_unlock -c 0x81000000 --auth pci:sha1:0,2,3,7 and returns the value of secret.bin. For extra measures, to make sure it works, I loaded secret.bin into "/etc/crypttab", ran # update-initramfs -u -k all, and rebooted. Upon reboot, the system unlocked.
I copied over the following code into "/etc/initramfs-tools/hooks/tpm2"
#!/bin/sh -e
if [ "$1" = "prereqs" ]; then exit 0; fi
. /usr/share/initramfs-tools/hook-functions
copy_exec /usr/local/bin/tpm2_unseal
copy_exec /usr/local/lib/libtss2-tcti-device.so
I appended my etc/crypttab from cryptname UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks to cryptname UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks,keyscript=/usr/local/bin/passphrase-from-tpm
I rewrote the following script because the tpm2-tools command was outdated, edited in the new command, and stored it in /usr/local/bin/passphrase-from-tpm:
#!/bin/sh
set -e
echo "Unlocking via TPM" >&2
export TPM2TOOLS_TCTI="device:/dev/tpm0"
/usr/local/bin/tpm2_unseal -c 0x81000000 --auth pcr:sha1:0,2,3,7
if [ $? -eq 0 ]; then
exit
fi
/lib/cryptsetup/askpass "Unlocking the disk fallback $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)\nEnter passphrase: "
I ran # update-initramfs -u -k all then rebooted. In reboot, I get the following error: /lib/cryptsetup/scripts/passphrase-from-tpm: line 5: /usr/local/bin/tpm2_unseal: not found
I have tried many times to edit passphrase-from-tpm unsuccessfully, including:
Moving both passphrase-from-tpm into "/boot/efi/EFI/BOOT/" and referencing crypttab to that file
Modifying passphrase-from-tpm to use a relative file path to tpm_unseal
Before I figured out how to create a backup linux boot using:
objcopy \
--add-section .osrel=/etc/os-release --change-section-vma .osrel=0x20000 \
--add-section .cmdline=cmdline.txt --change-section-vma .cmdline=0x30000 \
--add-section .linux="/boot/vmlinuz" --change-section-vma .linux=0x40000 \
--add-section .initrd="/boot/initrd.img" --change-section-vma .initrd=0x3000000 \
/usr/lib/systemd/boot/efi/linuxx64.efi.stub /boot/EFI/BOOT/BOOT_RECX64.EFI
I would be locked out of the system completely because of the error and had to reinstall Ubuntu about 40 times. I have suffered a lot and want to quit but I am too stubborn to throw in the flag.
just copy tpm2_unseal to /usr/local/bin/
I'm trying to make a working setup following basically those instructions, plus a few others I have found. While not working 100% yet, check that both /etc/initramfs-tools/hooks/tpm2 and /usr/local/bin/passphrase-from-tpm are marked executable (sudo chmod ug+x $filename).
After making the initramfs, you can run the following to ensure that the TPM related files are actually in the image. Replace the path in the filename by whatever update-initramfs said it was generating:
$ lsinitramfs /boot/initrd.img-5.0.0-37-generic | egrep "(tpm|libtss)"
lib/cryptsetup/scripts/passphrase-from-tpm
lib/modules/5.0.0-37-generic/kernel/crypto/asymmetric_keys/tpm_key_parser.ko
lib/modules/5.0.0-37-generic/kernel/crypto/asymmetric_keys/asym_tpm.ko
lib/udev/rules.d/tpm-udev.rules
usr/local/lib/libtss2-sys.so.0
usr/local/lib/libtss2-mu.so.0
usr/local/lib/libtss2-sys.so.0.0.0
usr/local/lib/libtss2-tcti-device.so
usr/local/lib/libtss2-tcti-device.so.0
usr/local/lib/libtss2-tcti-device.so.0.0.0
usr/local/lib/libtss2-mu.so.0.0.0
usr/local/bin/tpm2_unseal
Additionally, I have modified /usr/local/bin/passphrase-from-tpm to the following:
#!/bin/sh
TPM_DEVICE=/dev/tpm0
TPM_REGISTER=0x81000001
TPM_SEAL_POLICY=sha256:0,2,4,7
export TPM2TOOLS_TCTI="device:$TPM_DEVICE"
if [ "$CRYPTTAB_TRIED" -eq 0 ]; then
echo "Unlocking via TPM" >&2
/usr/local/bin/tpm2_unseal -H $TPM_REGISTER -L $TPM_SEAL_POLICY
UNSEAL_STATUS=$?
echo "Unseal status $UNSEAL_STATUS" >&2
if [ $UNSEAL_STATUS -eq 0 ]; then
exit
fi
else
echo "TPM unlocking previously failed for $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)" >&2
/lib/cryptsetup/askpass "Enter passphrase for $CRYPTTAB_SOURCE ($CRYPTTAB_NAME): "
fi
Note that the command line options to tpm2_unseal are for the 3.x versions of tpm2-tools. If you're using another version, you might need to update the options.
I pulled out various bits into variables at the top of the file. Modify TPM_REGISTER and TPM_SEAL_POLICY to match how you created the TPM object. set -e was removed since if any command failed, the whole script would exit, preventing the askpass fallback from ever running if tpm2_unseal failed.
Additionally, I noticed that if the script fails for some reason, systemd will attempt to run it again. If the secret in the TPM doesn't match the LUKS key, this will render the system unbootable, since the unseal succeeds, but unlocking fails, and systemd will run the script again.
Looking at the man page for crypttab, I discovered that one of the environment variables provided to the keyscript is CRYPTTAB_TRIED which is the number of tries it has attempted to unlock the volume. If CRYPTTAB_TRIED is 0, it'll attempt to use the TPM, as shown by this test (Running as non-root, so accessing the TPM device fails):
$ export CRYPTTAB_SOURCE=some_device
$ export CRYPTTAB_NAME=some_device_name
$ export CRYPTTAB_TRIED=0
$ ./passphrase-from-tpm
Unlocking via TPM
ERROR:tcti:src/tss2-tcti/tcti-device.c:440:Tss2_Tcti_Device_Init() Failed to open device file /dev/tpm0: Permission denied
ERROR: tcti init allocation routine failed for library: "device" options: "/dev/tpm0"
ERROR: Could not load tcti, got: "device"
Unseal status 1
When it tries running the script again, CRYPTTAB_TRIED will be greater than 0, making it display the password prompt instead:
$ export CRYPTTAB_TRIED=1
$ ./passphrase-from-tpm
TPM unlocking previously failed for some_device (some_device_name)
Enter passphrase for some_device (some_device_name):
Hopefully this is still of use to you, and helpful to anyone else trying to get the house of cards that is disk encryption with a TPM on Linux working.
I am using osquery to monitor files and folders to get events on any operation on those files. There is a specific syntax for osquery configuration:
"/etc/": watches the entire directory at a depth of 1.
"/etc/%": watches the entire directory at a depth of 1.
"/etc/%%": watches the entire tree recursively with /etc/ as the root.
I am trying to evaluate the memory usage in case of watching a lot of directories. In this process I found the following statistics:
"/etc", "/etc/%", "/etc/%.conf": only 1 inotify handle is found registered in the name of osquery.
"/etc/%%: a few more than 289 inotify handles found which are registered in the name of osquery, given that there are a total of 285 directories under the tree. When checking the entries in /proc/$PID/fdinfo, all the inodes listed in the file points to just folders.
eg: for "/etc/%.conf"
$ grep -r "^inotify" /proc/$PID/fdinfo/
18:inotify wd:1 ino:120001 sdev:800001 mask:3ce ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:01001200bc0f1cab
$ printf "%d\n" 0x120001
1179649
$ sudo debugfs -R "ncheck 1179649" /dev/sda1
debugfs 1.43.4 (31-Jan-2017)
Inode Pathname
1179649 //etc
The inotify watch is established on the whole directory here, but the events are only reported for the matching files /etc/*.conf. Is osquery filtering the events based on the file_paths supplied, which is what I am assuming, but not sure.
Another experiment that I performed to support the above claim was, use the source in the inotify(7) and run a watcher on a particular file. When I check the list of inotify watchers, it just shows :
$ ./a.out /tmp/inotify.cc &
$ cat /proc/$PID/fdinfo/3
...
inotify wd:1 ino:1a1 sdev:800001 mask:38 ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:a1010000aae325d7
$ sudo debugfs -R "ncheck 417" /dev/sda1
debugfs 1.43.4 (31-Jan-2017)
Inode Pathname
417 /tmp/inotify.cc
So, according to this experiment, establishing a watcher on a single file is possible (which is clear from the inotify man page). This supports the claim that osquery is doing some sort of filtering based on the file patterns supplied.
Could someone verify the claim or present otherwise?
My osquery config:
{
"options": {
"host_identifier": "hostname",
"schedule_splay_percent": 10
},
"schedule": {
"file_events": {
"query": "SELECT * FROM file_events;",
"interval": 5
}
},
"file_paths": {
"sys": ["/etc/%.conf"]
}
}
$ osqueryd --version
osqueryd version 3.3.2
$ uname -a
Linux lab 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
It sounds like some great sleuthing!
I think the comments in the source code support that. It's worth skimming it. I think the relevant files:
https://github.com/osquery/osquery/blob/master/osquery/tables/events/linux/file_events.cpp
https://github.com/osquery/osquery/blob/master/osquery/events/linux/inotify.cpp
I want to run GNU Screen from within an Emacs term-mode buffer. When I try to do this I get an error:
$ screen
Cannot find terminfo entry for 'eterm-color'.
This error suggests that I should install the eterm-color terminfo entry. I attempted to do this in 2 ways which both failed.
The first attempt was to use the TERMINFO environment variable.
My .bashrc:
export TERMINFO=~/.terminfo
Then I did:
$ mkdir -p ~/.terminfo/e/
$ cp /usr/local/Cellar/emacs/23.3a/share/emacs/23.3/etc/e/* ~/.terminfo/e/
In the term-mode buffer the eterm-color entry is not found. It says that it gets eterm-color from TERMCAP instead of from the actual file.
$ infocmp
# Reconstructed via infocmp from file: TERMCAP
eterm-color,
am, mir, xenl,
colors#8, cols#138, it#8, lines#41, pairs#64,
bel=^G, bold=\E[1m, clear=\E[H\E[J, cr=^M,
csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
cud=\E[%p1%dB, cud1=^J, cuf=\E[%p1%dC, cuf1=\E[C,
cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,
dch=\E[%p1%dP, dch1=\E[P, dl=\E[%p1%dM, dl1=\E[M, ed=\E[J,
el=\E[K, el1=\E[1K, home=\E[H, ht=^I, ich=\E[%p1%d#,
il=\E[%p1%dL, il1=\E[L, ind=^J, invis=\E[8m, kbs=\177,
kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA,
kdch1=\E[3~, kend=\E[4~, khome=\E[1~, knp=\E[6~, kpp=\E[5~,
nel=^M^J, op=\E[39;49m, rc=\E8, rev=\E[7m, rmir=\E[4l,
rmso=\E[27m, rmul=\E24m, rs1=\Ec, sc=\E7, setab=\E[4%p1%dm,
setaf=\E[3%p1%dm, sgr0=\E[m, smir=\E[4h, smso=\E[7m,
smul=\E[4m,
The second attempt was to use symlinks.
$ sudo mkdir /usr/share/terminfo/e/
$ sudo cp /usr/local/Cellar/emacs/23.3a/share/emacs/23.3/etc/e/* /usr/share/terminfo/e/
This failed in the same way as the first attempt.
Why did neither attempt work? How do I get the eterm-color terminfo entry installed?
I am running OS X 10.6.8 and Emacs 23.3.1.
The following worked for me on FreeBSD.
Put the termcap entry pasted below in /usr/share/misc/termcap and run # cd /usr/share/misc/ && cap_mkdb termcap.
# Termcap entry for eterm-color (taken from $TERMCAP set by emacs and formatted)
eterm-color|Emacs term.el terminal emulator term-protocol-version 0.96:\
:am:bs:mi:pt:xn:\
:Co#8:co#166:li#48:pa#64:\
:#7=\E[4~:AB=\E[4%dm:AF=\E[3%dm:AL=\E[%dL:DC=\E[%dP:\
:DL=\E[%dM:DO=\E[%dB:IC=\E[%d#:LE=\E[%dD:RI=\E[%dC:\
:UP=\E[%dA:al=\E[L:bl=^G:cb=\E[1K:cd=\E[J:ce=\E[K:\
:cl=\E[H\E[J:cm=\E[%i%d;%dH:cr=^M:cs=\E[%i%d;%dr:dc=\E[P:\
:dl=\E[M:do=^J:ei=\E[4l:ho=\E[H:im=\E[4h:kD=^[[3~:\
:kN=\E[6~:kP=\E[5~:kb=^?:kd=\EOB:kh=\E[1~:kl=\EOD:\
:kr=\EOC:ku=\EOA:le=^H:md=\E[1m:me=\E[m:mk=\E[8m:mr=\E[7m:\
:nd=\E[C:op=\E[39;49m:r1=\Ec:rc=\E8:sc=\E7:se=\E[27m:sf=^J:\
:so=\E[7m:ta=^I:ue=\E[m:\
:up=\E[A:us=\E[4m: