Where are files marked as assume-unchanged in Git? [duplicate] - eclipse

I like to modify config files directly (like .gitignore and .git/config) instead of remembering arbitrary commands, but I don't know where Git stores the file references that get passed to "git update-index --assume-unchanged file".
If you know, please do tell!

It says where in the command - git update-index
So you can't really be editing the index as it is not a text file.
Also, to give more detail on what is stored with the git update-index --assume-unchanged command, see the Using “assume unchanged” bit section in the manual

As others said, it's stored in the index, which is located at .git/index.
After some detective work, I found that it is located at the: assume valid bit of each index entry.
Therefore, before understanding what follows, you should first understand the global format of the index, as explained in my other answer.
Next, I will explain how I verified that the "assume valid" bit is the culprit:
empirically
by reading the source
Empirical
Time to hd it up.
Setup:
git init
echo a > b
git add b
Then:
hd .git/index
Gives:
00000000 44 49 52 43 00 00 00 02 00 00 00 01 54 e9 b6 f3 |DIRC........T...|
00000010 2d 4f e1 2f 54 e9 b6 f3 2d 4f e1 2f 00 00 08 05 |-O./T...-O./....|
00000020 00 de 32 ff 00 00 81 a4 00 00 03 e8 00 00 03 e8 |..2.............|
00000030 00 00 00 00 e6 9d e2 9b b2 d1 d6 43 4b 8b 29 ae |...........CK.).|
00000040 77 5a d8 c2 e4 8c 53 91 00 01 62 00 c9 a2 4b c1 |wZ....S...b...K.|
00000050 23 00 1e 32 53 3c 51 5d d5 cb 1a b4 43 18 ad 8c |#..2S<Q]....C...|
00000060
Now:
git update-index --assume-unchanged b
hd .git/index
Gives:
00000000 44 49 52 43 00 00 00 02 00 00 00 01 54 e9 b6 f3 |DIRC........T...|
00000010 2d 4f e1 2f 54 e9 b6 f3 2d 4f e1 2f 00 00 08 05 |-O./T...-O./....|
00000020 00 de 32 ff 00 00 81 a4 00 00 03 e8 00 00 03 e8 |..2.............|
00000030 00 00 00 00 e6 9d e2 9b b2 d1 d6 43 4b 8b 29 ae |...........CK.).|
00000040 77 5a d8 c2 e4 8c 53 91 80 01 62 00 17 08 a8 58 |wZ....S...b....X|
00000050 f7 c5 b3 e1 7d 47 ac a2 88 d9 66 c7 5c 2f 74 d7 |....}G....f.\/t.|
00000060
By comparing the two indexes, and looking at the global structure of the index, see that the only differences are:
byte number 0x48 (9th on line 40) changed from 00 to 80. That is our flag, the first bit of the cache entry flags.
the 20 bytes from 0x4C to 0x5F. This is expected since that is a SHA-1 over the entire index.
This has also though me that the SHA-1 of the index entry in bytes from 0x34 to 0x47 does not take into account the flags, since it did not changed between both indexes. This is probably why the flags are placed after the SHA, which only considers what comes before it.
Source code
Now let's see if that is coherent with source code of Git 2.3.
First look at the source of update-index, grep assume-unchanged.
This leads to the following line:
{OPTION_SET_INT, 0, "assume-unchanged", &mark_valid_only, NULL,
N_("mark files as \"not changing\""),
PARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, MARK_FLAG},
{OPTION_SET_INT, 0, "no-assume-unchanged", &mark_valid_only, NULL,
N_("clear assumed-unchanged bit"),
PARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, UNMARK_FLAG},
so the value is stored at mark_valid_only. Grep it, and find that it is only used at one place:
if (mark_valid_only) {
if (mark_ce_flags(path, CE_VALID, mark_valid_only == MARK_FLAG))
die("Unable to mark file %s", path);
return;
}
CE means Cache Entry.
By quickly inspecting mark_ce_flags, we see that:
if (mark)
active_cache[pos]->ce_flags |= flag;
else
active_cache[pos]->ce_flags &= ~flag;
So the function basically sets or unsets the CE_VALID bit, depending on mark_valid_only, which is a tri-state:
mark: --assume-unchanged
unmark: --no-assume-unchanged
do nothing: the default value 0 of the option set at {OPTION_SET_INT, 0
Next, by grepping under builtin/, we see that no other place sets the value of CE_VALID, so --assume-unchanged must be the only command that sets it.
The flag is however used in many places of the source code, which should be expected as it has many side-effects, and it is used every time like:
ce->ce_flags & CE_VALID
so we conclude that it is part of the ce_flags field of struct cache_entry.
The index is specified at cache.h because one of its functions is to be a cache for creating commits faster.
By looking at the definition of CE_VALID under cache.h and surrounding lines we have:
#define CE_STAGEMASK (0x3000)
#define CE_EXTENDED (0x4000)
#define CE_VALID (0x8000)
#define CE_STAGESHIFT 12
So we conclude that it is the very first bit of that integer (0x8000), just next to the CE_EXTENDED, which is coherent with my earlier experiment.

Related

How to retrieve details of the console port used by BIOS using efivars?

As part of installation of linux, I would like to set the "console device properties"(example, console=ttyS0,115200n1) via the kernel cmdline for Intel based platform.
There is No VGA console, only serial consoles via COM interface.
On these systems BIOS already has the required settings to interact using the appropriate serial port.
I see that EFI has variables ConIn, ConOut, ConErr which I am able to see from /sys/firmware/efi but unable to decode the contents of it.
Is it possible to identify which COM port is being used by the BIOS by examining the efi variables.
Example, of the EFI var on my box.
root#linux:~# efivar -p -n 8be4df61-93ca-11d2-aa0d-00e098032b8c-ConOut
GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
Name: "ConOut"
Attributes:
Non-Volatile
Boot Service Access
Runtime Service Access
Value:
00000000 02 01 0c 00 d0 41 03 0a 00 00 00 00 01 01 06 00 |.....A..........|
00000010 00 1a 03 0e 13 00 00 00 00 00 00 c2 01 00 00 00 |................|
00000020 00 00 08 01 01 03 0a 18 00 9d 9a 49 37 2f 54 89 |...........I7/T.|
00000030 4c a0 26 35 da 14 20 94 e4 01 00 00 00 03 0a 14 |L.&5.. .........|
00000040 00 53 47 c1 e0 be f9 d2 11 9a 0c 00 90 27 3f c1 |.SG..........'?.|
00000050 4d 7f 01 04 00 02 01 0c 00 d0 41 03 0a 00 00 00 |M.........A.....|
00000060 00 01 01 06 00 00 1f 02 01 0c 00 d0 41 01 05 00 |............A...|
00000070 00 00 00 03 0e 13 00 00 00 00 00 00 c2 01 00 00 |................|
00000080 00 00 00 08 01 01 03 0a 18 00 9d 9a 49 37 2f 54 |............I7/T|
00000090 89 4c a0 26 35 da 14 20 94 e4 01 00 00 00 03 0a |.L.&5.. ........|
000000a0 14 00 53 47 c1 e0 be f9 d2 11 9a 0c 00 90 27 3f |..SG..........'?|
000000b0 c1 4d 7f ff 04 00 |.M.... |
root#linux:~#
The contents of the ConOut variable are described in the UEFI specification - current version (2.8B):
3.3 - globally defined variables:
| Name | Attribute | Description |
|---------|------------|------------------------------------------------|
| ConOut | NV, BS, RT | The device path of the default output console. |
For information about device paths, we have:
10 - Protocols — Device Path Protocol:
Apart from the initial description of device paths, table 44 shows you the Generic Device Path Node structure, from which we can start decoding the contents of the variable.
The type of the first node is 0x02, telling us this node describes an ACPI device path, of 0x000c bytes length. Now jump down to 10.3.3 - ACPI Device Path and table 52, which tells us 1) that this is the right table (subtype 0x01) and 2) that the default ConOut has a _HID of 0x0a03410d and a _UID of 0.
The next node has a type of 0x01 - a Hardware Device Path, described further in 10.3.2, in this case table 46 (SubType is 0x01) for a PCI device path.
The next node describes a Messaging Device Path of type UART and so on...
Still, this only tells you what UEFI considers to be its default console, SPCR is what an operating system is supposed to be looking at for serial consoles. Unfortunately, on X86 the linux kernel handily ignores SPCR apart from for earlycon. I guess this is what you're trying to work around. It might be good to start some discussion on kernel development lists about whether to fix that and have X86 work like ARM64.
In my case since I know that console port is a "Serial IOPORT",
I could get the details now as follows.
a. Get hold of the /sys/firmware/acpi/tables/SPC table.
b. Read the Address offset 44-52. Actually one the last two bytes suffice.
Reference:
a. https://learn.microsoft.com/en-us/windows-hardware/drivers/serports/serial-port-console-redirection-table states that
Base Address 12 40
The base address of the Serial Port register set described using the ACPI Generic Address Structure.
0 = console redirection disabled
Note:
COM1 (0x3F8) would be:
Integer Form: 0x 01 08 00 00 00000000000003F8
Viewed in Memory: 0x01080000F803000000000000
COM2 (Ox2F8) would be:
Integer Form: 0x 01 08 00 00 00000000000002F8
Viewed in Memory: 0x01080000F802000000000000

Midi track meta event values which do not conform to the specification

Hi I have scraped a large number of midi files off the internet.
I am using them for training material to train a generative adversarial network. I find that many midi files conform to the midi standard but then I run into issues with midi meta events with values of FF11 and FF10 . I have looked up the midi specification from several sources and have never found midi meta events defined in this way. Here is the hex of a midi track event with some of the offending values:
4D 54 72 6B 00 00 1A 8D 00 FF 03 0D 47 75 69 74
61 72 20 44 41 44 47 41 44 00 FF 10 08 00 00 3E
39 37 32 2D 26 00 C0 19 00 C1 19 00 B0 65 00 00
B0 64 00 00 B0 06 02 00 B0 65 7F 00 B0 64 7F 00
E0 00 40 00 B1 65 00 00 B1 64 00 00 B1 06 02 00
B1 65 7F 00 B1 64 7F 00 E1 00 40 00 B0 0A 3F 00
B1 0A 3F 00 B0 5D 10 00 B0 5B 1E 00 B1 5D 10 00
B1 5B 1E 81 69 FF 11 01 00 00 90 3E 51 08 FF 11
I cant seem to find any information whatsoever on these values even though these midi files play over timidity and other midi player software perfectly. Can anyone point me to some information about them and what they mean? any help would be greatly , greatly appreciated. :-) resolving this issue would be a service to the miriad of people who are trying to use the python-midi library to train tensorflow models and this I am sure is only a fraction of the people who would be effected.
The SMF specification says:
As with chunks, future meta-events may be designed which may not be known to existing programs, so programs must properly ignore meta-events which they do not recognize, and indeed, should expect to see them.
I am not aware of any published extension that defines values 10h or 11h; it's likely that some sequencer uses these for its own purposes, and violated the specification by not using type 7F for that.

Inspecting binary over sockets

I'm using WireShark to inspect data sent/received over a web-socket, however, all I see is nonsense.
0000 1c 74 0d 7d 42 24 d8 5d e2 26 c1 7d 08 00 45 00 .t.}B$.].&.}..E.
0010 00 3c 75 4e 40 00 80 06 22 eb c0 a8 01 c0 4f 89 .<uN#...".....O.
0020 50 91 c4 f1 0f 78 72 e5 d0 f4 ea 5e 6e e2 50 18 P....xr....^n.P.
0030 00 40 91 b3 00 00 c2 8e 6d 06 87 95 7f 76 78 62 .#......m....vxb
0040 92 f9 54 2a 92 f9 b4 95 6c 06 ..T*....l.
I've seen this type of output before. The left is a line of binary, and the right is the decoded string (ASCII), right?
Is this data obfuscated/encrypted?
Is it possible to get cogent information from my socket?
Also, what do the [FIN] and [MASKED] flags mean?
If you copy and paste the data you supplied into a text file and append a line beginning with 0050 with nothing following it, you can then run text2pcap -a infile.txt outfile.pcap to convert the data to a pcap file that Wireshark can read and decode for you.
See the text2pcap man page for more information about this tool.
I have done this and the packet appears to just be a simple TCP segment. There is no [FIN] or [MASKED] flag, only PSH and ACK. For information about these TCP flags, refer to RFC 793, section 3.1, as well as the other RFC's mentioned at the top, which update this one.

Mifare Desfire Wrapped Mode: How to calculate CMAC?

When using Desfire native wrapped APDUs to communicate with the card, which parts of the command and response must be used to calculate CMAC?
After successful authentication, I have the following session key:
Session Key: 7CCEBF73356F21C9191E87472F9D0EA2
Then when I send a GetKeyVersion command, card returns the following CMAC which I'm trying to verify:
<< 90 64 00 00 01 00 00
>> 00 3376289145DA8C27 9100
I have implemented CMAC algorithm according to "NIST special publication 800-38B" and made sure it is correct. But I don't know which parts of command and response APDUs must be used to calculate CMAC.
I am using TDES, so MAC is 8 bytes.
I have been looking at the exact same issue for the last few days and I think I can at least give you some pointers. Getting everything 'just so' has taken some time and the documentation from NXP (assuming you have access) is a little difficult to interpret in some cases.
So, as you probably know, you need to calculate the CMAC (and update your init vec) on transmit as well as receive. You need to save the CMAC each time you calculate it as the init vec for the next crypto operation (whether CMAC or encryption etc).
When calculating the CMAC for your example the data to feed into your CMAC algorithm is the INS byte (0x64) and the command data (0x00). Of course this will be padded etc as specified by CMAC. Note, however, that you do not calculate the CMAC across the entire APDU wrapping (i.e. 90 64 00 00 01 00 00) just the INS byte and data payload is used.
On receive you need to take the data (0x00) and the second status byte (also 0x00) and calculate the CMAC over that. It's not important in this example but order is important here. You use the response body (excluding the CMAC) then SW2.
Note that only half of the CMAC is actually sent - CMAC should yield 16 bytes and the card is sending the first 8 bytes.
There were a few other things that held me up including:
I was calculating the session key incorrectly - it is worth double checking this if things are not coming out as you'd expect
I interpreted the documentation to say that the entire APDU structure is used to calculate the CMAC (hard to read them any other way tbh)
I am still working on calculating the response from a Write Data command correctly. The command succeeds but I can't validate the CMAC. I do know that Write Data is not padded with CMAC padding but just zeros - not yet sure what else I've missed.
Finally, here is a real example from communicating with a card from my logs:
Authentication is complete (AES) and the session key is determined to be F92E48F9A6C34722A90EA29CFA0C3D12; init vec is zeros
I'm going to send the Get Key Version command (as in your example) so I calculate CMAC over 6400 and get 1200551CA7E2F49514A1324B7E3428F1 (which is now my init vec for the next calculation)
Send 90640000010000 to the card and receive 00C929939C467434A8 (status is 9100).
Calculate CMAC over 00 00 and get C929939C467434A8A29AB2C40B977B83 (and update init vec for next calculation)
The first half of our CMAC from step #4 matches the 8 byte received from the card in step #3
Sry for my English,- its terrible :) but it's not my native language. I'm Russian.
Check first MSB (7 - bit) of array[0] and then shiffting this to the left. And then XOR if MSB 7 bit was == 1;
Or save first MSB bit of array[0] and after shiffting put this bit at the end of array[15] at the end (LSB bit).
Just proof it's here:
https://www.nxp.com/docs/en/application-note/AN10922.pdf
Try this way:
Zeros <- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SessionKey <- 00 01 02 03 E3 27 64 0C 0C 0D 0E 0F 5C 5D B9 D5
Data <- 6F 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00
First u have to encrypt 16 bytes (zeros) with SesionKey;
enc_aes_128_ecb(Zeros);
And u get EncryptedData.
EncryptedData <- 3D 08 A2 49 D9 71 58 EA 75 73 18 F2 FA 6A 27 AC
Check bit 7 [MSB - LSB] of EncryptedData[0] == 1? switch i to true;
bool i = false;
if (EncryptedData[0] & 0x80){
i = true;
}
Then do Shiffting of all EncryptedData to 1 bit <<.
ShiftLeft(EncryptedData,16);
And now, when i == true - XOR the last byte [15] with 0x87
if (i){
ShiftedEncryptedData[15] ^= 0x87;
}
7A 11 44 93 B2 E2 B1 D4 EA E6 31 E5 F4 D4 4F 58
Save it as KEY_1.
Try bit 7 [MSB - LSB] of ShiftedEncryptedData[0] == 1?
i = false;
if (ShiftedEncryptedData[0] & 0x80){
i = true;
}
Then do Shiffting of all ShiftedEncryptedData to 1 bit <<.
ShiftLeft(ShiftedEncryptedData,16);
And now, when i == true - XOR the last byte [15] with 0x87
if (i){
ShiftedEncryptedData[15] ^= 0x87;
}
F4 22 89 27 65 C5 63 A9 D5 CC 63 CB E9 A8 9E B0
Save it as KEY_2.
Now we take our Data (6F 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00)
As Michael say's - pad command with 0x80 0x00...
XOR Data with KEY_2 - if command was padded, or KEY_1 if don't.
If we have more like 16 bytes (32 for example) u have to XOR just last 16 bytes.
Then encrypt it:
enc_aes_128_ecb(Data);
Now u have a CMAC.
CD C0 52 62 6D F6 60 CA 9B C1 09 FF EF 64 1A E3
Zeros <- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SessionKey <- 00 01 02 03 E3 27 64 0C 0C 0D 0E 0F 5C 5D B9 D5
Key_1 <- 7A 11 44 93 B2 E2 B1 D4 EA E6 31 E5 F4 D4 4F 58
Key_2 <- F4 22 89 27 65 C5 63 A9 D5 CC 63 CB E9 A8 9E B0
Data <- 6F 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00
CMAC <- CD C0 52 62 6D F6 60 CA 9B C1 09 FF EF 64 1A E3
C/C++ function:
void ShiftLeft(byte *data, byte dataLen){
for (int n = 0; n < dataLen - 1; n++) {
data[n] = ((data[n] << 1) | ((data[n+1] >> 7)&0x01));
}
data[dataLen - 1] <<= 1;
}
Have a nice day :)

Decoding an arbitrary block of NSData?

If I have an arbitrary block of NSData as a hex value, is there a way to determine what the object might have been before it was archived or serialized? I don't mind a few guess and check methods, but I need some pointers in the right direction.
I have an NSData object with some hex in it. What methods of NSData should I look at? Are there other classes to try as well
Don't want to scare people away from answering, but I have a file of game data which was likely encoded using a Cocoa Touch class. The data, when viewed in a hex editor, shows gibberish and a username, which leads me to suspect that it's an archived or encoded object of sorts. I have copied the hex from the hex editor into a sample project which I am using to try and unarchive the data.
I don't believe this is related to the 3d format, the file extension is arbitrary.
Here's the data. I'm hoping it doesn't get lost in translation:
'µköXN[ÎÀü÷h/F9ó9Vìñ°ceE¸z¶=Hmoshbermú«ó¼Ppù#ÝVÔ=4â®L,K;Êç;ASÀ&Ë÷ëÓ%È;Úf¬G}tmQ;µéüø_87´y©ã©!߶óQòAçÛl©âSG4S½3ýJת9äô¡wxiD²M¼ÏB]39øþ:óñ7ª¾÷躣È3Ï¢ÍEFÍ¢ª»r]BmÁ'Ò+åygÞÅQ?luó>÷ú¼è6¸|}[¼[¶Ñ¦g!\OÎÒJSE..pSß&_ÈEäø)6òëó¨¼2¶ð°æà`ï7Ë=Ã¥:cƧ=L4qG-"µ(ÐÝïß ÓãXkÀ4fzæ·p\ññT<tu¥Æ©;Ìn4£³Ï¢ÌFåG´
And the corresponding hex:
27 B5 6B F6 01 00 00 00 58 4E 5B CE C0 FC F7 68 2F 46 86 87 83 39 F3 39 9E 56 EC F1 B0 63 9E 65 45 B8 7A B6 3D 07 99 48 6D 6F 73 68 62 65 72 6D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90 86 FA 03 0E AB F3 BC 0B 50 70 F9 23 DD 87 56 03 D4 3D 34 90 E2 AE 4C 2C 94 9E 8E 15 4B 0C 83 8C 3B 03 CA E7 3B 1B 41 53 C0 26 04 CB F7 EB D3 25 C8 3B DA 66 8A AC 47 7D 8A 7F 74 6D 51 3B B5 19 E9 FC F8 5F 38 37 B4 11 0C 79 A9 12 E3 A9 21 DF B6 F3 51 F2 41 E7 DB 85 02 9F 6C A9 E2 53 47 1F 34 86 53 BD 33 FD 4A D7 AA 39 C3 A4 F4 A1 77 78 69 44 B2 4D BC CF 42 5D 33 39 F8 FE 97 3A 81 F3 F1 10 37 AA BE 86 91 F7 1F E8 83 BA A3 C8 33 CF 1D A2 CD 45 7F 46 1F CD A2 AA BB 1A 72 5D 42 02 6D C1 0F 27 D2 2B E5 0B 79 67 DE C5 1A 51 3F 14 6C 75 F3 3E F7 FA BC E8 36 8E B8 7C 02 1C 7D 01 00 92 8C 19 5B BC 5B B6 D1 A6 67 7F 21 5C 84 13 4F CE 0C D2 4A 53 19 82 45 1B 2E 2E 96 70 53 DF 26 5F C8 1C 45 8F E4 F8 29 36 F2 EB 9D 95 F3 A8 BC 32 B6 F0 B0 E6 91 98 1A E0 99 60 EF 37 CB 3D C3 A5 3A 63 0C C6 A7 3D 4C 34 71 47 2D 22 B5 28 D0 DD EF DF 09 D3 E3 58 6B C0 17 34 66 7A E6 B7 70 5C F1 F1 54 3C 74 94 75 A5 C6 15 A9 9E 14 3B CC 15 10 83 6E 34 A3 B3 CF 0F A2 9C CC 8E 46 8C E5 00 00 47 B4 17 05 00 00 00 00
If anyone cares to help figure this out it would be much appreciated.
If I have an arbitrary block of NSData as a hex value, is there a way to determine what the object might have been before it was archived or serialized?
Not really. That's about as 'trivial' as reading arbitrary files correctly without the use of a UTI, extension, MIME type. Of course, your program would also need to support reading of all those files/formats.
I don't mind a few guess and check methods, but I need some pointers in the right direction.
You need to narrow your problem/inputs down, if you don't want an impossibly difficult task.
I have an NSData object with some hex in it. What methods of NSData should I look at?
It's just a data blob of length bytes. It could represent anything -- if you don't know where it came from.
Are there other classes to try as well?
Perhaps you would start by saving all your data via NSCoder or another serializer/archiver which offers some introspection and support for you to enter your own information (which would be comparable to a UTI or MIME type).
Edit:
Don't want to scare people away from answering, but I have a file of game data which was likely encoded using a Cocoa Touch class. The data, when viewed in a hex editor, shows gibberish and a username, which leads me to suspect that it's an archived or encoded object of sorts. I have copied the hex from the hex editor into a sample project which I am using to try and unarchive the data.
Using these APIs, the data may be represented multiple ways. You're probably facing something within the domain of 1) a proprietary file format through 2) a keyed archive.
The latter is easier for nontrivial data representations. You would need to define any objc classes you do not have available when unarchived. In that case, a few sample representations would offer a rough outline of the data structures you will need (under conventional implementations). It could also be an archive similar to an NSDictionary, if the unarchiver is capable of opening it. This is a problem which is easier than with other langs, since archiving often falls back on keys and values mapped to members in Cocoa.
Edit2:
The file came from the Draw Something directory. It's called gamedata.i3d
(shrug)
Try using NSKeyedUnarchiver to read it. It's not uncommon to use just the standard Foundation containers like NSArray, NSDictionary, and NSString to store data, so you might get lucky. That obviously won't work if custom classes were involved, but it might be worth 15 minutes of your time to try it.