Why are the hex numbers for big endian different than little endian? - cpu-architecture

#include<stdio.h>
int main()
{
typedef unsigned char *byte_pointer;
void show_bytes(byte_pointer start, size_t len)
{
int i;
for (i = 0; i < len; i++)
{
printf(" %.2x", start[i]);
printf("\n");
}
}
void show_int(int x)
{
show_bytes((byte_pointer) &x, sizeof(int));
}
void show_float(int x)
{
show_bytes((byte_pointer) &x, sizeof(float));
}
void show_pointer(int x)
{
show_bytes((byte_pointer) &x, sizeof(void *));
}
int a = 0x12345678;
byte_pointer ap = (byte_pointer) &a;
show_bytes(ap, 3);
return 0;
}
(Solutions according to the CS:APP book)
Big endian: 12 34 56
Little endian: 78 56 34
I know systems have different conventions for storage allocation but if two systems use the same convention but are different endian why are the hex values different?

Endian-ness is an issue that arises when we use more than one storage location for a value/type, which we do because somethings won't fit in a single storage location.
As soon as we use multiple storage locations for a single value that gives rise to the question of:  What part of the value will we store in each storage location?
The first byte of a two-byte item will have a lower address than the second byte, and in particular, the address of the second byte will be at +1 from the address of the lower byte.
Storing a two-byte item in two bytes of storage, do we store the most significant byte first and the least significant byte second, or vice versa?
We choose to use directly consecutive bytes for the two bytes of the two-byte item, so no matter which (endian) way we choose to store such an item, we refer to the whole two-byte item by the lower address (the address of its first byte).
We can express these storage choices with a formula, here item[0] refer to the first byte while item[1] refers to the second byte.
item[0] = value >> 8 // also value / 256
item[1] = value & 0xFF // also value % 256
value = (item[0]<<8) | item[1] // also item[0]*256 | item[1]
--vs--
item[0] = value & 0xFF // also value % 256
item[1] = value >> 8 // also value / 256
value = item[0] | (item[1]<<8) // also item[0] | item[1]*256
The first set of formulas is for big endian, and the second for little endian.
By these formulas, it doesn't matter what order we access memory as to whether item[0] first, then item[1], or vice versa, or both at the same time (common in hardware), as long as the formulas for one endian are consistently used.
If the item in question is a four-byte value, then there are 4 possible orderings(!) — though only two of them are truly sensible.
For efficiency, the hardware offers us multibyte memory access in one instruction (and with one reference, namely to the lowest address of the multibyte item), and therefore, the hardware itself needs to define and consistently use one of the two possible/reasonable orderings.
If the hardware did not offer multibyte memory access, then the ordering would be entirely up to the software program itself to define (accessing memory one byte at a time), and the program could choose big or little endian, even differently for each variable, as long as it consistently accesses the multiple bytes of memory in the same manner to reassemble the values stored there.
In a similar manner, when we define a structure of multiple items (e.g. struct point { int x; int y; }, software chooses whether x comes first or y comes first in memory ordering.  However, since programmers (and compilers) will still choose to use hardware instructions to access individual fields such as x in one go, the hardware's endian configuration remains necessary.

Related

Is Unity endian-ness platform independent?

I was just wondering if I can reliably expect C# code to be little endian in Unity.
I'm using an int as a bitmap to determine the state of a room where there are four statues. Each statue can have its arms up or down. I use 8 bits to represent the arms. 1 == up 0 == down.
int bit = (int)statueNumber * 2;
if (!isLeftArm) bit += 1;
bool up = (1 == ((roomState >> bit) & 1));
This tells me if an arm is up or down. Eventually I compare "roomState" to another integer represented the "correct" room state. Let's say the correct state is 1010101, then the answer is 85 in little endian. But if it's interpreted as big endian it's another number.

How to read and convert Bluetooth characteristic from byte data to proper values(Bluetooth for flutter)

I have to read and write some values to a Bike Smart trainer with BLE (Bluetooth Low Energy) used with Flutter. When I try to read the values from the GATT characteristic org.bluetooth.characteristic.supported_power_range (found on bluetooth.org site https://www.bluetooth.com/specifications/gatt/characteristics/ ) I get the return value of an Int List [0,0,200,0,1,0].
The GATT characteristic sais that there are 3 sint16 fields for Min., Max. and step size Watts (Power).
The Byte transmission order also sais that the least significant octet is transmitted first.
My guessings are, that the 3 parameters are returned in an Int array with 8bit value each. But I can't interpret the 200 for maybe the maximum Power setting. Because the smart trainer should provide max. 2300W Watts resistance (ELITE Drivo https://www.elite-it.com/de/produkte/home-trainer/rollentrainer-interaktive/drivo)
The Output results from this code snippet:
device.readCharacteristic(savedCharacteristics[Characteristics.SUPPORTED_POWER_RANGE]).then((List<int> result) {
result.forEach((i) {
print(i.toString());
});
});
// result: [0,0,200,0,1,0]
Maybe some one of u knows how to interpret the binary/hex/dec values of the flutter_blue characteristic output.
Or some hints would be great
Edit
For future readers, I got the solution. I'm a bit asheamed because I read the wrong characteristic.
The return value [0,0,200,0,1,0] was for supported resistance level. (which is 20% and the 200 shows the 20% with a resolution of 0.1 like described in the GATT spec)
I also got a return value for the supported power level which was [0,0,160,15,1,0]. Now the solution how to read the 2 Bytes of max powre level: you get the 160,15 the spec sais LSO (least significant octet first, don't confuse it with LSB least significant bit first). In fact of that you have to read it like 15,160. now do the math with the first Byte 15*256 + 160 = 4000 and thats the correct maximum supported power of the trainer like in the datasheet.
I hope I help someone with that. Thanks for the two replys they are also correct and helped me to find my mistake.
I had the same problem connecting to a Polar H10 to recover HR and RR intervals. It might not be 100% the same, but I think my case can guide you to solve yours.
I am receiving the same list as you like these two examples:
[0,60]
[16,61,524,2]
Looking at the specs of the GATT Bluetooth Heart Rate Service I figured that each element of the list retrieved matches 1 byte of the data transmitted by the characteristic you are subscripted to. For this service, the first byte, i.e., the first element of the list, has some flags to point out if there is an RR value after the HR value (16) or not (0). This is just two cases among the many different ones that can ocur depending on the flags values, but I think it shows how important this first byte can be.
After that, the HR value is coded as an unsigned integer with 8 bits (UINT8), that is, the HR values match the second element of the lists shown before. However, the RR interval is coded as an unsigned integer eith 16bits (UINT16), so it complicates the translation of those two last elements of the list #2 [16,61,524,2], because we should use 16 bits to get this value and the bytes are not in the correct order.
This is when we import the library dart:typed_data
import 'dart:typed_data';
...
_parseHr(List<int> value) {
// First sort the values in the list to interpret correctly the bytes
List<int> valueSorted = [];
valueSorted.insert(0, value[0]);
valueSorted.insert(1, value[1]);
for (var i=0; i< (value.length-3); i++) {
valueSorted.insert(i+2, value[i+3]);
valueSorted.insert(i+3, value[i+2]);
}
// Get flags directly from list
var flags = valueSorted[0];
// Get the ByteBuffer view of the data to recode it later
var buffer = new Uint8List.fromList(valueSorted).buffer; // Buffer bytes from list
if (flags == 0) {
// HR
var hrBuffer = new ByteData.view(buffer, 1, 1); // Get second byte
var hr = hrBuffer.getUint8(0); // Recode as UINT8
print(hr);
}
if (flags == 16) {
// HR
var hrBuffer = new ByteData.view(buffer, 1, 1); // Get second byte
var hr = hrBuffer.getUint8(0); // Recode as UINT8
// RR (more than one can be retrieved in the list)
var nRr = (valueSorted.length-2)/2; // Remove flags and hr from byte count; then split in two since RR is coded as UINT16
List<int> rrs = [];
for (var i = 0; i < nRr; i++) {
var rrBuffer = new ByteData.view(buffer, 2+(i*2), 2); // Get pairs of bytes counting since the 3rd byte
var rr = rrBuffer.getUint16(0); // Recode as UINT16
rrs.insert(i,rr);
}
print(rrs);
}
Hope it helps, the key is to get the buffer view of the sorted list, get the bytes that you need, and recode them as the standard points out.
I used print(new String.fromCharCodes(value)); and that worked for me.
value is your return from List<int> value = await characteristic.read();
I thank ukBaz for his answer to this question. Write data to BLE device and read its response flutter?
You can use my package byte_data_wrapper to transform this data to a decimal value which you can understand:
Get the buffer:
import 'dart:typed_data';
final buffer = Uint16List.fromList(result).buffer;
Create the byteDataCreator:
// Don't forget to add it to your pubspec.yaml
//dependencies:
// byte_data_wrapper:
// git: git://github.com/Taym95/byte_data_wrapper.git
import 'byte_data_wrapper/byte_data_wrapper.dart';
final byteDataCreator = ByteDataCreator.view(buffer);
Get your data :
// You can use getUint8() if valeu is Uint8
final min = byteDataCreator.getUint16();
final max = byteDataCreator.getUint16();
final stepSize = byteDataCreator.getUint16();
I know its too late to answer this but if there is anyone still having a trouble, just convert it manually to be an integer. Because I think you are receiving a type of ByteArray (correct me if I'm wrong).
num bytesToInteger(List<int> bytes) {
/// Given
/// 232 3 0 0
/// Endian.little representation:
/// To binary
/// 00000000 00000000 00000011 11101000
/// Combine
/// 00000000000000000000001111101000
/// Equivalent : 1000
num value = 0;
//Forcing to be Endian.little (I think most devices nowadays uses this type)
if (Endian.host == Endian.big) {
bytes = List.from(bytes.reversed);
}
for (var i = 0, length = bytes.length; i < length; i++) {
value += bytes[i] * pow(256, i);
}
return value;
}
and vice versa when you try to write over 255
Uint8List integerToBytes(int value) {
const arrayLength = 4;
return Uint8List(arrayLength)..buffer.asByteData().setInt32(0, value, Endian.little);
}
Hope this helps.
P.S. I also posted the similar problem here.

How to read and write bits in a chunk of memory in Swift

I would like to know how to read a binary file into memory (writing it to memory like an "Array Buffer" from JavaScript), and write to different parts of memory 8-bit, 16-bit, 32-bit etc. values, even 5 bit or 10 bit values.
extension Binary {
static func readFileToMemory(_ file) -> ArrayBuffer {
let data = NSData(contentsOfFile: "/path/to/file/7CHands.dat")!
var dataRange = NSRange(location: 0, length: ?)
var ? = [Int32](count: ?, repeatedValue: ?)
data.getBytes(&?, range: dataRange)
}
static func writeToMemory(_ buffer, location, value) {
buffer[location] = value
}
static func readFromMemory(_ buffer, location) {
return buffer[location]
}
}
I have looked at a bunch of places but haven't found a standard reference.
https://github.com/nst/BinUtils/blob/master/Sources/BinUtils.swift
https://github.com/apple/swift/blob/master/stdlib/public/core/ArrayBuffer.swift
https://github.com/uraimo/Bitter/blob/master/Sources/Bitter/Bitter.swift
In Swift, how do I read an existing binary file into an array?
Swift - writing a byte stream to file
https://apple.github.io/swift-nio/docs/current/NIO/Structs/ByteBuffer.html
https://github.com/Cosmo/BinaryKit/blob/master/Sources/BinaryKit.swift
https://github.com/vapor-community/bits/blob/master/Sources/Bits/Data%2BBytesConvertible.swift
https://academy.realm.io/posts/nate-cook-tryswift-tokyo-unsafe-swift-and-pointer-types/
https://medium.com/#gorjanshukov/working-with-bytes-in-ios-swift-4-de316a389a0c
I would like for this to be as low-level as possible. So perhaps using UnsafeMutablePointer, UnsafePointer, or UnsafeMutableRawPointer.
Saw this as well:
let data = NSMutableData()
var goesIn: Int32 = 42
data.appendBytes(&goesIn, length: sizeof(Int32))
println(data) // <2a000000]
var comesOut: Int32 = 0
data.getBytes(&comesOut, range: NSMakeRange(0, sizeof(Int32)))
println(comesOut) // 42
I would basically like to allocate a chunk of memory and be able to read and write from it. Not sure how to do that. Perhaps using C is the best way, not sure.
Just saw this too:
let rawData = UnsafeMutablePointer<UInt8>.allocate(capacity: width * height * 4)
If you're looking for low level code you'll need to use UnsafeMutableRawPointer. This is a pointer to a untyped data. Memory is accessed in bytes, so 8 chunks of at least 8 bits. I'll cover multiples of 8 bits first.
Reading a File
To read a file this way, you need to manage file handles and pointers yourself. Try the the following code:
// Open the file in read mode
let file = fopen("/Users/joannisorlandos/Desktop/ownership", "r")
// Files need to be closed manually
defer { fclose(file) }
// Find the end
fseek(file, 0, SEEK_END)
// Count the bytes from the start to the end
let fileByteSize = ftell(file)
// Return to the start
fseek(file, 0, SEEK_SET)
// Buffer of 1 byte entities
let pointer = UnsafeMutableRawPointer.allocate(byteCount: fileByteSize, alignment: 1)
// Buffer needs to be cleaned up manually
defer { pointer.deallocate() }
// Size is 1 byte
let readBytes = fread(pointer, 1, fileByteSize, file)
let errorOccurred = readBytes != fileByteSize
First you need to open the file. This can be done using Swift strings since the compiler makes them into a CString itself.
Because cleanup is all for us on this low level, a defer is put in place to close the file at the end.
Next, the file is set to seek the end of the file. Then the distance between the start of the file and the end is calculated. This is used later, so the value is kept.
Then the program is set to return to the start of the file, so the application starts reading from the start.
To store the file, a pointer is allocated with the amount of bytes that the file has in the file system. Note: This can change inbetween the steps if you're extremely unlucky or the file is accessed quite often. But I think for you, this is unlikely.
The amount of bytes is set, and aligned to one byte. (You can learn more about memory alignment on Wikipedia.
Then another defer is added to make sure no memory leaks at the end of this code. The pointer needs to be deallocated manually.
The file's bytes are read and stored in the pointer. Do note that this entire process reads the file in a blocking manner. It can be more preferred to read files asynchronously, if you plan on doing that I'll recommend looking into a library like SwiftNIO instead.
errorOccurred can be used to throw an error or handle issues in another manner.
From here, your buffer is ready for manipulation. You can print the file if it's text using the following code:
print(String(cString: pointer.bindMemory(to: Int8.self, capacity: fileByteSize)))
From here, it's time to learn how to read manipulate the memory.
Manipulating Memory
The below demonstrates reading byte 20..<24 as an Int32.
let int32 = pointer.load(fromByteOffset: 20, as: Int32.self)
I'll leave the other integers up to you. Next, you can alos put data at a position in memory.
pointer.storeBytes(of: 40, toByteOffset: 30, as: Int64.self)
This will replace byte 30..<38 with the number 40. Note that big endian systems, although uncommon, will store information in a different order from normal little endian systems. More about that here.
Modifying Bits
As you notes, you're also interested in modifying five or ten bits at a time. To do so, you'll need to mix the previous information with the new information.
var data32bits = pointer.load(fromByteOffset: 20, as: Int32.self)
var newData = 0b11111000
In this case, you'll be interested in the first 5 bits and want to write them over bit 2 through 7. To do so, first you'll need to shift the bits to a position that matches the new position.
newData = newData >> 2
This shifts the bits 2 places to the right. The two left bits that are now empty are therefore 0. The 2 bits on the right that got shoved off are not existing anymore.
Next, you'll want to get the old data from the buffer and overwrite the new bits.
To do so, first move the new byte into a 32-bits buffer.
var newBits = numericCast(newData) as Int32
The 32 bits will be aligned all the way to the right. If you want to replace the second of the four bytes, run the following:
newBits = newBits << 16
This moves the fourth pair 16 bit places left, or 2 bytes. So it's now on position 1 starting from 0.
Then, the two bytes need to be added on top of each other. One common method is the following:
let oldBits = data32bits & 0b11111111_11000001_11111111_11111111
let result = oldBits | newBits
What happens here is that we remove the 5 bits with new data from the old dataset. We do so by doing a bitwise and on the old 32 bits and a bitmap.
The bitmap has all 1's except for the new locations which are being replaced. Because those are empty in the bitmap, the and operator will exclude those bits since one of the two (old data vs. bitmap) is empty.
AND operators will only be 1 if both sides of the operator are 1.
Finally, the oldBits and the newBits are merged with an OR operator. This will take each bit on both sides and set the result to 1 if the bits at both positions are 1.
This will merge successfully since both buffers contain 1 bits that the other number doesn't set.

Optimizing NSNumber numberWithInt:

I am profiling an iPhone app and I noticed a strange pattern. In a certain block of code that's called quite frequently...
[item setQuadrant:[NSNumber numberWithInt:a]];
[item setIndex:[NSNumber numberWithInt:b]];
[item setTimestamp:[NSNumber numberWithInt:c]];
[item setState:[NSNumber numberWithInt:d]];
[item setCompletionPercentage:[NSNumber numberWithInt:e]];
[item setId_:[NSNumber numberWithInt:f]];
...the first call to [NSNumber numberWithInt:] takes an inordinate amount of time, in the order of 10-15x that of the remaining calls. I've verified that the results are consistent if I shuffle the lines (the first line is always the slow one, by the same ratio). Is there something going on that I'm not aware of?
Perhaps this happens because this block is inside a try/catch?
If I had to guess, NSNumber performs some code in it's +load implementation, which slows down the initial call to the class. Also note that NSNumber caches it's return value, so future calls to +numberWithInt: with the same value are faster than before, that could possibly be part of the issue.
Maybe the first value is much larger? apart from CoreFoundation's CFNumber caching, the "new" runtime uses tagged pointers, allowing integers within the range of 24 bit to be encoded directly into the pointer - the runtime then figure out it's a tagged pointer by looking at the last bit (and that its a CFNumber by looking at the 3 bits before the last bit and the target number size - 8, 16, 32, 64 bit - using the next 4 bits before).
On a 32-bit system (current iPhones), that means that for ("small") negative 32 bit numbers or large positive numbers, CoreFoundation will allocate an object. For everything else, it uses the following expression that is way faster:
case kCFNumberSInt32Type: {
int32_t value = *(int32_t *)valuePtr; // this loads the actual numerical value passed to CFNumberCreate()
#if !__LP64__
// We don't bother allowing the min 24-bit integer -2^23 to also be fast-pathed;
// tell anybody that complains about that to go ... hang.
int32_t limit = (1L << 23);
if (value <= -limit || limit <= value) break;
#endif
uintptr_t ptr_val = ((uintptr_t)((intptr_t)value << 8) | (2 << 6) | kCFTaggedObjectID_Integer);
return (CFNumberRef)ptr_val;
}
(note that !__LP64__ is true for 32-bit systems)
Taken from: http://www.opensource.apple.com/source/CF/CF-744.12/CFNumber.c
Also, there is a caching mechanism that prevents a range of numbers from being re-created multiple times, just search for "__CFNumberCache" in the same source file.

How to store data larger than 128 byte in JavaCard

I can't write data at index above 128 in byte array.
code is given below.
private void Write1(APDU apdu) throws ISOException
{
apdu.setIncomingAndReceive();
byte[] apduBuffer = apdu.getBuffer();
byte j = (byte)apduBuffer[4]; // Return incoming bytes lets take 160
Buffer1 = new byte[j]; // initialize a array with size 160
for (byte i=0; i<j; i++)
Buffer1[(byte)i] = (byte)apduBuffer[5+i];
}
It gives me error 6F 00 (It means reach End Of file).
I am using:
smart card type = contact card
using java card 2.2.2 with jcop using apdu
Your code contains several problems:
As already pointed out by 'pst' you are using a signed byte value which works only up to 128 - use a short instead
Your are creating a new buffer Buffer1 on every call of your Write1 method. On JavaCard there is usually no automatic garbage collection - therefore memory allocation should only be done once when the app is installed. If you only want to process the data in the adpu buffer just use it from there. And if you want to copy data from one byte array into another better use javacard.framework.Util.arrayCopy(..).
You are calling apdu.setIncomingAndReceive(); but ignore the return value. The return value gives you the number of bytes of data you can read.
The following code is from the API docs and shows the common way:
short bytesLeft = (short) (buffer[ISO7816.OFFSET_LC] & 0x00FF);
if (bytesLeft < (short)55) ISOException.throwIt( ISO7816.SW_WRONG_LENGTH );
short readCount = apdu.setIncomingAndReceive();
while ( bytesLeft > 0){
// process bytes in buffer[5] to buffer[readCount+4];
bytesLeft -= readCount;
readCount = apdu.receiveBytes ( ISO7816.OFFSET_CDATA );
}
short j = (short) apdu_buffer[ISO7816.OFFSET_LC] & 0xFF
Elaborating on pst's answer. A byte has 2^8 bits numbers, or rather 256. But if you are working with signed numbers, they will work in a cycle instead. So, 128 will be actually -128, 129 will be -127 and so on.
Update: While the following answer is "valid" for normal Java, please refer to Roberts answer for Java Card-specific information, as well additional concerns/approaches.
In Java a byte has values in the range [-128, 127] so, when you say "160", that's not what the code is really giving you :)
Perhaps you'd like to use:
int j = apduBuffer[4] & 0xFF;
That "upcasts" the value apduBuffer[4] to an int while treating the original byte data as an unsigned value.
Likewise, i should also be an int (to avoid a nasty overflow-and-loop-forever bug), and the System.arraycopy method could be handy as well...
(I have no idea if that is the only/real problem -- or if the above is a viable solution on a Java Card -- but it sure is a problem and aligns with the "128 limit" mentioned.)
Happy coding.