import dropbox (9.4.0)
dbx = dropbox.Dropbox("xxx")
print(dbx.users_get_space_usage().used)
outputs 606873703 or 592.65 MB but the dropbox client shows 578.76 MB. This is from the webpage and the windows 10 client.
Am is using the wrong function?
It looks like this is due to using different factors when converting between bytes and "MB" . For reference, see the different definitions of Megabyte vs. Mebibyte.
606873703 / 1000 / 1000 = 606.87 "Megabytes"
606873703 / 1000 / 1024 = 592.65 "Megabytes" (with "less common" 1024000 factor) [your conversion]
606873703 / 1024 / 1024 = 578.76 "Mebibytes" [Dropbox's conversion]
If you want to match Dropbox's formatting, use the "Mebibyte" convention instead.
Related
The microsoft documentation here:
https://learn.microsoft.com/en-us/azure/databricks/kb/sql/find-size-of-table#size-of-a-delta-table
suggests two methods:
Method 1:
import com.databricks.sql.transaction.tahoe._
val deltaLog = DeltaLog.forTable(spark, "dbfs:/<path-to-delta-table>")
val snapshot = deltaLog.snapshot // the current delta table snapshot
println(s"Total file size (bytes): ${deltaLog.snapshot.sizeInBytes}")`
Method 2:
spark.read.table("<non-delta-table-name>").queryExecution.analyzed.stats
For my table, they both return ~300 MB.
But then in storage explorer Folder statistics or in a recursive dbutils.fs.ls walk, I get ~900MB.
So those two methods that are much quicker than literally looking at every file underreport by 67%. This would be fine to use the slower methods except when I try to scale up to the entire container, it takes 55 hours to scan all 1 billion files and 2.6 PB.
So what is the best way to get the size of a table in ADLS Gen 2? Bonus points if it works for folders that are not tables as that's really the number I need. dbutils.fs.ls is single threaded and only works on the driver, so it's not even very parallelizable. It can be threaded but only within the driver.
deltaLog.snapshot returns just the current snapshot. You can have more files present in table's directory, those belong to historical versions that have been deleted/replaced from the current snapshot.
Also it returns 0 without complaints for non-delta paths. So I'm using this piece of code to get a database-level summary:
import com.databricks.sql.transaction.tahoe._
val databasePath = "dbfs:/<path-to-database>"
def size(path: String): Long =
dbutils.fs.ls(path).map { fi => if (fi.isDir) size(fi.path) else fi.size }.sum
val tables = dbutils.fs.ls(databasePath).par.map { fi =>
val totalSize = size(fi.path)
val snapshotSize = DeltaLog.forTable(spark, fi.path).snapshot.sizeInBytes
(fi.name, totalSize / 1024 / 1024 / 1024, snapshotSize / 1024 / 1024 / 1024)
}
display(tables.seq.sorted.toDF("name", "total_size_gb", "snapshot_size_gb"))
This does parallelize on driver only, still it's only file listing, so it's pretty fast. I admit I don't have a billion files, but well, if it's slow for you just use a bigger driver and tune the number of threads.
I am trying to write a .dbc file for a can-open data log (example of one of lines I am trying to use below)
Time 884.163000, ID:2a1, Data Bytes (0)7b (1)00 (2)95 (3)68 (4)e5 (5)8e (6)49 (7)54
I have written .dbc files using both motorola and intel endianness using candb++ covering 16 bit data over 2 bytes, but this has always been with sequential bytes, ie- (2),(3) or (5),(6).
The bytes I need to use for the particular data in the above example are (3) and (7) in intel format (54,68 in this case). I have written a .dbc for just byte 3 shown in the snip below.
BO_ 673 Rig_Pressure: 4 Vector__XXX
SG_ Pressure_Multiplex M : 15|8#0+ (1,0) [0|0] "" Vector__XXX
SG_ Pressure m0 : 31|8#0- (0.45,58) [0.399999999999999|115.15] "Bar" Vector__XXX
I am asking if there is a way to modify the text file (or use cabdb++) to specify each bit or pick 2 non sequential bytes in the .dbc, something like modifying the bit start something like
31|8#0- to 31|8#0- 63|8#0-
I am far from a computer programmer, I much prefer GUI based programs and am only starting out in learning python, so please be gentle!!!
Thank you!
I recently upgraded one of our Graphite instances from 0.9.2 to 1.1.1, and have since run into an issue where, for the lack of a better word, there is a rolling gap of data.
It shows the last few minutes correctly (I'm guessing what's in carbon cache), and after about 10-15 minutes past, it shows all of the data correctly as well.
However, inside that 10-15 minute gap, it's completely blank. I can see the gap both in Graphite, and in Grafana. It disappears after restarting carbon cache, and then comes back about a day later.
Example screenshot:
This happens for most graphs/dashboards I have.
I've spent a lot of effort optimizing disk IO, so I doubt it to be the case -> Cloudwatch shows 100% burst credit for disk. It's an m3.xlarge instance with 4 cores and 16 GB RAM. Swap file is on ephemeral storage and looks barely utilized.
Using 1 Carbon Cache instance with Whisper backend.
storage_schemas.conf:
[carbon]
pattern = ^carbon\.
retentions = 60:90d
[dumbo]
pattern = ^collectd\.dumbo # load test containers, we don't care about their data
retentions = 300:1
[collectd]
pattern = ^collectd
retentions = 10s:8h,30s:1d,1m:3d,5m:30d,15m:90d
[statsite]
pattern = ^statsite
retentions = 10s:8h,30s:1d,1m:3d,5m:30d,15m:90d
[default_1min_for_1day]
pattern = .*
retentions = 60s:1d
Non-default (or potentially relevant) carbon.conf settings:
[cache]
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 100 # was slagging disk write IO until I dropped it down from 500
MAX_CREATES_PER_MINUTE = 50
CACHE_WRITE_STRATEGY = sorted
RELAY_METHOD = rules
DESTINATIONS = 127.0.0.1:2004
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
Graphite local_settings.py
CARBONLINK_TIMEOUT = 10.0
CARBONLINK_QUERY_BULK = True
USE_WORKER_POOL = False
We've seen this with some workloads on 1.1.1, can you try updating carbon to current master? If not 1.1.2 will be released shortly which should solve the problem.
I still have problems with Octave. I need to receive large (>10M) images via a web socket connection, process the data and send it back. I work with Octave 4.2.1 in Windows 7, the sockets package is version 1.2.0 and can be found here: https://octave.sourceforge.io/sockets/
Here is the minimal code example:
pkg load sockets;
rcv_sck=socket(AF_INET, SOCK_STREAM, 0);
bind(rcv_sck,12345);
a=listen(rcv_sck,0);
b=accept(rcv_sck);
data = [];
bufflen = 4600;
total = 0;
count = 0;
while (total < 10000)
[buff,count]=recv(b, bufflen);
data = horzcat(data, buff);
total += count;
endwhile
disconnect (b);
disconnect (rcv_sck);
fileID = fopen('data.jpg','w');
fwrite('data.jpg', data);
fclose(fileID);
imshow ('data.jpg')
it can be tested with netcat
ncat.exe 127.0.0.1 12345 < test.jpg
My problem is, that I am not able to receive data bigger than 4608 bytes. If I send more data, the receive array is cut to this size. This is why i choose 4600 bytes as a buffer length. Now I try to put the chunks of small data blocks together to get the correct data.
But this has two serious issues:
1) I have to know the size of the data I am sending, a problem which can be solved by sending the size first as a parameter. In my example i have set the size to 10000 bytes.
2) More important: it is terribly slow. For a jpeg of 170kB it takes 7 seconds to send the data.
Any hints or tipps are greatly appreciated, thank you.
jan
If you want to read data until disconnect I would suggest
data = recv (b, Inf, MSG_WAITALL);
without any loop or horcat. And on your client side (netcat) use -q0:
netcat.exe -q0 localhost 12345 < test.jpg
Btw, if you want to store it on the server side, why don't you use
system ("netcat -l -p 12345 > data.jpg");
in the first place?
Every iPhone has a NORID (8 bytes) & CHIPID (12 bytes) unique to each phone.
Where is this stored? NOR? seczone? Can it be dumped?
An iPhone requires a NCK to unlock. From what I understand the NCK is 15 characters.
Is it numeric, alpha or alphanumeric?
The security token for check if the NCK is valid is stored encrypted at +0x400 in the seczone.
Is this correct?
Based on what I've read from dogbert's blog, the security token is created using a method similar to the following pseudo code:
deviceKey = SHA1_hash(norID+chipID)
nckKey = custom_hash(norID, chipID, SHA1_hash(NCK), deviceKey)
rawSignature = generateSignature(SHA1_hash(norID+chipID), SHA1_hash(chipID))
Signature = RSA_encrypt(rawSignature, RSAkey)
security token = TEA_encrypt_cbc(Signature, nckKey)
Is the pseudocode correct? If it is then what is the custom hash that is being used? What is being used to generate the rawSignature? What is the RSAKey that is being used? Is it a public key that can be found in the phone?
If the above pseudocode is CORRECT. Then we would have to bruteforce all 15 character combinations to find the correct NCK key right? Because, even though we are able to recover the NORID and CHIPID, we will not be able to use that information to shorten the amount of characters which we need to find.
Correct?
New generations of iPhone OS contains a wildcardticket that is generated during activation process.
but this should be no problem generating once we have the NCK right? Correct?
The NOR ID is the hardware chip id burned into the baseband chip of the device. I don't know where you are getting the 8 bytes from but it is actually burned into the chip and the size is 64 bytes for iPhone 3G and 128 bytes for the iPhone 3GS.
The NCK is a 15 digit (base 10 so it is not alpha-numeric). ie. the max NCK would be 999999999999999
Your device key is wrong.
It should read:
deviceUniqueKey = SHA(NCK + CHIPID + NORID)
teaEncryptedData = &seczone[0x400]
rsaEncryptedData = TEA_DECRYPT(teaEncryptedData, deviceUniqueKey)
validRSAMessage = RSA_DECRYPT(rsaEncryptedData, rsaKey)
When your NCK produces a valid RSA message, you have found the correct NCK to unlock your device.
Here is the python script that can decrypt iPhone baseband memory so you will be able to get all NCK tokens like
CHIP ID
NOR ID
IMEI hushes
Tea hashes
But this script was used only for old basebands (S-Gold chipset) but you can always make your own.
Also here are some ways to dump iphone baseband into the file by using iPhone core dump function or by other script like NOR dumper. Hope this help