I am working on a file downloader in Perl (UNIX/Mac OS X on terminal). I am actually searching for libraries that would work with handling checksum verification, progress bars and actually such things that the CPAN library itself uses. Which libraries and places to look can you recommend? Are there maybe some finished things like that around I don't know of?
More detailed:
downloading files with a progress bar
logging of all the action taken
file checksum verification
reading and parsing configuration files (example: YAML format)
sending of the results to a web service
CPAN is the main place to look for support modules. If you want to do it in Perl, it's quite likely someone's already done it.
For example, for your requirements:
MD5 (checksum) support: Digest::MD5
Progress bar: Term::ProgressBar
Logging: Log::Log4perl
Configuration files: YAML
Do not miss LWP. Specifically, LWP::Simple is likely most of what you need to get started. For checksumming, HTTP headers and the like, you probably want the full LWP user agent.
Look at CPAN::Checksums for the stuff that CPAN uses to create the CHECKSUMS file in each author's directory.
Sad to say, you really have to use POE currently.. Specifically you need to use POE::Component::Client::HTTP (for (Keep Alive)POE::Component::Server::HTTP::KeepAlive pooling), and probably a few more components. I just did this task I had to download 150k photos daily (in SQL) store by the sha1 of their url, resize them, hash them to their SHA1 of the image and hard link this to the sha1 of the url, and update a database to show the sha1 of the image and date downloaded. I did all of this with POE. And, other than a few hard to debug quirks that I'll probably never fix, and random POE core dumps, it works rather well.
We provide our third party affiliates with a much simpler image downloader that takes an image identified in a CSV through the VIN's row, downloads the image, and renames it to vin hyphen [1..n]. You can find it on github. It uses Parallel::ForkMangaer which is another solution but the nature of using it eliminates KA, and pooling that you can get rather easily with POE.
I'd highly suggest not rolling your own threaded solution, history tells us those are typically the worst.
Related
I have a MacOS app coded in swift, and when someone right clicks > show package contents there is a file that reveals some information I do not want the user to see. Is it at all possible to hide that file?
There's no way to secure data on the client (mac) side. If your program can read something, so can a hacker. You can do 3 things about it:
Make it obfuscated enough to make it annoying to deal with, hoping that bad actors would get discouraged.
Make the reward of reading the sensitive data lower, so there's less incentive to do so
Make the sensitive data be black boxed by a server you control and have secured, and have all the sensitive operations be out-sourced to computation on that secure server.
No, you can't hide files in a meaningful way.
If you name the file starting with a dot (".") they are not shown in the Finder by default, but that's very easy to get around.
Better to encrypt the file and decrypt it in your app. That way nosy users can see the file but can't make any sense out of the contents.
I'm looking to create a script for my 80-year old grandmother that downloads the books she needs, and converts them using the command-line version of Calibre, to kindle format so she can read them on her kindle. She gets a lot of her books from a service in the form of Adobe .epub books. AFAIK, none of these books have DRM on them prior to being converted, so let me be clear - I'm not asking how to strip DRM from an ebook.
What I am asking is whether there is a way to programmatically (from the command line is fine if Adobe Digital Editions supports CL args) use the ticket file to request a book from the library, and download it, in .epub form, to the local hard drive. I simply don't want my grandmother to have to go through all of the unneeded screens in Adobe Digital Editions' interface - she gets confused easily, and the interface tends to be overwhelming for her. I simply want to write a function (it can be a system() call to a command... that's fine) that will allow her to take a file received from the library or digital service and automatically retrieve the proper .epub file.
I have all of the other steps ready to go... I just can't find any way to retrieve the book from the service without using the DE interface.
Any suggestions?
Check this S.O. posting, I know it will help ;-)
pdf-adobe-digital-edition
I need to define a processing rule for web data in iOS and thought it would be a good idea to pull the processing rule as a script file from my server and execute it on the iOS device, since the web API I'm interacting with might change URLs or response syntax and I need to be able to fix such issues fast and cannot rely on pushing an update (takes forever).
I wanted to do it with a small JS file that is pulled from my server every once and a while, but unfortunately iOS doesn't include the JavaScriptCore framework.
Are there other options?
Apple developer agreement will not let you run a downloaded, interpreted script, on the device.
Your best bet is probably downloading a data structure (potentially in JSON format) and parse that and take some predefined actions in your client code based on that, rather than trying to execute the downloaded code directly.
You can let a UIWebView run a Javascript snippet, or you could use another scripting language like LUA (don't forget to add LUA for this). The real problem is: You are not allowed to download code from a webserver or somewhere else. Everything must either be already on the device, or calculated at runtime.
Depending on the information that you want, you could use an XML file that includes the new URLs and parse it, but I don't know if this fits your need.
You can compile JavaScriptCore into your app, evidently, and have it approved by Apple. However, as Mehrdad notes, any scripts run in the app must already be in the app at the time the app is reviewed.
I would like to create my own data format for an iPhone app. The files should be similar structured as e.g. Apple's iWork files (.pages). That means, I have a folder with some files in it:
The file 'Juicy.fruit' contains:
Fruits
---> Apple.xml
---> Banana.xml
---> Pear.xml
---> PreviewPicture.png
This folder "Fruits" should be packed in a handy file 'Juicy.fruit'. Compression isn't necessary. How could I achieve this? I've discovered some open source ZIP-libraries. However, I would like to to build my own data format with the iPhones built-in libs (if possible).
Best regards,
Stefan
Okay, so there are three ways I am reading your question, here's my best guess on each one:
You want your .fruit files to be associated with your app via Safari/SMS/some network connection (aka when someone wants to download files made for your app or made by your app).
In this case, you can register a protocol for your app, as discussed here:
iPhone file extension app association
You want the iPhone to globally associate .fruit files with your app, in which case you want to look into Uniform Type Identifiers. Basically, you set up this association in your installer's info.plst file.
You want to know how you can go from having a folder with files in it to that folder being a single file (package) with your .fruit extension.
If that's the case, there are many options out there and I don't see a purpose in rolling your own. Both Microsoft and Adobe simply use a standard zip compression method and use their own extension (instead of .zip). If you drop any office 2007 document, such as docx or Adobe's experimental .pdfxml file into an archive utility (I like 7z, but any descent one will do), you will get a folder with several xml files, just like you're describing for your situation. (This is also how Java's jar file type works, fyi). So unless you have a great reason to avoid standard compression methods (I vote gzip), I would follow the industry lead on this one.
I can definitly appreciate the urge to go DIY at every level possible, but you're basically asking (if it's #3) how you can create your own packaging algorithm, and after reading how some of the most basic compression methods work, I would leave that one alone. Plus I really doubt that Apple has built in libraries for doing something that most people will just use standard methods for.
One last note:
If you are really gunning to do it from scratch (still suggest not), since your files are all XML, you could just create a new XML file that will act as a wrapper of sorts, and have each file go into that wrapper file. But this would be really redundant when it came time to unwrap, as it would have to load the whole file every time. But it would be something like:
Juicy.fruit --
<fruit-wrapper>
<fruit>
<apple>
... content from apple.xml
</apple>
</fruit>
<fruit>
<banana>
... content from banana.xml
</banana>
</fruit>
<fruit>
<pear>
... content from pear.xml
</pear>
</fruit>
<picture>
...URL-encoded binary of preview picture
</picture>
</fruit-wrapper>
But with this idea, you either have to choose to unpack it, and thus risk losing track of the files, overwriting some but not all, etc etc, or you always treat it like one big file, in which case, unlike with archives, you have to load all of the data each time to pull anything out, instead of just pulling the file you want from the archive.
But it could work, if you're determined.
Also, if you are interested, there is a transfer protocol intended specifically for XML over mobile called WBXML (Wap Binary XML). Not sure if it is still taken seriously, but if there is an iPhone library for it, you should research it.
Could anybody explain to me the process of uploading to and downloading form and ftp server with the iPhone SDK. If you could just point me in the right direction (e.g. documentation etc.). How difficult is a task like this?
Thanks in advance.
You can use this. It support all the basic ftp operations:
Download file
Upload file
Delete file
Delete directory
Create directory
List directory contents
[DISCLAIMER] I am the developer of the library, I needed a ftp library too in the past and came over this answer. However, I decided to write one myself because s7ftprequest didn't support at that point several operations that I needed.(like download or list directory)
Try this Simple FTP Download
The Apple documentation will provide far more info in general than I could. Have a look at:
http://developer.apple.com/iphone/library/documentation/Networking/Conceptual/CFNetwork/CFFTPTasks/CFFTPTasks.html#//apple_ref/doc/uid/TP30001132-CH9-SW1
which details the FTP information you need. If you prefer a PDF with all the networking info in it, have a look at:
http://developer.apple.com/iphone/library/documentation/Networking/Conceptual/CFNetwork/CFNetwork.pdf
In this, you'll be particularly interested in Chapter 5. Both detail working with FTP sites, including uploading, downloading, retrieving directory listings, etc.
s7ftprequest only for uploading files to FTP.
The below is sample code from apple
http://developer.apple.com/library/ios/#samplecode/SimpleFTPSample/Introduction/Intro.html
Limitations:
FTPS (that is, FTP over TLS)
deleting items
renaming items
other less common FTP commands
custom FTP commands
I ended up using GoldRacoon. It turns out that in iOS/objc land, there's BlackRaccoon as the original FTP client lib, then WhiteRaccoon was forked from that, and then GoldRacoon was forked from WhiteRaccoon (I think)
pod search GoldRaccoon
... will show you.
I ended up making a few changes (in my own fork) so that you can pass in a successBlock & failBlock into any request, so that block-y callers (like my classes) don't have extra work to manage the delegate callbacks. Github link (my fork): https://github.com/xaphod/GoldRaccoon