How to track progress of medium-sized file download using Spray Client in Scala - scala

I am trying to use Spray to download a medium-sized file, say a video, from my Scala app.
What I need is to download the file and, for every chuck that I receive (possibly the chuck dimension is something I would set via config), to track the progress of the whole process (a.k.a. progress-bar like).
Additional, I would like to avoid to have the whole file in memory before saving it, instead saving each chuck as I receive it (and log it).
I looked at the documentation, and a bit online.
However, I can't seem to find a clear way of doing this.
The only thing I found was this link which says that this use-case was not supported by spray-client at the time and I should use spray-can directly...no example though, so I am still a bit confused.
Can anybody suggest an approach? Point me in the right direction?
How would you download a file of, say, 30MB?
Thanks

Related

Is there a way to get access to the camera buffer in Bevy?

I would like to get access to the camera buffer and pipe and that out to a machine learning application, but am having trouble figuring it out. Apparently camera_node.rs and particularly camera_node_system contain the buffer, but how would I get access to this in a different system? Sorry I am new to Bevy, so not sure if I could just pipe this out to an intermediate variable and re-use that, or if this is basically impossible in the Bevy framework.
I suppose I could co-opt the camera_node_system and put the code I need in there but that is neither elegant nor scales well.
Please take this with a grain of salt, as I too am very new to Bevy, but I suspect this is enabled by way of a recent commit entitled "Add support for reading from mapped buffers"
I am still working my way through this, and will add to this answer as I learn more. Meanwhile, please feel free to add to, or supplant altogether this incomplete answer.

Finding dead code in Template toolkit

We have a number of interconnected tt's in our project. Is there a way to find template code which is never used?
At least in theory, I could hack into Template, make it mark control blocks somehow and write usage data to a file for later analysis (much like Devel::Cover does). But maybe there's already a tool for that? I haven't found one though.
Here's a useful post on this topic but not quite what I want.

Paster vs. ArchGenXML

I questionning myself : is it better to use Paster for creating content types, browser view, portlet, etc... or ArchGenXML ?
Which one of those two create the better source code ?
Is there an advantage of using one or the other ?
Thanks.
They're two quite different things.
Paster creates an initial skeleton. Once it's done, you're on your own.
ArchGenXML needs a UML model first, and then it can create quite complex systems of code. As long as you modify your code only within prescribed regions of the .py file, you can change your model and rerun ArchGenXML as often as you wish.
Either one only generates code as good as its authors have provided, and while I use ArchGenXML extensively, I see a fair bit of deprecated code generated. otoh, I've never seen it generate completely invalid code.
I use ArchGenXML because I like having my original source in UML
Just to give you another point of view:
I think that ArchGenXML is a tool for someone that doesn't really want to get his/her hands dirty for as long as possible a tool that let you take a little more time planning and a little less time coding (I see this as a negative point) . Paster on the contrary is just a comfort to speed up some work, and you will get your hands dirty very soon.
I started coding in Plone using paster, then after a while when I felt secure, I abandoned it (like the babywalker :D ). And then you learn to run, and after some time you get old and lazy and you realize that, after all, paster is still your friend.
ArchGenXML, on the other hand, I think that's an obstruction for your learning.
Plus, if you use paster and in the end of the project your code sucks you can blame just yourself (I see this as a good point).
These are just my 2ยข.

How should I do RPC in Perl with Catalyst?

I've been trying to find a good form of RPC to standardize on, but so far I've ran into a ton of walls and was wondering what the stackoverflow communities view was.
My ideal RPC would provide the following:
Somewhat wide support in other languages, in that people shouldn't have to write a custom stack to use our server
Input validation
Ideally, some way to turn the above input validation into some sort of automated documentation to distribute
Clean and maintainable code
I am a fan of the catalyst framework and would prefer to stick to it, but if there is a clearly better alternative for RPC servers I'd be open to that as well.
So far I have looked at the following:
Catalyst::Controller::SOAP
Doesn't appear to support returning of complex data structures, only string('literals'). I could probably serialize data on top of that, but that seems very hacky. It also lets you return a pre-formed XML object, but I couldn't get that to work and it looks like you'd need to re-create a lot of SOAP data structure for it to work.
I do like the idea of WSDLs, but the complexity of the overall spec also makes me wonder how well support for communicating with other languages will be.
Custom POSTing XML based controller
We tried to roll our own by hand in a similar way to how we've seen two other projects do it recently where there is a dispatch url that you post XML to. This lets you have XSD validation/documentation, but required creating a lot more code than we want to maintain at this point.
Catalyst::Plugin::Server::XMLRPC
Gave a warning about using a deprecated method that will be removed in a future version of Catalyst.
No input validation or doc creation, but otherwise the best I've found
JSONRPC
Looks pretty similar to XMLRPC only the module is actually updated. I'll probably go with this next unless someone suggests something better
There also appears to be two different modules for catalyst that do JSONRPC
I realize that REST isn't pure RPC (only a subset), but...
I would recommend the Catalyst::Controller::REST and Catalyst::Action::REST modules. They're frequently updated and the documentation is fairly good. There is also a good (but rather dated) example of using Catalyst::Controller::Rest in the 2006 Catalyst Advent calendar titled Day 9 - Web Services with Catalyst::Action::REST.
FWIW, Catalyst::Controller::SOAP does support returning complex data. Take a look at the documentation http://search.cpan.org/~druoso/Catalyst-Controller-SOAP-1.23/lib/Catalyst/Controller/SOAP.pm, which will show you that you can use a WSDL to describe your service. Also, see http://daniel.ruoso.com/categoria/perl/soap-today.html.en for a more detailed step-by-step process.

Is there a good obfuscater for Perl code?

Does anyone know of a good code obsfucator for Perl? I'm being ask to look into the option of obsfucating code before releasing it to a client. I know obsfucated code can still be reverse engineered, but that's not our main concern.
Some clients are making small changes to the source code that we give them and it's giving us nightmares when something goes wrong and we have to fix it, or when we release a patch that doesn't work with what they've changed. So the intention is just to make it so that it's difficult for them to make their own changes to the code(they're not supposed to be doing that anyway).
I've been down this road before and it's an absolute nightmare when you have to work on "obfuscated" code because it drives up costs tremendously trying to debug a problem on the client's server when you, the developer, can't read the code. You wind up with "deobfuscators", copying the "real code" to the client's server or any of a number of other issues which just become a real hassle to maintain.
I understand where you're coming from, but it sounds like management has a problem and they're looking to you to implement a chosen solution rather than figuring out what the correct solution is.
In this case, it sounds like it's really a licensing or contractual issue. Let 'em have the code open source, but make it a part of the license that any changes they submit have to come back to you and be approved. When you push out patches, check the md5 sums of all code and if it doesn't match what's expected, they're in license violation and will be charged accordingly (and it should be a far, far higher rate). (I remember one company which let us have the code open source, but made it clear that if we changed anything, we've "bought" the code for $25,000 and they were no longer responsible for any bug fixes or upgrades unless we bought a new license).
Don't. Just don't.
Write it into the contract (or revise the contract if you have to), that you are not responsible for changes they make to the software. If they're f-ing up your code and then expecting you to fix it, you have client problems that aren't going to be solved by obfuscating the code. And if you obfuscate it and they encounter an actual problem, good luck in getting them to accurately report line number, etc., in the bug report.
Please don't do that. If you don't want people to alter your Perl code then put it under an appropriate licence and enforce that licence. If people change your code when you licence says that they shouldn't do that, then it's not your problem when your updates not longer work with their installation.
See perlfaq3's answer to "How Can I hide the source for my Perl programs? for more details.
It would seem your main issue is clients modifying code which then makes it difficult for you to support it. I would suggest you ask for checksums (md5,sha, etc) of their files when they come to you for support, and similarly check files' checksums when patching. For example, you can ask the client to provide the output of a provided program which goes through their install and checksums all the files.
Ultimately they have the code, so they can do whatever they want to it. The best you can do is enforce your licenses and to make sure you only support unmodified code.
In this case obfuscating is the wrong approach.
When you release the code to the client you should keep a copy of the code you send them (either on disk or preferably in your version control as a tag/branch).
Then if your client makes changes you can compare the code they have to the code you sent them and easily spot the changes. After all if they feel the need to make changes there is a problem somewhere and you should fix it in the master codebase.
Another alternative for converting your program into a binary is the free PAR-Packer tool on CPAN. There are even filters for code obfuscation, though as others have said, that's possibly more trouble than it's worth.
I agree with the previous suggestions.
However if you really want to, you can look into PAR and/or Filter::Crypto CPAN modules. You can also use them together.
I used the latter (Filter::Crypto) as a really lightweight form of "protection" when we were shipping our product on optical media. It doesn't "protect" you, but it will stop 90% of the people that want to modify your source files.
This isn't a serious suggestion, however take a look at Acme::Buffy.
It will at least brighten your day!
An alternative to obfuscation is converting your script to a binary using something like ActiveState's Perl Dev Kit.
I am running a Windows O/S and use perl2exe from IndigoSTAR. The resulting .EXE file will be unlikely to be changed on-site.
As others have said, "how do I obfuscate it" is the wrong question. "How do I stop the customer from changing the code" is the right one.
The checksum and contract ideas are good for preventing the "problems" you describe, but if the cost to you is the difficulty of rolling-out upgrades and bug-fixes, how are your clients making changes that don't pass the comprehensive test suite? If they are capable of making these changes (or at least, making a change which expresses what they want the code to do), why not simply make it easy/automated for them to open a support ticket and upload the patch? The customer is always right about what the customer wants (they might not have a clue how to do it "the right way", but that's why they are paying you.)
A better reason to want an obfuscator would be for mass-market desktop deployment where you don't have every customer on a standing contract. In that case, something like PAR -- anything which packs the encryption/obfuscation logic into a compiled binary is the way to go.
As several folks have already said: don't.
It's pretty much implicit, given the nature of the Perl interpreter, that anything you do to obfuscate the Perl must be undoable before Perl gets its hands on it, which means you need to leave the de-obfuscation script/binary lying around where the interpreter (and thus your customer) can find it :)
Fix the real problem: checksums and/or a suitably worded license. And support staff trained to say 'you changed it? we're invoking clause 34b of our license, and that'll be $X,000 before we touch it'....
Also, read why-should-i-use-obfuscation for a more general answer.
I would just invite them into my SVN tree on their own branch so they can provide changes and I can see them and integrate their changes into my development tree.
Don't fight it, embrace it.
As Ovid says, it's a contractual, social problem. If they change the code, they invalidate the warranty. Charge them a lot to fix that, but at the same time, give them a channel where they can suggest changes. Also, look at what they want to change and make that part of the configuration if you can. They have something they want to do, and until you satisfy that, they are going to keep trying to get around you.
In Mastering Perl, I talk a bit about defeating obfucators. Even if you do things like making nonsense variables names and the like, modules such as B::Deparse and B::Deobfuscate, along with Perl tools such as Perl::Tidy, make it pretty easy for the knowledgable and motivated person to get your source. You don't have to worry about the unknowledgable and unmotivated so much because they don't know what to do with the code anyway.
When I talk to managers about this, we go through the normal cost benefit analysis. There is all sorts of stuff you could do, but not much of it costs less than the benefit you get.
Good luck,
Another not serious suggestion is to use Acme::Bleach, it will make your code very clean ;-)