Related
I am trying to use Spray to download a medium-sized file, say a video, from my Scala app.
What I need is to download the file and, for every chuck that I receive (possibly the chuck dimension is something I would set via config), to track the progress of the whole process (a.k.a. progress-bar like).
Additional, I would like to avoid to have the whole file in memory before saving it, instead saving each chuck as I receive it (and log it).
I looked at the documentation, and a bit online.
However, I can't seem to find a clear way of doing this.
The only thing I found was this link which says that this use-case was not supported by spray-client at the time and I should use spray-can directly...no example though, so I am still a bit confused.
Can anybody suggest an approach? Point me in the right direction?
How would you download a file of, say, 30MB?
Thanks
I questionning myself : is it better to use Paster for creating content types, browser view, portlet, etc... or ArchGenXML ?
Which one of those two create the better source code ?
Is there an advantage of using one or the other ?
Thanks.
They're two quite different things.
Paster creates an initial skeleton. Once it's done, you're on your own.
ArchGenXML needs a UML model first, and then it can create quite complex systems of code. As long as you modify your code only within prescribed regions of the .py file, you can change your model and rerun ArchGenXML as often as you wish.
Either one only generates code as good as its authors have provided, and while I use ArchGenXML extensively, I see a fair bit of deprecated code generated. otoh, I've never seen it generate completely invalid code.
I use ArchGenXML because I like having my original source in UML
Just to give you another point of view:
I think that ArchGenXML is a tool for someone that doesn't really want to get his/her hands dirty for as long as possible a tool that let you take a little more time planning and a little less time coding (I see this as a negative point) . Paster on the contrary is just a comfort to speed up some work, and you will get your hands dirty very soon.
I started coding in Plone using paster, then after a while when I felt secure, I abandoned it (like the babywalker :D ). And then you learn to run, and after some time you get old and lazy and you realize that, after all, paster is still your friend.
ArchGenXML, on the other hand, I think that's an obstruction for your learning.
Plus, if you use paster and in the end of the project your code sucks you can blame just yourself (I see this as a good point).
These are just my 2ยข.
I recently became part of a complex embedded project team for which I will be developing a part. For the part which is my responsibility there is only old code and not much documentation.
I am keen to make a good start but shyness and fear of appearing stupid makes it difficult to ask questions. How to ask questions ?
I wanted to ask what techniques do you guys use to understand a project ? I mean there are of lots of technical details which one must remember and keep in context in order to make a design. Your read the code and get some facts but how to move ahead ?
For instance you read the code and the document(s) and get some facts A and fact B . How to reach suitable conclusion X for which you may or may not have needed to take into account facts C and D also ?
Code-reading can be particularly difficult if there is not enough documentation and the code is poorly documented and badly written. I guess the best way now is to find the entry point of the code, and slowly understand its flows and what data it uses. I would keep a look out for
Structure - are there any partitioning of entities/system? Where in the code (and how) do they communicate with each other?
Data - what sort of structures are used to hold the global data? How are the data accessed and saved?
If you are doing C or C++, it is also important to find out how memory is handled and for C++ (and other related non-managed memory OOP languages, I guess), how are object ownership contained.
Since it is an embedded project, are there any non-standard code or coding constructs used?
Reading the code is balanced by writing the documentation.
Write the documentation that your replacement will need. Imagine someone who knows less than you. Explain it for that person.
When you cannot explain something to your replacement, ask questions.
When you have a complete description, you will "know" the system.
And you will have produced complete documentation.
You don't mention what kind of tests exist. If there are test cases, modify them and trace how this would affect the end result.
You might want to look at diagrams which give the entire picture of the logical structure of the system, like, for example, looking at class diagrams in an OOP system would be of great help. Looking at the design diagrams of large and complex apps gives you a clear understanding of how the internal modules of the system are organized and this way its makes the task of figuring out what functionality does a particular piece of code does much much easier. In the absence of diagrams, you're best bet would be to start from the entry point of the app, like main() and proceed from there while you draw(literally draw or write down on paper) your own conclusions about the system(this way you can have your own documentation) and ask your peers if they're correct.
My experience is that it's best to start with some kind of task -- a bug fix or other small change. That will provide focus to your learning. I find it hard to read through a binder or sift through pages of source code or documentation without having a way to apply it.
If you have a sandbox where you can play with changes that you've made without messing up the code base, that can be even more helpful.
I'm occasionally unfortunate enough to have to make alterations to very old, poorly not documented and poorly not designed code.
It often takes a long time to make a simple change because there is not much structure to the existing code and I really have to read a lot of code before I have a feel for where things would be.
What I think would help a lot in cases like this is a tool that would allow one to visualise an overview of the code, and then maybe even drill down for more detail. I suspect such a tool would be very hard to get right, given that is trying to find structure where there is little or none.
I guess this is not really a question, but rather a musing. I should make it into a question - What do others do to assist in getting their head around other peoples code, the good and the bad?
Hmm, this is a hard one, so much to say so little time ...
1) If you can run the code it makes life soooo much easier, breakpoints (especially conditional) break points are you friend.
2) A purists' approach would be to write a few unit tests, for known functionality, then refactor to improve code and understanding, then re-test. If things break, then create more unit tests - repeat until bored/old/moved to new project
3) ReSharper is good at showing where things are being used, what's calling a method for instance, it's static but a good start, and it helps with refactoring.
4) Many .net events are coded as public, and events can be a pain to debug at the best of times. Recode them to be private and use a property with add/remove. You can then use break point to see what is listening on an event.
BTW - I'm playing in the .Net space, and would love a tool to help do this kind of stuff, like Joel does anyone out there know of a good dynamic code reviewing tool?
I have been asked to take ownership of some NASTY code in the past - both work and "play".
Most of the amateurs I took over code for had just sort of evolved the code to do what they needed over several iterations. It was always a giant incestuous mess of library A calling B, calling back into A, calling C, calling B, etc. A lot of the time they'd use threads and not a critical section was to be seen.
I found the best/only way to get a handle on the code was start at the OS entry point [main()] and build my own call stack diagram showing the call tree. You don't really need to build a full tree at the outset. Just trace through the section(s) you're working on at each stage and you'll get a good enough handle on things to be able to run with it.
To top it all off, use the biggest slice of dead tree you can find and a pen. Laying it all out in front of you so you don't have to jump back and forward on screens or pages makes life so much simpler.
EDIT: There's a lot of talk about coding standards... they will just make poor code look consistent with good code (and usually be harder to spot). Coding standards don't always make maintaining code easier.
I do this on a regular basis. And have developed some tools and tricks.
Try to get a general overview (object diagram or other).
Document your findings.
Test your assumptions (especially for vague code).
The problem with this is that on most companies you are appreciated by result. That's why some programmers write poor code fast and move on to a different project. So you are left with the garbage, and your boss compares your sluggish progress with the quick and dirtu guy. (Luckily my current employer is different).
I generally use UML sequence diagrams of various key ways that the component is used. I don't know of any tools that can generate them automatically, but many UML tools such as BoUML and EA Sparx can create classes/operations from source code which saves some typing.
The definitive text on this situation is Michael Feathers' Working Effectively with Legacy Code. As S. Lott says get some unit tests in to establish behaviour of the lagacy code. Once you have those in you can begin to refactor. There seems to be a sample chapter available on the Object Mentor website.
I strongly recommend BOUML. It's a free UML modelling tool, which:
is extremely fast (fastest UML tool ever created, check out benchmarks),
has rock solid C++ import support,
has great SVG export support, which is important, because viewing large graphs in vector format, which scales fast in e.g. Firefox, is very convenient (you can quickly switch between "birds eye" view and class detail view),
is full featured, intensively developed (look at development history, it's hard to believe that so fast progress is possible).
So: import your code into BOUML and view it there, or export to SVG and view it in Firefox.
See Unit Testing Legacy ASP.NET Webforms Applications for advice on getting a grip on legacy apps via unit testing.
There are many similar questions and answers. Here's the search https://stackoverflow.com/search?q=unit+test+legacy
The point is that getting your head around legacy is probably easiest if you are writing unit tests for that legacy.
I haven't had great luck with tools to automate the review of poorly documented/executed code, cause a confusing/badly designed program generally translates to a less than useful model. It's not exciting or immediately rewarding, but I've had the best results with picking a spot and following the program execution line by line, documenting and adding comments as I go, and refactoring where applicable.
a good IDE (EMACS or Eclipse) could help in many cases. Also on a UNIX-platform, there are some tools for crossreferencing (etags, ctags) or checking (lint) or gcc with many many warning options turned on.
First, before trying to comprehend a function/method, i would refactor it a bit to fit your coding conventions (spaces, braces, indentation) and remove most of the comments if they seem to be wrong.
Then I would refactor and comment the parts you understood, and try to find/grep those parts over the whole source tree and refactor them there also.
Over the time, you get a nicer code, you like to work with.
I personally do a lot of drawing of diagrams, and figuring out the bones of the structure.
The fad de jour (and possibly quite rightly) has got me writing unit tests to test my assertions, and build up a safety net for changes I make to the system.
Once I get to a point where I'm comfortable enought knowing what the system does, I'll take a stab at fixing bugs in the sanest way possible, and hope my safety nets neared completion.
That's just me, however. ;)
i have actuaally been using the refactoring features of ReSharper to help m get a handle on a bunch of projects that i inherited recently. So, to figure out another programmer's very poorly structured, undocumented code, i actually start by refactoring it.
Cleaning up the code, renaming methods, classes and namespaces properly, extracting methods are all structural changes that can shed light on what a piece of code is supposed to do. It might sound counterintuitive to refactor code that you don't "know" but trut me, ReSharper really allows you to do this. Take for example the issue of red herring dead code. You see a method in a class or perhaps a strangely named variable. You can start by trying to lookup usages or, ungh, do a text search, but ReSharper will actually detect dead code and color it gray. As soon as you open a file you see in gray and with scroll bar flags what would have in the past been confusing red herrings.
There are dozens of other tricks and probably a number of other tools that can do similar things but i am a ReSharper junky.
Cheers.
Get to know the software intimately from a user's point of view. A lot can be learnt about the underlying structure by studying and interacting with the user interface(s).
Printouts
Whiteboards
Lots of notepaper
Lots of Starbucks
Being able to scribble all over the poor thing is the most useful method for me. Usually I turn up a lot of "huh, that's funny..." while trying to make basic code structure diagrams that turns out to be more useful than the diagrams themselves in the end. Automated tools are probably more helpful than I give them credit for, but the value of finding those funny bits exceeds the value of rapidly generated diagrams for me.
For diagrams, I look for mostly where the data is going. Where does it come in, where does it end up, and what does it go through on the way. Generally what happens to the data seems to give a good impression of the overall layout, and some bones to come back to if I'm rewriting.
When I'm working on legacy code, I don't attempt to understand the entire system. That would result in complexity overload and subsequent brain explosion.
Rather, I take one single feature of the system and try to understand completely how it works, from end to end. I will generally debug into the code, starting from the point in the UI code where I can find the specific functionality (since this is usually the only thing I'll be able to find at first). Then I will perform some action in the GUI, and drill down in the code all the way down into the database and then back up. This usually results in a complete understanding of at least one feature of the system, and sometimes gives insight into other parts of the system as well.
Once I understand what functions are being called and what stored procedures, tables, and views are involved, I then do a search through the code to find out what other parts of the application rely on these same functions/procs. This is how I find out if a change I'm going to make will break anything else in the system.
It can also sometimes be useful to attempt to make diagrams of the database and/or code structure, but sometimes it's just so bad or so insanely complex that it's better to ignore the system as a whole and just focus on the part that you need to change.
My big problem is that I (currently) have very large systems to understand in a fairly short space of time (I pity contract developers on this point) and don't have a lot of experience doing this (having previously been fortunate enough to be the one designing from the ground up.)
One method I use is to try to understand the meaning of the naming of variables, methods, classes, etc. This is useful because it (hopefully increasingly) embeds a high-level view of a train of thought from an atomic level.
I say this because typically developers will name their elements (with what they believe are) meaningfully and providing insight into their intended function. This is flawed, admittedly, if the developer has a defective understanding of their program, the terminology or (often the case, imho) is trying to sound clever. How many developers have seen keywords or class names and only then looked up the term in the dictionary, for the first time?
It's all about the standards and coding rules your company is using.
if everyone codes in different style, then it's hard to maintain other programmer code and etc, if you decide what standard you'll use have some rules, everything will be fine :) Note: that you don't have to make a lot of rules, because people should have possibility to code in style they like, otherwise you can be very surprised.
Does anyone know of a good code obsfucator for Perl? I'm being ask to look into the option of obsfucating code before releasing it to a client. I know obsfucated code can still be reverse engineered, but that's not our main concern.
Some clients are making small changes to the source code that we give them and it's giving us nightmares when something goes wrong and we have to fix it, or when we release a patch that doesn't work with what they've changed. So the intention is just to make it so that it's difficult for them to make their own changes to the code(they're not supposed to be doing that anyway).
I've been down this road before and it's an absolute nightmare when you have to work on "obfuscated" code because it drives up costs tremendously trying to debug a problem on the client's server when you, the developer, can't read the code. You wind up with "deobfuscators", copying the "real code" to the client's server or any of a number of other issues which just become a real hassle to maintain.
I understand where you're coming from, but it sounds like management has a problem and they're looking to you to implement a chosen solution rather than figuring out what the correct solution is.
In this case, it sounds like it's really a licensing or contractual issue. Let 'em have the code open source, but make it a part of the license that any changes they submit have to come back to you and be approved. When you push out patches, check the md5 sums of all code and if it doesn't match what's expected, they're in license violation and will be charged accordingly (and it should be a far, far higher rate). (I remember one company which let us have the code open source, but made it clear that if we changed anything, we've "bought" the code for $25,000 and they were no longer responsible for any bug fixes or upgrades unless we bought a new license).
Don't. Just don't.
Write it into the contract (or revise the contract if you have to), that you are not responsible for changes they make to the software. If they're f-ing up your code and then expecting you to fix it, you have client problems that aren't going to be solved by obfuscating the code. And if you obfuscate it and they encounter an actual problem, good luck in getting them to accurately report line number, etc., in the bug report.
Please don't do that. If you don't want people to alter your Perl code then put it under an appropriate licence and enforce that licence. If people change your code when you licence says that they shouldn't do that, then it's not your problem when your updates not longer work with their installation.
See perlfaq3's answer to "How Can I hide the source for my Perl programs? for more details.
It would seem your main issue is clients modifying code which then makes it difficult for you to support it. I would suggest you ask for checksums (md5,sha, etc) of their files when they come to you for support, and similarly check files' checksums when patching. For example, you can ask the client to provide the output of a provided program which goes through their install and checksums all the files.
Ultimately they have the code, so they can do whatever they want to it. The best you can do is enforce your licenses and to make sure you only support unmodified code.
In this case obfuscating is the wrong approach.
When you release the code to the client you should keep a copy of the code you send them (either on disk or preferably in your version control as a tag/branch).
Then if your client makes changes you can compare the code they have to the code you sent them and easily spot the changes. After all if they feel the need to make changes there is a problem somewhere and you should fix it in the master codebase.
Another alternative for converting your program into a binary is the free PAR-Packer tool on CPAN. There are even filters for code obfuscation, though as others have said, that's possibly more trouble than it's worth.
I agree with the previous suggestions.
However if you really want to, you can look into PAR and/or Filter::Crypto CPAN modules. You can also use them together.
I used the latter (Filter::Crypto) as a really lightweight form of "protection" when we were shipping our product on optical media. It doesn't "protect" you, but it will stop 90% of the people that want to modify your source files.
This isn't a serious suggestion, however take a look at Acme::Buffy.
It will at least brighten your day!
An alternative to obfuscation is converting your script to a binary using something like ActiveState's Perl Dev Kit.
I am running a Windows O/S and use perl2exe from IndigoSTAR. The resulting .EXE file will be unlikely to be changed on-site.
As others have said, "how do I obfuscate it" is the wrong question. "How do I stop the customer from changing the code" is the right one.
The checksum and contract ideas are good for preventing the "problems" you describe, but if the cost to you is the difficulty of rolling-out upgrades and bug-fixes, how are your clients making changes that don't pass the comprehensive test suite? If they are capable of making these changes (or at least, making a change which expresses what they want the code to do), why not simply make it easy/automated for them to open a support ticket and upload the patch? The customer is always right about what the customer wants (they might not have a clue how to do it "the right way", but that's why they are paying you.)
A better reason to want an obfuscator would be for mass-market desktop deployment where you don't have every customer on a standing contract. In that case, something like PAR -- anything which packs the encryption/obfuscation logic into a compiled binary is the way to go.
As several folks have already said: don't.
It's pretty much implicit, given the nature of the Perl interpreter, that anything you do to obfuscate the Perl must be undoable before Perl gets its hands on it, which means you need to leave the de-obfuscation script/binary lying around where the interpreter (and thus your customer) can find it :)
Fix the real problem: checksums and/or a suitably worded license. And support staff trained to say 'you changed it? we're invoking clause 34b of our license, and that'll be $X,000 before we touch it'....
Also, read why-should-i-use-obfuscation for a more general answer.
I would just invite them into my SVN tree on their own branch so they can provide changes and I can see them and integrate their changes into my development tree.
Don't fight it, embrace it.
As Ovid says, it's a contractual, social problem. If they change the code, they invalidate the warranty. Charge them a lot to fix that, but at the same time, give them a channel where they can suggest changes. Also, look at what they want to change and make that part of the configuration if you can. They have something they want to do, and until you satisfy that, they are going to keep trying to get around you.
In Mastering Perl, I talk a bit about defeating obfucators. Even if you do things like making nonsense variables names and the like, modules such as B::Deparse and B::Deobfuscate, along with Perl tools such as Perl::Tidy, make it pretty easy for the knowledgable and motivated person to get your source. You don't have to worry about the unknowledgable and unmotivated so much because they don't know what to do with the code anyway.
When I talk to managers about this, we go through the normal cost benefit analysis. There is all sorts of stuff you could do, but not much of it costs less than the benefit you get.
Good luck,
Another not serious suggestion is to use Acme::Bleach, it will make your code very clean ;-)