I have a list of binaries that link some static libraries. It was identified that a bunch of these libraries are circular dependent. We never ran into troubles because we enclosed these static libraries between -Wl,--start-group, and -Wl,--end-group
Having understood that this is a bad practice, I'm trying to clean the system.
I've come up with a perl script that tells me how these libraries are dependent on each other, like so:
libchld.a depends on libprnt.a, libgprnt.a
libprnt.a depends on libncle.a, libgprnt.a
and goes one.
Now, I should sort these topologically with each node either pointing upwards or downwards.
And then If I find a set of cyclic dependent libraries while sorting topologically, I will have to enclose only those inside a --start-group and --end-group(than enclosing the entire bunch of libraries) there by cleaning up the system.
Are there some perl modules that does this type of sorting already?
Sort::Topological
Graph::Directed
are those I'm trying to check. But, they don't appear to handle if the graph is circular.
Having understood that this is a bad practice, I'm trying to clean the system.
It's a bad practice because you are not using proper layering, not because it's somehow bad for the linker.
Therefore, cleaning up the link line without re-arranging the libraries into a proper hierarchy with no circular dependencies is a pointless exercise.
And if you do rearrange the libraries, then their proper order will be easy to understand and you wouldn't need to use Perl for that.
Related
I don't get how standard library like libc is linked.I use MingW compiler.
I see that there is no libc.dll file in its bin folder.Then how libc is linked?
How does compiler know difference between custom and dynamic library?
We use build tools because they are a practical way to compile code and create executables, deployables etc.
For example. Consider that you have a large Java application consisting of thousands of separate source files split over multiple packages. Suppose that it has a number of JAR file dependencies, some of them for libraries that you have developed, and others for external libraries that can be downloaded from standard places.
You could spend hours manually downloading external JAR files and putting them in the right place. Then you could manually run javac for each of the source files, and jar in multiple times. Each time in the correct directory with the correct command line arguments ... and in the correct order.
And each time you change the source code ... repeat the relevant parts of the above process.
And make sure you don't make a mistake which will cause you to waste time finding test failures, runtime errors, etc caused by not building correctly.
Or ... you could use a build tool that takes care of it all. And does it correctly each time.
In summary, the reasons we use build tools are:
They are less work than doing it by hand
They are more accurate
The results are more reproducible.
I want to know why compiler can't do it?
Because compilers are not designed to perform the entire build process. Just like your oven is not designed to cook a three course meal and serve it up at the dinner table.
A compiler is (typically) designed to compile individual source code files. There is more to building than doing that.
I'm researching about how to package some of my Perl apps and better manage their dependencies to make distribution easier for me and my customers, which most likely doesn't include uploading to CPAN at all. Instead, I would provide custom repos if necessary or, more likely, access to SCMs like Subversion.
CPAN::Meta::Spec seems to provide what I need to describe my apps, their dependencies and even where to get them from, but what I'm wondering is about the level of detail of pre-requisites. The spec contains the following sentence:
The set of relations must be specified as a Map of package names to version ranges.
Requiring packages seems a little too low level for my needs, I would prefer requiring distributions instead. Pretty much the level (from my understanding) tools like Maven and Gradle work at, e.g. Apache Commons Lang vs. Apache Commons IO etc. instead of individual classes like org.apache.commons.lang3.AnnotationUtils or org.apache.commons.io.ByteOrderMark. OTOH, the example in the docs contains the following lines:
requires => {
'perl' => '5.006',
'File::Spec' => '0.86',
'JSON' => '2.16',
},
The line containing perl doesn't look like a package to me and I didn't find some package perl or perl.pm anywhere on my system. Seems to me like that is handled differently to the other things of the example.
I have a system wide folder containing e.g. some utility packages, which seems comparable to some abstract perl to me. That folder should get defined as one distribution, maintain a version number for all of the packages in that folder and therefore should allow other apps to require that whole thing. If I understand the docs correctly, I would need to create not only the META.yml in the folder, but additionally some e.g. sysutils.pm containing package sysutils; and defining some version.
Is there some way to avoid creating that file and really require the distribution itself only?
The META.yml already contains a name and version on it's own, so looks like some abstract thing one could require in theory. I don't see the need of adding an additional .pm-file representing the distribution itself only to allow require to work. It wouldn't contain any business logic in my case.
Thanks!
That's really not what you want to do. You want to pre-req what you actually require. So, for example, if you need File::Spec, that's what you need, regardless of whether it comes from perl core or from a separate CPAN distribution.
I've seen cases where certain modules have moved from CPAN to core, or vice versa. By requiring the module directly, you don't need to ship new releases of your dependent distributions simply because someone you depend on changed their method of distribution.
I've also seen cases where certain modules are split off from their original distributions when it was determined they were valuable as standalone modules. Depending on the module means that you no longer drag in a bunch of other modules for a simple dependency.
What you're more or less looking for is akin to the Task::* modules. No real logic in most of them, just a list of further dependencies.
The Perl dependency system works entirely on package names, on multiple levels. When a CPAN distribution is uploaded, each package within is indexed by PAUSE, which also checks if the uploader has permissions for that package and that the package has a newer version than the currently indexed package. None of these checks care about the distribution as a whole (though the indexer does do other checks at that level).
Then, when a CPAN client sees a dependency, or you tell it to install something, it checks the index for that package name, which tells it what distribution release to install. If it depends on a certain version, that is checked against the $VERSION declared in that package if you have it installed; whereas once a distribution is installed, its "version" is no longer tracked. The distribution level is almost entirely meaningless except that it is what is ultimately downloaded and installed to satisfy these dependencies. This is important, because modules can and do move between distributions, maintaining their version increments, and the package index will always tell you which distribution to get the version you need.
As you noticed, the perl dependency is weird. It's a special case that has been there forever, as a convention to declare what version of Perl you require, you declare a runtime requirement of perl. It is not an indexed module, and every CPAN client and other consumer of CPAN metadata special cases this to either ignore it or treat it as a minimum Perl version, rather than something that can be installed. There's no way to extend this to work for distributions in general, and it would be a bad idea to try.
As an additional note, the CPAN meta spec is a specification for the file named META.json included in CPAN distributions (META.yml is the legacy version), but this file is automatically generated by your authoring tool. It should never be manually created, though you may have your authoring tool manually add certain keys (in which case reading the spec is important), including prereqs. See neilb's blog post for how to specify dependencies for various authoring tools, which will then transpose these into the generated META file, and also how to use cpanfiles to specify dependencies in general.
I'm in the process of moving a fairly large typescript project from internal modules to external modules. I do this because I want to create one core bundle, which, if and when required, can load other bundles. A seccond requirement which I'm keeping in mind is that I'd like to be able to run the bundles (with some modifications if required) on the server with nodeJS aswell.
I first tried using AMD & require.js to build the core bundle, but I came across an issue with circular dependencies. After having reading that this is common with require.js and commonJS is more often adviced for large project I tried that. But now that it's set up together with browserify I have the exact same issue coming up at the exact same place when I run the compiled bundle.
I have something like 10 base classes that havily rely on eachother and form multiple circular dependencies. I don't see any way to remove all of them.
A simplified setup to explain why I don't think I can remove the circular dependencies:
Triples are made of 3 Resources (subject,predicate,object)
Resource has TripleCollections to keep track of which triples its used in
Resource has multiple functions which rely on properties of Triple
Triple has some functions which handle TripleCollections
TripleCollection has multiple functions which uses functions of Triple
TripleCollection.getSubjects() returns a ResourceCollection
ResourceCollection.getTriples() returns a TripleCollection
Resource keeps track of the objects of its triples in ResourceCollections
ResourceCollection uses multiple functions of Resource
I've read multiple related issues here on SO (this one being most helpful), and from what I can gather I only have a few options:
1) Put all the base classes with circular dependencies in 1 file.
This means it will become one hell of a file. And imports will always need aliases:
import core = require('core');
import BaseA = core.BaseA;
2) Use internal modules
The core bundle worked fine (with its circular dependencies) when I used internal typescript modules and reference files. However if I want to create separate bundles and load them at run time, I'm going to have to use shims for all modules with require.js.
Though I don't really like all the aliasing, I think I will try option 1 now, because if it works I can keep going with CommonJS and browserify and later on I can also more easily run everything on the server in node. And I'll have to resort to option 2 if 1 doesn't work out.
Q1: Is there some possible solution I havn't mentioned?
Q2: what is the best setup for a typescript project with 1 core bundle which loads other bundles (which build upon the core) on demand. Which seems to cannot go around circular dependencies. And which preferably can run both on the client and the server.
Or am I just asking for the impossible? :)
Simply put (perhaps simplistically, but I don't think so), if you have ModuleA and ModuleB and both rely on each other, they aren't modules. They are in separate files, but they are not acting like modules.
In your situation, the classes are highly interdependent, you can't use any one of those classes without needing them all, so unless you can refactor the program to try an make the dependencies one-way, rather than two-way (for example), I would treat the group of classes as a single module.
If you do put all of these in a single module, you may still be able to break it into modules by moving some of the dependencies, for example (and all of these questions are largely rhetorical as they are the kind of question you would need to ask yourself), what do Triple and Resource share on each other? Can that be moved into a class that they could both depend on? Why does a Triple need to know about a TripleCollection (probably because this represents some hierarchical data)? There may only be some things you can move, but any dependency removed from this current design will reduce complexity. For example, if the two-way relationship between Triple and Resource could be removed.
It may be that you can't substantially change this design right now, in which case you can put it all in one module and that will largely solve the module loading issue and it will also keep code together that is likely to change for the same reason.
The summary of all of this is:
If you have a two way module dependency, make it a one-way module dependency. Do this by moving code to make the dependency one way or by moving it all into a single larger module that more honestly represents the coupling between the modules.
So my view on your options is slightly different to those in your questions...
1) Try refactoring the code to reduce coupling to see if you can keep smaller modules
Failing that...
2) Put all the base classes with circular dependencies in 1 file.
I wouldn't place the internal modules option on the list - I think external modules are a much better way of managing a large program.
I have been on a "cleaning spree" lately at work, doing a lot of touch-up stuff that should have been done awhile ago. One thing I have been doing is deleted modules that were imported into files and never used, or they were used at one point but not anymore. To do this I have just been deleting an import and running the program's test file. Which gets really, really tedious.
Is there any programmatic way of doing this? Short of me writing a program myself to do it.
Short answer, you can't.
Longer possibly more useful answer, you won't find a general purpose tool that will tell you with 100% certainty whether the module you're purging will actually be used. But you may be able to build a special purpose tool to help you with the manual search that you're currently doing on your codebase. Maybe try a wrapper around your test suite that removes the use statements for you and ignores any error messages except messages that say Undefined subroutine &__PACKAGE__::foo and other messages that occur when accessing missing features of any module. The wrapper could then automatically perform a dumb source scan on the codebase of the module being purged to see if the missing subroutine foo (or other feature) might be defined in the unwanted module.
You can supplement this with Devel::Cover to determine which parts of your code don't have tests so you can manually inspect those areas and maybe get insight into whether they are using code from the module you're trying to purge.
Due to the halting problem you can't statically determine whether any program, of sufficient complexity, will exit or not. This applies to your problem because the "last" instruction of your program might be the one that uses the module you're purging. And since it is impossible to determine what the last instruction is, or if it will ever be executed, it is impossible to statically determine if that module will be used. Further, in a dynamic language, which can extend the program during it's run, analysis of the source or even the post-compile symbol tables would only tell you what was calling the unwanted module just before run-time (whatever that means).
Because of this you won't find a general purpose tool that works for all programs. However, if you are positive that your code doesn't use certain run-time features of Perl you might be able to write a tool suited to your program that can determine if code from the module you're purging will actually be executed.
You might create alternative versions of the modules in question, which have only an AUTOLOAD method (and import, see comment) in it. Make this AUTOLOAD method croak on use. Put this module first into the include path.
You might refine this method by making AUTOLOAD only log the usage and then load the real module and forward the original function call. You could also have a subroutine first in #INC which creates the fake module on the fly if necessary.
Of course you need a good test coverage to detect even rare uses.
This concept is definitely not perfect, but it might work with lots of modules and simplify the testing.
I have different versions of some libraries I'm developing, and want to, from within various plperl functions I've written, load a certain version based on current_database().
(IIRC using use rather than require is preferred, I think because it might cache the library?)
However, my fear is that different databases on the same server will have problems, in either way that I'm thinking of doing it:
1) use lib and then use--if more than one path gets stuck on #INC, it may not be the right one that gets used
2) require--even if this means the right one is always used in the current script, does it mean that the library gets reloaded every time? And either way, if libraries are kept loaded once used, is it possible that namespace pollution from different versions could result in bugs? (E.g. if I have something branch based on whether a variable is defined, and in one version it is by default, and in another it's not--will all the versions now act as if it is, unless I explicitly undefine it rather than just not defining it?)
If plperl is not loaded through shared_preload_libraries, each database session will have its own interpreter freshly initialized at first use, so the libraries included by one session cannot possibly interfere with another session.
See PL/Perl Under the Hood in the manual for more.