I'm trying to find a way to rid redundant compilation and js from a client's GWT code. Problem is that they have a multiple EntryPoint site and a massive model that gets compiled for every module. We're talking about 30 GWT modules and entry points each compiling the entire model package of the app separately. It takes about 15 minutes on my 8 core monster just to GWT compile this beast. And yes, compilation is parallellized and uses all cores (can hardly move my mouse in Ubuntu :) )
To change the architecture to a single module is not really an option I think. Is there no way to have inherits be shared between modules? The modules aren't necessarily that big all of them, but the problem is again that all inherits are compiled redundantly for each module. This of course has negative effects for the end user as well since every page basically has to load the entire model-js again and again.
According to
http://www.gwtproject.org/doc/latest/DevGuideOrganizingProjects.html#DevGuideModuleXml
the suggestion still seems to be to just make one great monolithic module. Isn't there any better way?
Any tips highly appreciated!
As it is said in the GWT Documentation you refer to, GWT mechanism to face the problem of avoiding redundant code is merging all modules in just a a super-gwt-module which includes all sub-modules you have in your applications.
I suppose you are producing a module for a different page or feature at your website, so using a unique module, as I say, implies that you will need a mechanism to run the appropriate application-code per page, based on the url or something.
You can take advantage of using code-splitting, so your modules will be EntryPoints instead of RunAsyncCallbacks, and each module will be compiled in one js fragment which will be loaded asynchronously.
Note that you will include the same javascript fragment in all pages, and this will load other fragments depending on the page.
The advantages of this solution are many:
You only have one compilation process. It could take a long time, but for sure it will take much less than compiling all modules individually because redundant code will be compiled once.
You can maintain different .gwt.xml, one to continue developing the individual modules with its own EntryPoint, and another without EntryPoint which will be inherited by your super-module.
Once compiled, the first fragment loaded (shared by all apps) would be very small, and it will be cached just once, so all apps would load very fast.
Many of the code shared by the modules (gwt-core, jre, etc), could go to the first fragment and would be shared by all the modules, decreasing the final downloaded size of each app.
This is an out-of-the-box solution, gwt compiler makes a good job splitting the code, merging shared code to intermediate modules, and adding the methods to load fragments asynchronously when demanded.
Java ecosystem facilitates modular apps (dependencies, maven, etc).
Otherwise, if you continue wanting individual modules, the way to compile all of them is what you are actually doing: executing gwt compiler once per module (and permutation). You can improve your compilation time, though, having a continuous integration cluster like Jenkins and running jobs in parallel, or using more brute force (memory, cpu, ...).
As you probably know GWT compiles each module into one big JavaScript file and optimizes everything based on all available information about everything in the whole module. This is why you need to compile everything for each module.
One solution might be to do create one big module, but use code splitting similar to the module structure. Than you don't get one very large monolithic JavaScript file, but 'modules' are loaded as needed.
Did you try compiling with less localworkers, instead of using all possible available cores? I've had the best results with localworker set to 4 (even on a 6-core machine).
Related
Suppose I have a source code directory, I want to run a script that would scan the code in the directory and return the languages, frameworks and libraries used in it. I've tried github/linguist, its a great tool which even Github uses to detect the programming languages used in a source code, however I am not able go beyond that and detect the framework exactly.
I even tried tools like it-depends, to fetch the dependencies but, its getting messed up.
Could someone help me out to figure out how I can do this stuff, with an existing tool or if have to make one such tool how should I approach it.
Thanks in Advance
This is, in the general case, impossible. The halting problem precludes any program from being able to compute, in finite time, what other programs may or may not do - including what dependencies it requires to run. Sure, you can make it work for some inputs - but never for all.
So you have to compromise:
which languages do you need to support? it-depends does not try to support Java, for example. Different languages have different ways of calling in dependencies from their source-code. For example, if working with C, you will want to look at #includes.
which build-chains to you need to support? parsing a standard Makefile for C is very different from, say, looking into a Maven pom.xml for Java. Additionally, build-chains can perform arbitrary computation -- and again, due to the halting problem, your dependency-detection program will not be able to "statically" figure out intended behavior. It is entirely possible to link against one library or another one (or none at all) depending on what is detected to exist. What should you output in this case?. For programs that have no documented build process, you simply cannot know their dependencies. Often, the build-process is human-documented but not machine-readable...
what do you consider a library/framework? long-lived libraries can evolve through many different versions, and the fact that one version is required and not another may not be explicit in the source-code. If a code-base depends on behavior found in only a specific, now superseded, version of a library, and no explicit mention of that version is found -- your dependency-detection program will have no way to know about it (unless you code in library-version-specific detection; which is doable, but on a case-by-case basis, and requires deep knowledge of differences between versions).
Therefore the answer to your question is that... it depends (they go into a fair amount of detail regarding limitations). For the specific case of Java + Maven, which is not covered by it-depends, you can use Maven itself, via mvn dependency:tree. Choose a subset of the problem instead of trying to solve it all at once.
GWT used be able to produce and use *.gwtar files before to speed up (at least a little bit) the transpilation - libraries could come with their *.gwtar files and only modified code would need to be fully rebuilt.
I am working on a large GWT application that allow for more modules to be added by the product's end user - kind of like plugins. These modules are specifically packaged and must obey certain contracts. During deployment we rebuild the app using the combined code (existing + module being added). The production build process takes significant time already but we can live with it in development, we use Super Dev mode for the rest. However, end users see multi-minute builds (say 10-20!) when they add a module and that is rather inconvenient. *.gwtar files no longer help/work - we noticed that they started breaking some stuff with GWT 2.7, actually (compiler reported errors with 3rd party libraries but only when *.gwtars were used).
GWT isn't really made to be able to have independently compiled modules talk to one another without rebuilding (that would be a very welcome feature), but we are looking for a way to leverage GWT incremental compilation to speed up the process and enhance our end users' experience. I have not been able to find the documentation on where the intermediate artifacts are stored and whether/how they can be reused.
While the concern about stability of this system exists - i.e. the files may change from release to release, the greatest time is taken by the base product, which also supplies the tooling. Thus, we can change this with every release too, as needed - even if plugins don't come with these artifacts, the transpilation may still be faster.
Can anyone, please, help me figure out how to leverage this (GWT 2.8 incremental transpilation) for the above use case?
I have an app with two components.
A customer facing one for submitting restaurant orders.
A vendor facing one for viewing restaurant orders.
Should I have two modules with different entry points as there is no shared code(except for the domain model objects) between the components?
There is one reason I can think of for why you may want to do this - which is to reduce download size, since some screens/logic may not be used by the customer (and you want the customer pages to load as fast as possible). However you can also achieve this with code splitting: https://developers.google.com/web-toolkit/doc/latest/DevGuideCodeSplitting
I think having two modules is fine as well. No big deal there.
If you are not going to deploy them on two separate nodes, I would go with one module. Because you have to maintain only one I18n files, less static files (html), there will be just one module descriptor (no duplication).
If you decide to use just one module, the code splitting is a good thing to consider to reduce the size of JS user have to download.
There can't be 100% correct answer, it really depends on your project.
Separation into two compiled modules might be good idea, in case when the size of your common logic, which has to be shared between two modules is quite small compared to customer/vendor specific logic and most of the time you are writing code only for customer/vendor. In such case you will get faster refresh time in development mode and faster compilation of individual modules to the case when everything is merged together.
But there is catch, at some point of time, there might be a requirement to create merged customer/vendor mode, because there are users who are customers and vendors at the same time.
I personally prefer approach, when different logical parts of application get their own gwt module, and then there is a root module which links all of them together, plus you have couple of DevOnly modules, which allow you to start only some specific part of application. Example module structure:
Customer module - not compiled separately, depends on Common module
Vendor module - not compiled separately,depends on Common module
Common module - not compiled separately
App module - compiled separately, depends on Customer and Vendor modules
VendorStandalone module - compiled separately, depends on Vendor module, used only for
development
CustomerStandalone module - compiled separately, depends on Customer module, used only for development
Such structure allows you to have fast developing mode ( if possible at all), and at the same time you prepared for case when Vendor & Customer functionality have to be provided together.
My choice of design(Using MVP):
1)Single module
2)Same login page (User pojo must have type i.e vendor or customer).
3)In OnmoduleLoad based upon type I'l open Corresponding vendor or customer presenter
Why??
1)Code re-usability.
2)Reducing maintenance of 2 modules.
Well,i am also waiting to see more design options.
Please refer
Over the years my application has grown from 1MB to 25MB and I expect it to grow further to 40, 50 MB. I don't use DLL's, but put everything in this one big executable.
Having one big executable has certain advantages:
Installing my application at the customer is really: copy and run.
Upgrades can be easily zipped and sent to the customer
There is no risk of having conflicting DLL's (where the customer has not version X of the EXE, but version Y of the DLL)
The big disadvantage of the big EXE is that linking times seem to grow exponentially.
Additional problem is that a part of the code (let's say about 40%) is shared with another application. Again, the advantages are that:
There is no risk on having a mix of incorrect DLL versions
Every developer can make changes on the common code which speeds up developments.
But again, this has a serious impact on compilation times (everyone compiles the common code again on his PC) and on linking times.
The question Grouping DLL's for use in Executable mentions the possibility of mixing DLL's in one executable, but it looks like this still requires you to link all functions manually in your application (using LoadLibrary, GetProcAddress, ...).
What is your opinion on executable sizes, the use of DLL's and the best 'balance' between easy deployment and easy/fast development?
A single executable has a huge positive impact on maintainability. It is easier to debug, deploy (size issues aside) and diagnose in the field. As you point out, it completely sidesteps DLL hell.
The most straightforward solution to your problem is to have two compilation modes, one that builds a single exe for production and one that builds lots of little DLLs for development.
The tenet is: reduce the number of your .NET assemblies to the strict minimum. Having a single assembly is the ideal number. This is for example the case for Reflector or NHibernate that both come as a very few assemblies. My company published free two white books on the topic One big executable or many small DLL's:
Partitioning code base through .NET assemblies and Visual Studio projects (8 pages)
Defining .NET Components with Namespaces (7 pages)
Arguments are developed in these white-books come with invalid/valid reasons to create an assembly and a case study on the code base of the tool NDepend.
The problem is that MS fosters(and is still fostering) the idea that assemblies are components while assemblies are just physical artifact to pack code. The notion of component is a logical artifact and typically an assemblies should contains several components. It is a good idea to partition component with the notion of namespaces although it is not always practicable (especially in the case of a framework with a public API where namespace are used to partition the API and not necessarily the components)
One big executable is definitely beneficial - you can have whole program optimization and less overhead and maintenance is much simpler.
As for the link time - you could have both the "many DLLs" and "one big executable" at the same time. For each DLL have a project configuration that builds a static library. So when you debug things you compile the "DLL" configuration of the project and when you need to ship you compile the "static library" configurations of your projects. Sometimes you will have different behavior in different configurations, but this will have to be addressed per incident.
An easier way to maintain large programs is to compose them into smaller manageable parts. A program can be composed into a shell and modules that add feature to the shell. Large programs like Visual Studio, outlook all use the same concepts. Try this approach to make a more maintainable and robust programs.
I am considering making use of GWT as the front-end to an existing web application.
I can't justify a complete rewrite to 100% GWT in one go. It is likely that I would migrate parts of the system to GWT gradually. However for consistency I would like to make use of the GWT TabPanel, MenuBar, etc as global interface elements from day one.
As an experiment to see how 'legacy' parts of the system could be incorporated, I have done the following.
The application's main page template now loads a small 'wrapper' GWT module on every page. This GWT module looks for a selection of DIVs in the dynamically generated host page. If the DIV is found, a suitable widget is slotted into place, i.e. menuBar, tabPanel.
A lot of the configuration for the included widgets can also be slotted into the host page as JSON structures. For instance, I have implemented an adapter that dynamically sets up a TabPanel in this way. I've also added some very simple widgets that load remote HTML, etc.
As a prototype, this all appears to work perfectly and loads quickly.
However, it seems that GWT apps are really designed to be run from a single host page, not hundreds of dynamically generated ones.
Can anyone highlight any issues that the above approach may run into, particularly as the GWT module increases in size? I would aim to keep the legacy wrapper module intentionally lean. Other functionality would be implemented in separate modules.
How have other people integrated GWT into their front end in a gradual fashion?
One of the ways GWT was designed to be used is exactly as you've used it. We have done that in many of our apps - where there is one GWT module with multiple 'parts' that are loaded based on whether a given id exists on a page or not. So I don't see that you'll have any issues at all going this way. We often use this approach even for new web applications, where we just want a few 'widgets' on the page, rather than coding the whole application in GWT.
It won't make a huge difference, but one thing I would suggest is not putting the GWT javascript code into your main template, but rather only put it on the pages that need it. It's true that if you're not running HTTPs it is cached basically forever, but it seems wrong to get people to load in the module if it's not actually needed on that page. This of course depends on how people use your site, if they are likely to download it anyway then it won't make any difference.
You're doing it right. Avoid avoid avoid the temptation to try to 'minimize' the GWT footprint by breaking it up into multiple separate apps.
The key to GWT performance is to have as few downloads as possible and to make sure they're cached. Loading a 250k bundle once is much better than two 200k bundles and because compression get's better with larger files you really start to reap benefits as things grow.
y-slow & firebug can be really helpful when it comes to convincing yourself of this.
One performance trick you might check out is available in the sample chapter here: http://www.infoq.com/articles/progwt
It shows a mini-architecture around loading GWT widgets into any number of slots and pre-populating data in JavaScript variables. This allows your GWT widgets to load and not require a second HTTP GET to get the data they use. In practice I found that this was a nice performance boost.