How can I add a custom task to sbt build definition that consumes (uses classes, runs methods etc.) the project sources ? It seems it tries to find them before compiling even.
I need to know why you want to call the methods, since this changes the answer. If you want to do something ...
Build related
Want to use class/ methods which do something build related ( minfiy things, uploaded jar/wars to servers, etc..)
you have to put this in a plugin or you need to put the sources in a project folder.
The code cannot be called from your project
If it is build related, someone has probably dealt with a similar problem and there is probably a sbt plugin already, but if not, then let me know and I can explain creating sbt plugins.
Non-build related
Want to just call / test methods which don't have anything to do with the build cycle.
You can put this in an object in your project called Script (name it whatever), start up console and then import script object and run.
To make this even easier you can make a custom import script for the console which imports all scripts automatically, which you can then run
So for example,
package script
object Script {
def foo = println("I am doing things non-build related")
}
in sbt now run
console
>> import script._
>> foo // prints out "I am doing things non-build related"
Related
To speed up our development workflow we split the tests and run each part on multiple agents in parallel. However, compiling test sources seem to take most of the time for the testing steps.
To avoid this, we pre-compile the tests using sbt test:compile and build a docker image with compiled targets.
Later, this image is used in each agent to run the tests. However, it seems to recompile the tests and application sources even though the compiled classes exists.
Is there a way to make sbt use existing compiled targets?
Update: To give more context
The question strictly relates to scala and sbt (hence the sbt tag).
Our CI process is broken down in to multiple phases. Its roughly something like this.
stage 1: Use SBT to compile Scala project into java bitecode using sbt compile We compile the test sources in the same test using sbt test:compile The targes are bundled in a docker image and pushed to the remote repository,
stage 2: We use multiple agents to split and run tests in parallel.
The tests run from the built docker image, so the environment is the
same. However, running sbt test causes the project to recompile even
through the compiled bitecode exists.
To make this clear, I basically want to compile on one machine and run the compiled test sources in another without re-compiling
Update
I don't think https://stackoverflow.com/a/37440714/8261 is the same problem because unlike it, I don't mount volumes or build on the host machine. Everything is compiled and run within docker but in two build stages. The file modified times and paths are retained the same because of this.
The debug output has something like this
Initial source changes:
removed:Set()
added: Set()
modified: Set()
Invalidated products: Set(/app/target/scala-2.12/classes/Class1.class, /app/target/scala-2.12/classes/graph/Class2.class, ...)
External API changes: API Changes: Set()
Modified binary dependencies: Set()
Initial directly invalidated classes: Set()
Sources indirectly invalidated by:
product: Set(/app/Class4.scala, /app/Class5.scala, ...)
binary dep: Set()
external source: Set()
All initially invalidated classes: Set()
All initially invalidated sources:Set(/app/Class4.scala, /app/Class5.scala, ...)
Recompiling all 304 sources: invalidated sources (266) exceeded 50.0% of all sources
Compiling 302 Scala sources and 2 Java sources to /app/target/scala-2.12/classes ...
It has no Initial source changes, but products are invalidated.
Update: Minimal project to reproduce
I created a minimal sbt project to reproduce the issue.
https://github.com/pulasthibandara/sbt-docker-recomplile
As you can see, nothing changes between the build stages, other than running in the second stage in a new step (new container).
While https://stackoverflow.com/a/37440714/8261 pointed at the right direction, the underlying issue and the solution for this was different.
Issue
SBT seems to recompile everything when it's run on different stages of a docker build. This is because docker compresses images created in each stage, which strips out the millisecond portion of the lastModifiedDate from sources.
SBT depends on lastModifiedDate when determining if sources have changed, and since its different (the milliseconds part) the build triggers a full recompilation.
Solution
Java 8:
Setting -Dsbt.io.jdktimestamps=true when running SBT as recommended in https://github.com/sbt/sbt/issues/4168#issuecomment-417655678 to workaround this issue.
Newer:
Follow recomendation in https://github.com/sbt/sbt/issues/4168#issuecomment-417658294
I solved the issue by setting SBT_OPTS env variable in the docker file like
ENV SBT_OPTS="${SBT_OPTS} -Dsbt.io.jdktimestamps=true"
The test project has been updated with this workaround.
Using SBT:
I think there is already an answer to this here: https://stackoverflow.com/a/37440714/8261
It looks tricky to get exactly right. Good luck!
Avoiding SBT:
If the above approach is too difficult (i.e. getting sbt test to consider that your test classes do not need re-compiling), you could instead avoid using sbt but instead run your test suite using java directly.
If you can get sbt to log the java command that it is using to run your test suite (e.g. using debug logging), then you could run that command on your test runner agents directly, which would completely preclude sbt re-compiling things.
(You might need to write the java command into a script file, if the classpath is too long to pass as a command-line argument in your shell. I have previously had to do that for a large project.)
This would be a much hackier approach that the one above, but might be quicker to get working.
A possible solution might be defining your own sbt task without dependencies or try to change the test task. For example you could create a task to run a JUnit runner if that was your testing framework. To define a task see this on Implementing Tasks.
You could even go as far as compiling sending the code and running the remotes from the same task as it is any scala code you want. From the sbt reference manual
You could be defining your own task, or you could be planning to redefine an existing task. Either way looks the same; use := to associate some code with the task key
I'm using C++98, and besides the standard library, I only have access to an old version of Boost (which thankfully has Boost Test). The documentation though is daunting, long, and I just don't know where to begin.
I have some experience doing unit testing in Java (and I'm looking for unit testing in C++), and I've seen test packages that contain the unit test code separate from the src package, and I also saw Where do you put your unit test? as well as Unit Testing with Boost and Eclipse. Their suggestions vary, and present reasoning for different packaging structures for separating test code from production code, or having them together.
Before I even started looking into Boost Test, I created this structure within Eclipse (perhaps erroneously):
-- ProjectName
|-- Debug
|-- src
|-- test
and I wrote another main method to run test functions. Eclipse didn't like that, because I had two main methods in the same project. I fumbled around through Project Properties and didn't find anything useful for separating my production code from test code when building (linking, really). My temporary fix was to just use g++ in the terminal and ad hoc compile just my "test" code.
I found something suggesting on Boost::Test -- generation of Main()? that Boost actually generated its own main method, so this is currently my plan of attack for unit testing, especially for having a library of testing tools already available.
What is the conventional way of organizing unit tests for C++?
How do I get started with Boost Test? (Boost is already installed)
Is there anything I need to change in Eclipse to be able to run my Boost unit tests separate from my production code within the IDE? (One of the nice things about IntelliJ, with Java, is how it'll automatically run any main method you like with a click) -- The goal here to be able to build and run my tests within Eclipse.
Should my tests be in a separate Eclipse project? (this was suggested in an answer to the second SO question I linked)
Edit: I found this article for an introduction to Boost Test, but it doesn't discuss how it can be handled within an IDE setting.
I figured out how to do this on my own, and I'll document my solution for others who are just starting with C++ as well and need to do testing of their code. There currently is no good introduction anywhere that I could find. Here are the resources that I used though (and found useful):
C++ Unit Testing With Boost Test which introduces Boost Test in a much better fashion than Boost's documentation.
Where do you put your unit test? which discusses the most conventional ways of doing tests with C++.
unit test in eclipse g++ which discusses Eclipse build configurations which allow testing
The C++ convention for testing is like that of other coding languages, just write your tests in a directory called test under the project. Using Boost Test requires that you link the unit test framework: -l boost_unit_test_framework which in Eclipse:
Right click on your project, go to Properties, C/C++ Build, Settings, Tool Settings, GCC C++ Linker, Libraries, and add the library name boost_unit_test_framework (add -mt to the name if you require multithreading; additionally, once the testing build configuration exists, you can go back and choose just that configuration to link the library -- it'll reduce the size of your executable for your other builds).
To be able to run the unit tests in Eclipse separate from your main method, we need to establish a new build configuration. That way, Eclipse knows to exclude your source file with a main method when you're executing your tests.
Click on Project, Build Configurations, Manage..., and select New... and call it Test (or something other than test). Choose your existing configuration so that we'll inherit properties from the production build.
Next, we need to differentiate the build configurations from one another so when we build them, they actually correspond to the production and test builds.
Right click on test, Resource Configurations, Exclude from Build..., and select the builds that represent your production build (i.e. Debug and or Release). Once done with that, right click on your source file with your main method, and exclude that from the Test build.
There's still some things we need to change. We don't have any test code yet, but we still can't run our test build, nor would our test build know about resources existing in src because Eclipse wouldn't automatically include those source files. They're practically invisible to your test code in test.
Right click on your project, go to Properties, C/C++ Build, Settings, Tool Settings, GCC C++ Compiler, Includes, and add the path /.../workspace/ProjectName.
Now to be able to run your test build in Eclipse, it needs to know what executable you're expecting the IDE to run.
Click on Run, Run Configurations..., and looking at your current run configuration, consolidate these settings by giving, for example, a debug build the name "Debug Build", the C/C++ Application "Debug/Artifact_Name", and the Build Configuration "Debug". Next, create a new run configuration, and call it something like "Test Build", set C/C++ Application to "Test/Artifact_Name", and ensure the build configuration is Test.
Now you'll be able to switch between running production code and test code by selecting either the "active" build configuration or running the correct run configuration.
Finally, here's an example of a unit test with Boost Test once all of this is set up:
//unit_tests.cpp
#define BOOST_TEST_DYN_LINK
#define BOOST_TEST_MODULE someModuleName
#include <boost/test/unit_test.hpp>
#include <src/some_object.h>
struct template_objects {
some_object x;
template_objects() {
BOOST_TEST_MESSAGE("Setting up testing objects");
}
~template_objects() {
BOOST_TEST_MESSAGE("Tearing down testing objects");
}
}
BOOST_FIXTURE_TEST_SUITE(testSuiteName, template_objects)
BOOST_AUTO_TEST_CASE(testCase1) {
x.update();
BOOST_CHECK(x.is_up_to_date());
}
BOOST_AUTO_TEST_CASE(testCase2) {
BOOST_CHECK(x.is_not_up_to_date());
}
BOOST_AUTO_TEST_SUITE_END()
This demonstrates some critical things about using Boost Test:
defining BOOST_TEST_DYN_LINK is recommended; you need to define a way of linking the testing framework before including any of Boost's libraries
you must name the "module" something, doesn't have to be the file name
to get automated setup up and teardown of objects before entering test cases, Boost Test has fixtures, which allow you to call an object's pre-existing state multiple times
the struct is what groups these fixtures, and it's implied that your object should have a well defined constructor and destructor for automatic scoping (if you didn't call new, you don't need delete in teardown)
test suites are just a way of logically grouping your test cases (I haven't tested it yet, but you can probably break up your suites into multiple files for better logical separation)
Additional tidbit: to make Boost Test more verbose, go to your run configuration for the test build, and add the argument --log_level=test_suite.
I have 4 projects, which are depended each other,I am suppose to run 4 projects one by one.
Is there any way to run as single package?
I suppose that you're running the SOAPUI project using testRunner. Looking at testRunner documentation you can see that it's only possible to pass one project to the call.
However if you're using some automation build tool like gradle you can create and approach creating a custom task to for example make various calls to testRunner passing your projects to execute all sequentially, for more details take a look on this answer.
If you're not using any automation tool or you don't know how to implement it then as a possible workaround you can simply can create a CLI script to do so. For example supposing that you've the SOAPUI_HOME/bin in your classpath on Windows you can create myTestRunner.bat with the follow content:
call testrunner "path/to/your/project1.xml"
call testrunner "path/to/your/project2.xml"
call testrunner "path/to/your/project3.xml"
...
Hope it helps,
I have an issue when trying to import in scala. The object Database exists under com.me.project.database but when I try to import it:
import com.me.project.database.Database
I get the error:
object Database is not a member of package com.me.project.controllers.com.me.project.database
Any ideas what the problem is?
Edit:
It is worth mentioning that the import is in the file Application.scala under the package com.me.project.controllers, I can't figure out why it would append the import to the current package though, weird...
Edit 2:
So using:
import _root_.com.me.project.database.Database
Does work as mentioned below. But should it work without the _root_? The comments so far seem to indicate that it should.
Answer:
So it turns out that I just needed to clean the project for the import to work properly, using both:
import _root_.com.me.project.database.Database
import com.me.project.database.Database
are valid solutions. Eclipse had just gotten confused.
imports can be relative. Is that the only import you have? be careful with other imports like
import com.me
ultimately, this should fix it, then you can try to find more about it:
import _root_.com.me.project.database.Database
In my case I also needed to check that object which is not found as a member of package is compiled successfully.
I realize this question already has an accepted answer, but since I experienced the same problem but with a different cause I figured I'd add an answer.
I had a bunch of interdependent projects which suddenly needed a root import in order to compile. It turned out that I had duplicated the package declaration in a single file. This caused some kind of chain reaction and made it very hard to find the source of the problem.
In summary I had
package foo.bar
package foo.bar
on the top of the file instead of just
package foo.bar
Hope this saves someone some really tedious error hunting.
In my case I had to run sbt clean.
I had faced similar issue where IntelliJ showed error on importing one file from the same project.
What did not resolve the issue in my case:
adding _root_ in import statement
sbt clean
restarting machine
What actually resolved the issue:
main menu => select File => click on Invalidate Caches / Restart => pop-up dailog => click on invalidate the caches and restart.
I was using IDEA (2019.2.2 Ultimate Edition) on macOs mojave 10.14.6
Java -> Scala conversion without cleaning
Don't forget to clean if you convert some file in a project from Java to Scala. I had a continuous integration build running where I couldn't get things to work, even though the build was working locally, after I had converted a Java class into a Scala object. Solution: add 'clean' to the build procedure on the CI server. The name of the generated .class file in Scala is slightly different than for a Java class, I believe, so this is very likely what was causing the issue.
If you are using gradle as your build tool, then ensure that jar task is not disabled.
I had multiple modules in my project, where one module was dependent on a few other modules. However, I had disabled jar task in build.gradle:
jar {
enabled = false
}
That caused it to fail to resolve classes in the dependent modules and fail with the above error.
I will share my story, just in case it may help someone.
Scenario: intellij compilation succeeds, but gradle build fails on import com.foo.Bar, where Bar is a scala class.
TLDR reason: Bar was located under src/main/java/... as opposed to src/main/scala/...
Actual reason: Bar was not being compiled by compileScala gradle task (from gradle scala plugin) because it looks for scala sources only under src/<sourceSet>/scala.
From docs.gradle.org:
All the Scala source directories can contain Scala and Java code. The
Java source directories may only contain Java source code.
Hope this helps
I had a similar problem but none of the solutions here worked for me. What did work however was a simple restart of my machine.
Perhaps it was something with my Intellij but after a quick restart, everything seems to be working fine.
I had a similar situation, which was failing in both IntelliJ and maven on the command line. I went to apply the suggested temp fix (adding _root_) but intellij was glitching so bad that wasn't even possible.
Eventually I noticed that I had mis-created a package so that it repeated the whole path of the package. That meant that the directory my class was in had a subfolder called "com", and the start of my file looked like:
package com.mycompany.mydept.myproject.myfunctionality.sub1
import com.holdenkarau.spark.testing.DataFrameSuiteBase
where I had another package called
com.mycompany.mydept.myproject.myfunctionality.sub1.com.mycompany.mydept.myproject.myfunctionality.sub2
And the compiler was looking for "holdenkarau" under com.mycompany.mydept.myproject.myfunctionality.com and failing.
I had this issue while using Intellij and the built-in sbt shell (precisely, I was trying to run the command console, which invokes a compiler check of the code).
In my case, after trying the other suggested solutions on this thread, I found that I could restart the sbt shell and it would go away. There's a button on the left-hand side of a looped green arrow and a small grey square which does this in one click (obviously, this is subject to Jet Brains not changing the design of the IDE!!!).
I hope this helps some people get past this issue quickly.
In my case, In Intellij, Just renaming the package file to something else >> see if it updates the import statements >> run the code >> then renaming back to the original name worked.
I am new to Scala and to using emacs + ensime + sbt setup for my Scala development.
This setup is quite nice and light, but there is one thing which drives me nuts - inability to auto-compile / reload changes into Scala console started from sbt.
I use REPL a lot and would like to be able to start REPL from sbt with console command and test my changes to scala classes from REPL without having to close it and reload every time I make a change.
I come from Erlang environment and this way of development is easy with Erlang but seems to be difficult with SBT. I have the JRebel plug-in installed but it doesn't seem to be working for the situation I described.
Has anybody been able to make something similar work and would be willing to share the configuration steps?
Much appreciated in advance.
There are two things possible in sbt:
Causing automatic recompilation of the project sources triggered by a file change by prefixing a command with ~ (tilde). The console, or console-quick, or console-project commands can be prefixed, too, but you have to exit REPL to make the recompilation happen (just hit Ctrl+D and wait.)
Causing automatic execution of REPL commands just after firing the console. They can be defined as properties (e.g. in build.sbt):
initialCommands in console := """
import some.library._
def someFun = println("Hello")
"""
It's not necessary to define the property separately in consoleQuick because it defaults to the one defined in console, but if you would like to use the console-project command you have to define it separately.
On a final note: remember to leave empty line between every property in an *.sbt file. They're necessary to parse the properties correctly. In the example above there are no blank lines in between so it means that everything goes into the initialCommands property (and that's what we want.)