I am using Scala IDE for working with a sbt project. I use it to work with Spark and MLlib.
I just run a simple example from the documentation. Everything seems right but the sbt output instead of reporting the log file with "debug" or "info" word in the start of each line, it uses "error" word in the start of each line:
this:
backgroung log: error: 15/04/2013 11:44:19 INFO Executor: Finished task 2.0 in stage 114.0 <TID 219> in 8 ms on localhost <1/4>
instead of this:
backgroung log: info: 15/04/2013 11:44:19 INFO Executor: Finished task 2.0 in stage 114.0 <TID 219> in 8 ms on localhost <1/4>
It's weird! I think it shouldn't use "error" because there is no error and it's just an information. What is your opinion?
Related
Im trying to build the TPCDS benchmark datasets, by following this website.
https://xuechendi.github.io/2019/07/12/Prepare-TPCDS-For-Spark
when I run this:
scala> [troberts#master1 spark-sql-perf]$ spark-shell --master yarn --deploy-mode cliers /home/troberts/spark-sql-perf/target/scala-2.11/spark-sql-perf_2.11-0.5.1-SNAPSHOT.jar -i TPCDPreparation.scala
I get this error? Im wondering if its something to do with permissions as the file dsdgen definitely exists at that location on each of the worker nodes /home/troberts/spark-sql-perf/tpcds-kit/tools
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Aborting TaskSet 0.0 because task 0 (partition 0)
cannot run anywhere due to node and executor blacklist.
Most recent failure:
Lost task 0.0 in stage 0.0 (TID 0, worker1.mycluster.com, executor 1): java.lang.RuntimeException: Could not find dsdgen at /home/troberts/spark-sql-perf/tpcds-kit/tools/dsdgen or //home/troberts/spark-sql-perf/tpcds-kit/tools/dsdgen. Run install
at scala.sys.package$.error(package.scala:27)
Any ideas appreciated.
Cheers
Could not find dsdgen at /home/troberts/spark-sql-perf/tpcds-kit/tools/dsdgen or //home/troberts/spark-sql-perf/tpcds-kit/tools/dsdgen
You need to have TPCDS installed first.
spark-sql-perf docs from tool you've used:
Before running any query, a dataset needs to be setup by creating a Benchmark object.
Generating the TPCDS data requires dsdgen built and available on the machines.
We have a fork of dsdgen that you will need.
The fork includes changes to generate TPCDS data to stdout, so that this library can pipe them directly to Spark, without intermediate files.
Therefore, this library will not work with the vanilla TPCDS kit.
TPCDS kit needs to be installed on all cluster executor nodes under the same path!
Please, configure TPCDC toolkit from databricks
I integrated quartz 2 and spring 4 with maven and java annotation ( using servlet 3 ), also i am using tomcat 7 maven plugin for deploying my project,my quartz Configuration class like as below :
and my job class define simply like as below :
then i use the quartz Scheduler for using fire my job trigger immediately as below :
but my problem is : when i call fireNow methode with "job1" , "mygroup" parameters nothing happens and my job1 do not call immediately and don't print anything in console, i also track the db tables an i noticed
after running the fireNow method new row inserted in my qrtz_triggers table in mysql:
If Quartz scheduler is not set to start automatically. You need to start it explicitly.
scheduler.start();
If Quartz scheduler started successful, you should see information in your log or console output similar as below.
[main] INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.2.1)'org.springframework.scheduling.quartz.SchedulerFactoryBean#0' with instanceId 'MyScheduler'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
...
[main] INFO org.quartz.core.QuartzScheduler - started
Finally I found solution for my problem, after enabling quartz log4j (adding log4j.logger.org.quartz=DEBUG in my log4j.properties ), I saw the jdbc exception in console, the exception related to using outdated quartz-query.
I added quartz 2.2.1 dependency in my POM but I used quartz sql query for 2.1.7 version and that mismatched between quartz jar and quartz sql query version cause missing some table like SCHED_TIME.
Trying to use sbt with
Keys.fork := true
With this option all messages from slf4j logger shown as error-message
It looks like
[error] 0 [main] INFO test - Test
Without fork it looks like
1 [run-main] INFO test - Test
sbt version: 0.13
This is documented at http://www.scala-sbt.org/0.13.2/docs/Detailed-Topics/Forking.html , in the “Configuring output” section:
By default, forked output is sent to the Logger, with standard output logged at the Info level and standard error at the Error level. This can be configured with the outputStrategy setting,
This problem seems to be already raised in Stackoverflow, but my case is quite different, file or folder location hadoop looking for is created in C:/tmp/hadoop-SYSTEM/mapred/local/taskTracker/jobcache/, in this location job folder are created while run the wordcount example, but even the files and folder are avalilable, its throwing the file not found exception, it seems like files not been identified, i even tried the re-formating of namenode which is one of the solution provided in forums,but still problem exist
Note: Hadoop version 0.20.2
ERROR:
13/04/11 10:24:20 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
13/04/11 10:24:21 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/04/11 10:24:21 INFO input.FileInputFormat: Total input paths to process : 1
13/04/11 10:24:22 INFO mapred.JobClient: Running job: job_201304111023_0001
13/04/11 10:24:23 INFO mapred.JobClient: map 0% reduce 0%
13/04/11 10:24:34 INFO mapred.JobClient: Task Id : attempt_201304111023_0001_m_000002_0, Status : FAILED
java.io.FileNotFoundException: File C:/tmp/hadoop-SYSTEM/mapred/local/taskTracker/jobcache/job_201304111023_0001/attempt_201304111023_0001_m_000002_0/work/tmp does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:519)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
Check if the permissions to that folder have been set properly, this type of error may occur if write permissions are not given to that folder.
After struggling to get proper testsuites, I'm now pretty disappointed by the fact that , while following as close as possible this tutorial (pretty straightforward, right ?) Setting up Selenium server on a headless Jenkins CI build machine, Jenkins keeps looping on the current build, outputting :
So I decided to run a selenium build by hand on the ci machine, and got this :
user#machine:/var/log$ export DISPLAY=":99" && java -jar /var/lib/selenium/selenium- server.jar -browserSessionReuse -htmlSuite *firefox http://staging.site.com /var/lib/jenkins/jobs/project/workspace/tests/selenium/testsuite.html /var/lib/jenkins/jobs/project/workspace/logs/selenium.html
24 janv. 2012 19:27:56 org.openqa.grid.selenium.GridLauncher main
INFO: Launching a standalone server
19:27:59.927 INFO - Java: Sun Microsystems Inc. 20.0-b11
19:27:59.929 INFO - OS: Linux 3.0.0-14-generic amd64
19:27:59.951 INFO - v2.17.0, with Core v2.17.0. Built from revision 15540
19:27:59.958 INFO - Will recycle browser sessions when possible.
19:28:00.143 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4444/wd/hub
19:28:00.144 INFO - Version Jetty/5.1.x
19:28:00.145 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
19:28:00.147 INFO - Started HttpContext[/selenium-server,/selenium-server]
19:28:00.147 INFO - Started HttpContext[/,/]
19:28:00.183 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler#16ba8602
19:28:00.184 INFO - Started HttpContext[/wd,/wd]
19:28:00.199 INFO - Started SocketListener on 0.0.0.0:4444
19:28:00.199 INFO - Started org.openqa.jetty.jetty.Server#6f7a29a1
HTML suite exception seen:
java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:900)
at org.openqa.selenium.server.SeleniumServer.runHtmlSuite(SeleniumServer.java:603)
at org.openqa.selenium.server.SeleniumServer.boot(SeleniumServer.java:287)
at org.openqa.selenium.server.SeleniumServer.main(SeleniumServer.java:245)
at org.openqa.grid.selenium.GridLauncher.main(GridLauncher.java:54)
19:28:00.218 INFO - Shutting down...
19:28:00.220 INFO - Stopping Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=4444]
While understanding the output is'nt that hard, finding what to do to remove this issue is.
Any chance you guys already have been facing that kind of stuff ? Thanks
I only just got past these problems myself, but I was able to run your command when I pointed it at my .jar, testSuite and report file. I'm thinking that perhaps the location of your files under,
/var/lib/selenium
could be part of the problem. Try putting them where your user has permission perhaps under
/home/USERNAME/selenium
Other than that the only thing I can say is make sure your .jar, testSuite and report file are valid.
Also (I assume this is an error of copy and paste into stack overflow) but, this part of your command is incorrect
/var/lib/selenium/selenium- server.jar
You are not getting the error I would expect from an incorrect jar location so I assume something was lost when you pasted to stackoverflow.