unable to start a Process in Scala/Play application - scala

I have a ccm.py script which runs Cassandra cluster on local machine. I am able to run the command using windows command prompt. But I get error if I try to do so using Process class in Scala.
I want to run it before starting my test cases. So I am calling Process in beforeAll.
override def beforeAll():Unit = {
println("starting cassandra cluster locally")
val ccmCommand = Process("ccm.py start").!
}
The error I get is
An exception or error caused a run to abort: Cannot run program "ccm.py": CreateProcess error=193, %1 is not a valid Win32 application
java.io.IOException: Cannot run program "ccm.py": CreateProcess error=193, %1 is not a valid Win32 application
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
I also tried using cassandra -f command (which also works from cmd prompt) but I got error
An exception or error caused a run to abort: Cannot run program "cassandra": CreateProcess error=2, The system cannot find the file specified
java.io.IOException: Cannot run program "cassandra": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
How can I solve the issue?
Update
Thanks to Alex Ott, I have made some progress but I think there has to be a better and more reliable way to test the code and also make the code more portable
The code which seem to work so far is
override def beforeAll():Unit = {
println("starting cassandra cluster locally")
val ccmCommand = Process("python",Seq("C:\\Users\\manu\\Documents\\manu\\ccm-3.1.4.tar\\dist\\ccm-3.1.4\\ccm.py","start")).run //NOT GOOD. Using hardcoded path!!
println(s"ccm returned ${ccmCommand}")
Thread.sleep(30000) //the wait is too long and there is no guarantee that the cluster will be up properly
}
Is there a better (w.r.t reliable without hard coding the path) to bring up the cluster before tests get executed.

Related

Eclipse / PyDev Stop button not working with OSError: [WinError 6] The handle is invalid error

I have the following Eclipse version on Windows 10:
Version: 2020-09 (4.17.0)
Build id: 20200910-1200
I am using PyDev along with it.
In my code I am using selenium to make a number of url calls (web scraping). When it happens that a particular url is not present or at least not present in the way most of the urls I am reading are, I get the following error:
Traceback (most recent call last):
File "C:\Users\foobar\eclipse-workspace\WeatherUndergroundUnderground\historical\BWI_Fetch.py", line 44, in <module>
main(city, month_date, start_year, end_year)
File "C:\Users\foobar\eclipse-workspace\WeatherUndergroundUnderground\historical\BWI_Fetch.py", line 22, in main
driver.get(city_url);
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror
Exception ignored in: <function Popen.__del__ at 0x0000019267429F70>
Traceback (most recent call last):
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 945, in __del__
self._internal_poll(_deadstate=_maxsize)
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1344, in _internal_poll
if _WaitForSingleObject(self._handle, 0) == _WAIT_OBJECT_0:
OSError: [WinError 6] The handle is invalid
When I get this particular error, eclipse is still running and pushing the red stop button does not work to end the program. I can usually use the red stop button for just about any other python program I have written, but this code/error seems to hang things. How can I end the process from within the Eclipse application?
The error in the stack trace is not really related to PyDev, so, the stack trace error is only really fixable in Selenium/Python (the error says that it's trying to access a process which is already dead on the __del__).
Now, related to the reason why PyDev wasn't able to kill it, I think that you probably have some process which spawned a subprocess and is not reachable anymore because the parent process died and thus it's not possible to create a tree to kill that process from the initial process launched in PyDev.
The actual code which does this in PyDev is: https://github.com/fabioz/winp/blob/master/native/winp.cpp#L208
I think that it should be possible to use the windows api to create a JobObject and then AssignProcessToJobObject and on kill also kill the JobObject so that it kills all associated processes so that things are setup in a way that that this doesn't happen, but this isn't currently done.
As a note, usually I have an alias for: taskkill /im python.exe /f (which will kill all the python.exe processes running in the machine) and it's what I usually use in such cases, so, if something like that happens I just kill all the python.exe processes in the machine.
Although note that if you spawned some other process... say, chrome.exe -- in that process tree, that process must also be killed for the initial shell that launched python to be really disposed.
This error message...
Exception ignored in: <function Popen.__del__ at 0x0000019267429F70>
...implies that the builtins module was destroyed before running __del__ in process of garbage collecting.
Hence PyDev is no more able to communicate with the relevant python modules. As a result Stop button isn't functioning and raises the error:
OSError: [WinError 6] The handle is invalid

How to fix missing simulink simulation artificats issue when running test in parallel mode?

I have 29 Simulink/Matlab Test. It has a lot of different reference models. Before running a 20 second simulation , it has to load all reference models and create a lot of simulation artifacts in a work folder. A lot of reference model are shared in-between test.
When running one test at a time, I have no issue, all simulation artifact are created and used to run the various simulation. Everything Passes.
When running it all via parallel processing. I have a issue.Some simulation artifact are not built or missing, hence my simulation fails even before running.But surprisingly, not all 29 of them fail. It actually random,last time it was 17, another time it was 22. And it even ran once with 0 fail.
Another note, I only have this issue when running it on a self-hosted computer on Azure-Pipelines for CI purposes.
I would like to fix this issue and reproduce stable test pass/fail results of one at a time run, but on parallel process run. How would I do that?
Error:
2020-11-03T03:16:27.1083996Z Making simulation target "Foo_src_sfun", ...
2020-11-03T03:16:27.1084227Z
2020-11-03T03:16:27.1084361Z
2020-11-03T03:16:27.1084502Z
2020-11-03T03:16:27.1084789Z Microsoft (R) Program Maintenance Utility Version 14.00.24210.0
2020-11-03T03:16:27.1085188Z Copyright (C) Microsoft Corporation. All rights reserved.
2020-11-03T03:16:27.1085441Z
2020-11-03T03:16:27.1085815Z NMAKE : fatal error U1052: file 'Foo_src_sfun.mak' not found
2020-11-03T03:16:27.1086175Z Stop.
2020-11-03T03:16:27.1089399Z ================================================================================
2020-11-03T03:16:27.1089936Z Error occurred in TestSim/testSim(File=test_FooTest1_slx) and it did not run to completion.
2020-11-03T03:16:27.1090308Z
2020-11-03T03:16:27.1090497Z ---------
2020-11-03T03:16:27.1090720Z Error ID:
2020-11-03T03:16:27.1090946Z ---------
2020-11-03T03:16:27.1091254Z 'Slvnv:simcoverage:SimulationFailed'
2020-11-03T03:16:27.1091481Z
2020-11-03T03:16:27.1091669Z --------------
2020-11-03T03:16:27.1091919Z Error Details:
2020-11-03T03:16:27.1092186Z --------------
2020-11-03T03:16:27.1092419Z Error using cvsim
2020-11-03T03:16:27.1092659Z Simulation failed
2020-11-03T03:16:27.1092864Z
2020-11-03T03:16:27.1093112Z Error in testRunner (line 145)
2020-11-03T03:16:27.1093477Z [cvdo, simOutRes] = cvsim(testObj,paramStruct) ;
2020-11-03T03:16:27.1093765Z
2020-11-03T03:16:27.1094034Z Error in TestSim/testSim (line 30)
2020-11-03T03:16:27.1094373Z [cvdo, simOutRes, ErrLog] = testRunner(File,20);
2020-11-03T03:16:27.1094638Z
2020-11-03T03:16:27.1094830Z Caused by:
2020-11-03T03:16:27.1095168Z Error using autobuild_kernel>autobuild_local (line 219)
2020-11-03T03:16:27.1095612Z Unable to create mex function 'Foo_src_sfun.mexw64'
2020-11-03T03:16:27.1096006Z required for simulation.
2020-11-03T03:16:27.1096427Z ================================================================================
Update:
I found that I have also another kind of error, leads pretty much to same result.
2020-11-03T03:18:36.1668328Z Making simulation target "Foo2_src_sfun", ...
2020-11-03T03:18:36.1668601Z
2020-11-03T03:18:36.1668735Z
2020-11-03T03:18:36.1669087Z 'Foo2_src_sfun.bat' is not recognized as an internal or external command,
2020-11-03T03:18:36.1669483Z operable program or batch file.
2020-11-03T03:18:36.1669685Z
2020-11-03T03:18:36.1669892Z >>Removing MiL paths...
2020-11-03T03:18:36.1670104Z >>Done
I made a runSingleTest() that I run before my parallel run. Before running it creates all required model reference mexw64 files in the **/work/sim_artifact folder.
Hence when the parallel run they don't need to create any new files, they either use whats already there or update the files.
I have been having no issue since that change. Just a longer run time because of that repetitive test.

How to solve error on docker:layers_calculator to compute the Merkle tree on private tangle?

I want to setup a private tangle on my own virtual machine with Ubuntu 18.04, 4GB RAM and 20GB memory.
I have follow this instructions: https://docs.iota.org/docs/compass/0.1/how-to-guides/set-up-a-private-tangle. Every command works fine until reach this one: bazel run //docker:layers_calculator.
It shows an error as follows:
Starting local Bazel server and connecting to it...
ERROR: /home/istabraq/compass/third-party/maven_deps.bzl:3:5: Traceback (most recent call last):
File "/home/istabraq/compass/WORKSPACE", line 42
maven_jars()
File "/home/istabraq/compass/third-party/maven_deps.bzl", line 3, in maven_jars
native.maven_jar(<4 more arguments>)
type 'struct' has no method maven_jar()
ERROR: error loading package '': Encountered error while reading extension file 'protobuf_deps.bzl': no such package '#com_google_protobuf_deps//': error loading package 'external': Could not load //external package
ERROR: error loading package '': Encountered error while reading extension file 'protobuf_deps.bzl': no such package '#com_google_protobuf_deps//': error loading package 'external': Could not load //external package
INFO: Elapsed time: 4.743s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
How can I solve this problem? what I have missed?
read carefully the message given after running bazel installer:
Make sure you have "/home/yourusername/bin" in your path. You can also activate bash completion by adding the following line to your :
source /home/yourusername/.bazel/bin/bazel-complete.bash
You can check with: "bazel info" or "bazel version"
Unfortunately, there are further errors:
https://github.com/iotaledger/compass/issues/142
I have solve this issue by using this commands:
Step 3: Set up your environment
If you ran the Bazel installer with the --user flag as above, the Bazel executable is installed in your $HOME/bin directory. It’s a good idea to add this directory to your default paths, as follows:
export PATH="$PATH:$HOME/bin"
You can also add this command to your ~/.bashrc or ~/.zshrc file to make it permanent.
reference:
https://docs.bazel.build/versions/master/install-ubuntu.html

Spark Pipe example

I'm new to Spark and trying to figure out how the pipe method works. I have the following code in Scala
sc.textFile(hdfsLocation).pipe("preprocess.py").saveAsTextFile(hdfsPreprocessedLocation)
The values hdfsLocation and hdfsPreprocessedLocation are fine. As proof, the following code works from the command line
hadoop fs -cat hdfsLocation/* | ./preprocess.py | head
When I run the above Spark code I get the following errors
14/11/25 09:41:50 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.IOException: Cannot run program "preprocess.py": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at org.apache.spark.rdd.PipedRDD.compute(PipedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
... 12 more
In order to solve this for Hadoop streaming I would just use the --files attribute, so I tried the same thing for Spark. I start Spark with the following command
bin/spark-shell --files ./preprocess.py
but that gave the same error.
I couldn't find a good example of using Spark with an external process via pipe, so I'm not sure if I'm doing this correctly. Any help would be greatly appreciated.
Thanks
I'm not sure if this is the correct answer, so I won't finalize this, but it appears that the file paths are different when running spark in local and cluster mode. When running spark without --master the paths to the pipe command are relative to the local machine. When running spark with --master the paths to the pipe command are ./
UPDATE:
This actually isn't correct. I was using SparkFiles.get() to get the file name. It turns out that when calling .pipe() on an RDD the command string is evaluated on the driver and then passed to the worker. Because of this SparkFiles.get() is not the appropriate way to get the file name. The file name should be ./ because SparkContext.addFile() should put that file on ./ relative to to where each worker is run from. But I'm so sour on .pipe now that I've take .pipe out of my code in total in favor of .mapPartitions in combination of a PipeUtils object that I wrote here. This is actually more efficient because I only have to incur the script startup costs once per partition instead of once per example.

Chain psexec and nunit is not working (psexec exit with code 1 and nunit is crashed)

Sorry for unclear subject but i really don't understand where problem.
I use a system for autotests c#+webdriver for chrome + nunit (2.6) + cc.net ( 1.6.7981.1)
And my tests is run on remote machine (Windows XP SP 2 x86)via psexec which start bat file on remote machine
bat file includes 2 commands:
run msbuild
run nunit-console
Well, psexec has next arguments
/accepteula \MachineName -u Login -p Pass -s Path\start.bat
Bat file has next command
nunit-console-x86.exe "%PROJECT_FOLDER%\Path\Tests.dll" "/xml=%TEST_RESULT_PATH%"
When cc.net launch psexec tests is started and the tests work during some time. Unexpected, psexec exit with code 1
message level="Error">Path\start.bat exited on MachineName with error code 1. message
But nunit-console on remote machine is continue working
And after some time console is crashed with error
szAppName : nunit-console-x86.exe szAppVer : 2.6.0.12051
szModName : kernel32.dll szModVer : 5.1.2600.2180 offset :
0001eb33
Message from Event log
Application: nunit-console-x86.exe Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.IOException Stack: at
System.IO._Error.WinIOError(Int32, System.String) at
System.IO._ConsoleStream.Write(Byte[], Int32, Int32) at
System.IO.StreamWriter.Flush(Boolean, Boolean) at
System.IO.StreamWriter.Write(Char[], Int32, Int32) at
System.IO.TextWriter.WriteLine(System.String) at
System.IO.TextWriter.WriteLine(System.String, System.Object) at
System.IO.TextWriter+SyncTextWriter.WriteLine(System.String,
System.Object) at System.Console.WriteLine(System.String,
System.Object) at NUnit.ConsoleRunner.Runner.Main(System.String[])
at NUnit.ConsoleRunner.Class1.Main(System.String[])
If run nunit locally than no problem(nunit-console works fun).
Some time ago system worked, and i don't know what is changed (This is most bad case)
How can i resolve this problem?
Thanks!
This problem is resolved!
Now system is working