while running kafka on windows.
C:\Program Files\kafka_2.12-2.1.0>.\bin\windows\kafka-server-start.bat .\config\server.properties
And getting the error
The system cannot find the path specified.
The syntax of the command is incorrect.
Error: Could not find or load main class Files\kafka_2.12-2.1.0.logs
You cannot have spaces in the file path, e.g Program Files
There's no specific reason Kafka needs to be in your Program Files folder. You could move it to C:\kafka for example, and I've been able to run it on Windows 10 (out of my users folder), so it does work
Related
My OS is Windows 11 and Apache Spark version is spark-3.1.3-bin-hadoop3.2
I try to use Spark structured streaming with pyspark. Belows are my simple spark structured streaming codes.
spark = SparkSession.builder.master("local[*]").appName(appName).getOrCreate()
spark.sparkContext.setCheckpointDir("/C:/tmp")
The same Spark codes without spark.sparkContext.setCheckpointDir line throws no errors on Ubuntu 22.04. However the above codes do not work successfully on Windows 11. The exemptions are
pyspark.sql.utils.IllegalArgumentException: Pathname /C:/tmp/67b1f386-1e71-4407-9713-fa749059191f from C:/tmp/67b1f386-1e71-4407-9713-fa749059191f is not a valid DFS filename.
I think the error codes mean checkpoint directory are generated on Hadoop file system of Linux, not on Windows 11. My operating system is Windows and checkpoint directory should be Windows 11 local directory. How can I configure Apache Spark checkpoint with Windows 11 local directory? I used file:///C:/temp and hdfs://C:/temp URL for test. But the errors are still thrown.
Update
I set below line to be comments.
#spark.sparkContext.setCheckpointDir("/C:/tmp")
Then the exceptions are thrown.
WARN streaming.StreamingQueryManager: Temporary checkpoint location created which is deleted normally when the query didn't fail: C:\Users\joseph\AppData\Local\Temp\temporary-be4f3586-d56a-4830-986a-78124ab5ee74. If it's required to delete it under any circumstances, please set spark.sql.streaming.forceDeleteTempCheckpointLocation to true. Important to know deleting temp checkpoint folder is best effort.
pyspark.sql.utils.IllegalArgumentException: Pathname /C:/Users/joseph/AppData/Local/Temp/temporary-be4f3586-d56a-4830-986a-78124ab5ee74 from hdfs://localhost:9000/C:/Users/joseph/AppData/Local/Temp/temporary-be4f3586-d56a-4830-986a-78124ab5ee74 is not a valid DFS filename.
I wonder why hdfs url contains c:/ driver letters and I want to know how to set spark.sql.streaming.forceDeleteTempCheckpointLocation to true.
step 1)
Since you are running spark from a windows machine, make sure winutils.exe file added in hadoop bin folder reference link for same (6th Step) https://phoenixnap.com/kb/install-spark-on-windows-10.
step 2)
then try to add like this
spark.sparkContext.setCheckpointDir("D:\Learn\Checkpoint")
spark.sparkContext.setCheckpointDir("D:\Learn\Checkpoint")
Make sure spark user does have the permission to write in mentioned checkpoint directory
When i try to run the same code in client mode, the code runs successfully. But when run on cluster mode it fails to create the file and prompts with error No such File or Directory.
Below is the code sample:
new File("UnexistingLocation").mkdirs()
---> Directories created in client mode
---> Code do not give error in cluster mode but i cannot see directory created. Also while creating File inside the directory gives error No such file or Directory.
Is there a workaround by which i could create files on driver node local filesystem?
due to your error analysis I assume this is run in driver-scoped code. If you submit using --deploy-mode cluster your driver will be started on an arbitrary node which means that's where your directory will be. It won't be on the node where you do your spark-submit from
My source is an old external hard drive formatted HFS+ (used to be data from server running 10.4.11) connected to an iMac running 10.8.5 with an upgraded version of rsync 3.0.9.
The destination is a Centos 6.4 server running rsync 3.0.9 as well.
We have tried to transfer a FONTS folder (Source size = 4.7GB) to the destination but the size of the folder is not kept (Destination size = 655MB).
Below is the command that I run to preserve hard links - ACLs.....
/usr/local/bin/rsync -aHpEXotg --devices --specials --ignore-errors /Users/london/Desktop/FONTS root#192.168.200.253:/home/TEST
Also getting errors: rsync: rsync_xal_set: lsetxattr(""/home/TEST/FONTS/ Folder/Kfz-EURO Schrift MAC+PC/MAC/FE Mittelschrift.image"","user.com.apple.FinderInfo") failed: Operation not supported (95)
Most of the files are showing as Unix files and can't be open.
This issue has been time consuming so if someone can guide me.
Thanks..
Ran across this today as I encountered similar errors. I ended up running rsync with minimum options to complete the copy:
rsync -r --progress /path/to/source /path/to/destination
-r is recursive
--progress shows additional copy info (versus v for verbose output)
If you leave out --progress, rsync will only show you files that error and will transfer the rest - that can be useful to know which files you're not getting if there aren't very many with errors. Course, alternatively if there are a lot of errors, that can indicate bad sectors on the drive.
I was trying to start the ATG publishing servers on JBoss. From my JBoss bin directory, I used the following command .\run.bat -c ATGPublishing -b 0.0.0.0.I was getting the following error. I am using Oracle 11g. Any idea why this error is coming up??
2013-07-18 10:22:43,025 ERROR [nucleusNamespace.atg.dynamo.messaging.SqlJmsProvider] (/atg/dynamo/service/Scheduler-reusablejobhandler-PATCHBAY-RESTART) could not create client ID for client name Admin-VAIO:8850 : DB failure or maybe the client name already exists in the DB
com this side most of you r confusing how to install local dynamo DB here is solutin....
1).download java and install this will goes your c drive program file anyway
2).now download dynaolocalDB and put in that directory where you want to work
3).unzip it
4).now just know where java install i mean path like C:\Program Files (x86)\Java\jre7\bin
5).open cmd (comand prompt) and set java path like set path="C:\Program Files (x86)\Java\jre7\bin"
6).now go that dir where you put your dbdynamoDb like C:\developer\node_modules\npm\dbdynamoDb
copy and past this
$ java –Djava.library.path=. -jar DynamoDBLocal.jar
if error then
java -jar DynamoDBLocal.jar
still error then specify port
java -jar DynamoDBLocal.jar -- port8050
i have a tomcat6 server running on a CentOS 6 machine and so far so good.
in one of my webapps i need to use a context param to access an external folder located in the filesystem, i configured my server.xml like this (relevant portion of <Host> tag only) :
<Context path="/userimages" docBase="/home/someuser/faces/32x32" debug="0" reloadable="true" crossContext="true"/>
when i start the server i get this error :
java.lang.IllegalArgumentException: Document base /home/someuser/faces/32x32 does not exist or is not a readable directory
i read something about folder permission so i set both "32x32" and "webapps" folder to 777, but it's still not working...any idea of how to fix this ?
P.S. on windows OS it works perfectly
My suggestion is to put your data into /usr/share/tomcat6/conf/context.xml which is a symlink to /etc/tomcat6/context.xml on CentOS 6. At least tomcat6 does read the contents of that file when it restarts, and I had some luck getting resource data loaded from there. It would seem that this file is new in tomcat6.
I used strace to check which files it was visiting and it does run stat() on the various files like /var/lib/tomcat6/webapps/*/META-INF/context.xml but nowhere does it actually open() those files, so I'm pretty sure it does not read the contents. Maybe some bug? Maybe imaginary future feature?
I managed to get Plandora (uses context to supply MySQL database connection details) running on CentOS 6 with these packages (from yum):
apache-tomcat-apis-0.1-1.el6.noarch
java-1.6.0-openjdk-1.6.0.0-1.61.1.11.11.el6_4.i686
mysql-connector-java-5.1.17-6.el6.noarch
tomcat6-6.0.24-52.el6_4.noarch
tomcat6-servlet-2.5-api-6.0.24-52.el6_4.noarch
tomcat6-el-2.1-api-6.0.24-52.el6_4.noarch
tomcat6-admin-webapps-6.0.24-52.el6_4.noarch
tomcat6-jsp-2.1-api-6.0.24-52.el6_4.noarch
tomcat6-lib-6.0.24-52.el6_4.noarch
tomcat6-webapps-6.0.24-52.el6_4.noarch
Just in case anyone else is trying to get Plandora to work on CentOS 6, you also need to make sure you symlink:
ln -s /usr/share/java/mysql-connector-java.jar /usr/share/tomcat6/lib/