Shell script exporting Hadoop library classpath does not working - eclipse

I am trying to develop Python client which interacts with Hadoop file system 3.3 using pyarrow package. My OS is CentOS 8 and IDE is Eclipse pydev. The sample code is simple.
from pyarrow import fs
hdfs = fs.HadoopFileSystem('localhost', 9000)
But the prerequisite command has to be executed for the successful python codes running. The command is
# export CLASSPATH=`$HADOOP_HOME/bin/hdfs classpath --glob`
Whenever I run Eclipse IDE, I have to execute the above export CLASSPATH command before Eclipse. To prevent this inconvenience and to automate the IDE configuration, I make the shell script file like below,
#! /bin/bash
export CLASSPATH=$(/usr/local/hadoop/bin/hdfs classpath --glob)
echo $CLASSPATH
When I execute this script, output messages from shell script are correct. But when I execute the Eclipse on the next line, the above pyarrow Python codes do not work successfully. I have no idea of this failure.
When I execute the export CLASSPATH command on the shell itself, the Python codes on Eclipse work correctly. But when I execute shell script, the Eclipse Python codes throws errors.
Update
I enter the [Run Configurations] menu and choose [Environment] tab. I click [Add] and input the below values onto [New Environment Variable] Dialog.
Name : CLASSPATH
Value : $(/usr/local/hadoop/bin/hdfs classpath --glob)
But my configuration is not correct, the same errors are still brought again.
I attach the image of my IDE configuration.

As mentioned in comments, in eclipe's "Run Configurations" menu, you need to add en environment variable entry for called CLASSPATH.
The value must be whatever the shell returns when you run this command:
/usr/local/hadoop/bin/hdfs classpath --glob
Don't try to put $(/usr/local/hadoop/bin/hdfs classpath --glob), it will be interpreted as a string an won't work.

Related

How does one debug a module that is run as a script (with -m flag) from within PyDev?

I have a module that is part of a library I am writing. The library has multiple executables. These are run from the command line as a script with the syntax python3 -m mypackage.mysubpackage.MyModule
I am familiar with eclipse, its debugger and PyDev, but I have not found how to set a debug/run configuration to launch MyModule within PyDev.
I have tried to add -m or -m mypackage.mysubpackage.MyModule as VM arguments to python.exe within the Arguments section of the Debug Configuration, but this does not work even setting the appropriate working directory to have mypackage in the PYTHONPATH.
I know how to launch the program outside eclipse and connect it to eclipse using a debug server. I know I can also make a little auxiliary script that is not part of the library just to launch the program so I can run it inside eclipse, but I would prefer the convenience of being able to launch the module directly within eclipse itself and I feel PyDev should be able to do this.
If you just press F9 PyDev should launch your module.
Although note it'll launch the file directly. If you require it to be launched with the -m flag (so that your relative imports resolve in that file), you can configure it in the preference:
preferences > PyDev > Run > Launch modules with python -m ...
PyDev will compute the module name based on the file location and PYTHONPATH configured in this case.
p.s.: You may want to take a look at http://www.pydev.org/manual_101_run.html for more details on re-launching your module, unit-tests, etc.

Import problem with numpy on Eclipse 2018-09

I'm have a problem using numpy with the latest Eclipse 2018-09 and PyDev 7.0.3. I'm using the latest Anaconda Python 3.7.0 which includes numpy version 1.15.1
I believe I have PyDev setup properly as I can run and debug python program that do not use numpy.
Whenever I try to "import numpy as np", I get this error:
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: DLL load failed: The specified module could not be found.
I've found a work-around by adding this single directory to the system path (not the PYTHONPATH):
C:\ProgramData\Anaconda3\Library\bin
I do this outside of Eclipse and need to restart Eclipse after changing the system path.
Hopefully, someone can explain the proper way to configure PyDev so that I don't have to pollute my system path.
Thanks!
I have had to do a half 4 or 5 installs of anaconda python in the last half year, and I think every one has been a struggle to get numpy to work. Today I can remember only the final trick that got me past that error:
This system is clean install of Windows 10, and Eclipse version 2019-12(4.14.0) Build id 20191212-1212. 64bit. In Eclipse the Python interpreter (3.7.6) is working otherwise. And Python loads numpy from a command prompt, just not within Eclipse.
In Eclipse Navigator window, right click on a PyDev project and select properties.
On the left select PyDev - Interpreter/Grammar.
On the right 'Click here to configure an interpreter not listed'. (But this will actually configure the listed interpreter.)
Then click on Open interpreter preferences page.
Now in the left tree, select Pydev / Interpreters / Python Interpreter.
On the right lower, choose the Environment Tab, On the far right lower click Add.
In the New Environment Variable window,
in Name enter PATH
in Value enter \anaconda3\Library\bin
Happyness for me! It worked without having to re-start Eclipse.
I had previously added the same path to the user's PATH in the Windows environment settings, without effect. I had earlier checked 'load conda env vars before run', and have left that checked.
In the interpreter configuration (window > preferences > PyDev > Interpreters > Python interpreter), you should be able to select the interpreter you want, go to the environment tab and set the PATH environment variable to be as you want (so, any launch with that interpreter will use the PATH you specified).
Now, I see you're using Anaconda... have you tried checking the Load conda env vars before load? to see if it fixes it for you? (that appears in the package tab when you select the interpreter).

Eclipse Pydev adding PYTHONPATH=../:. before running a pytest module

I am developing something in Python in Eclipse PyDev. Per the instructions from my assignment, I need to run this program like so:
PYTHONPATH=../:. python grade_analysis.py
However I am just running the program with the standard Run button and (no surprise) it's not working correctly. It is a unit test that claims it runs 0 tests.
Does anyone know how to run this specific file with that pythonpath in Eclipse PyDev? I've been playing with settings in the Run Configuration and no luck
You have to configure those paths to be source folders inside of PyDev (a source folder is the folder that should be in the PYTHONPATH).
See: http://www.pydev.org/manual_101_project_conf2.html for details.

How to execute Testng project without main method from command line

I have a TestNG project where I use testng.xml file to execute.
How to take a jar of my project and execute it on another machine.
I know how to execute it on the same machine by giving path of bin & lib folders i.e
cd C:\Workspace\projectname java -cp
C:\Workspace\projectname\lib*;C:\Workspace\projectname\bin
org.testng.TestNG testng.xml.
But if i want to do on another machine how to do this?
You can create a bash file which points to your testng.xml file.
Use this:
Bash file is running fine in windows for testng but it is not working in linux/mac
Steps:-
1. Create a notepad file
2. Copy -> java -cp ./src/lib/*:./bin org.testng.TestNG testng.xml(use ; instead of : if you are using windows) (./src/lib/* -> All your jars files must present in this location, ./bin -> all your class files must be present here)
3. save the file.
4. Rename the file as something.bat
5. Double click on file if you are using windows or use **bash filename** if using linux or mac
Note: - Java, require browsers must be present in other computer and it's a prerequisite which is nothing to do with your automation script run. they must be pre-installed

Setting up Pylint with PyDev

I have installed pylint via easy_install. I can run pylint <filename> with success. But pydev refuses to use it.
I checked "use pylint"
I configured correct path
I updated my python interpreter in eclipse to have pylit in pythonpath
I use Eclipse Galileo
I have build automatically checked
I tried cleaning whole project and no errors
What am I doing wrong?
I'm guessing you may need to mark the folder that contains your code as a source folder. You can do this under project properties.
Project->Properties->PyDev-PYTHONPATH
add relevant folders to the list of sources.
I've noticed that Pydev won't run Pylint on files with dashes (hyphens) the the filename; according to the Pydev devs, '-' is not a valid Python module name character, and it doesn't look like they intend to fix it:
http://sourceforge.net/tracker/index.php?func=detail&aid=1915426&group_id=85796&atid=577329
http://sourceforge.net/tracker/index.php?func=detail&aid=2888890&group_id=85796&atid=577332
As if nobody writes scripts in Python... Grumble grumble...
To provide an alternative solution, pylint can be used as an External Tool in Eclipse. This requires having previously installed the pylint package for a Python installation.
Remember to first have a pylintrc file somewhere where pylint can find it, failing which a "No config file found" error is printed. Typically I would touch pylintrc inside the project directory, and keep the file in version control. For reference, see pylint command-line options.
In Eclipse, select Run, External Tools, External Tools Configurations.... Click the toolbar button to create a New launch configuration. Configure as below or as desired:
Name: pylint
Location: ${system_path:pylint}
Working Directory: ${project_loc}
Arguments: --reports=n "${resource_loc}"
Remember to click inside an open file to switch focus to it before running the above external tool for it. Failing this, an error can occur.
The output of the external tool will by default be printed to the console.
Related: Setting up pep8 as an External Tool in Eclipse