Pydev tags import as “unresolved import” all compiled extensions - eclipse

PyDev with Python3.5 seems unable to recognize imports from c-compiled extensions, including packages compiled via Cython.
I am working on an up-to-date debian/stretch machine with a stripped-down self-installed (in home dir) Eclipse/Neon with PyDev added via update site, if it matters.
I have both "Python 2.7.13" and "Python 3.5.2+" installed.
One of the problematic packages is lxml. I installed the debian packages and also tried installing manually via pip (and re-created the interpreter in eclipse afterwards to ensure full finding). In all cases packages do work.
In Python2 everything works as advertised.
Under Python3 PyDev flags from lxml import etree as error (but resulting program works):
Unresolved import: etree richiedi_certificato_dispositivo.py /trasmissione-telematica/Serializzazione line 8 PyDev Problem
Note: I can import lxml with no error, but then any access to lxml.etree... will be flagged in error. Data completion is consistent (i.e.: etree won't be in the offered list).
lxml is installed in the usual location:
mcon#vocore:~$ ls -l /usr/lib/python3/dist-packages/lxml
total 2156
-rw-r--r-- 1 root root 8152 Sep 5 2014 builder.py
-rw-r--r-- 1 root root 3366 May 5 2016 cssselect.py
-rw-r--r-- 1 root root 18387 May 5 2016 doctestcompare.py
-rw-r--r-- 1 root root 7641 Sep 25 2011 ElementInclude.py
-rw-r--r-- 1 root root 9490 Aug 20 06:48 _elementpath.py
-rw-r--r-- 1 root root 1710088 Aug 24 10:14 etree.cpython-35m-x86_64-linux-gnu.so
drwxr-xr-x 3 root root 4096 Jan 3 08:58 html
drwxr-xr-x 3 root root 4096 Jan 3 08:58 includes
-rw-r--r-- 1 root root 551 Oct 7 2012 __init__.py
drwxr-xr-x 4 root root 4096 Jan 3 08:58 isoschematron
-rw-r--r-- 1 root root 17450 Aug 20 06:48 lxml.etree_api.h
-rw-r--r-- 1 root root 8902 Aug 20 06:48 lxml.etree.h
-rw-r--r-- 1 root root 366440 Aug 24 10:14 objectify.cpython-35m-x86_64-linux-gnu.so
drwxr-xr-x 2 root root 4096 Jan 3 08:58 __pycache__
-rw-r--r-- 1 root root 92 Sep 5 2014 pyclasslookup.py
-rw-r--r-- 1 root root 8531 Nov 20 2014 sax.py
-rw-r--r-- 1 root root 230 Sep 25 2011 usedoctest.py
mcon#vocore:~/trasmissione-telematica$
As You can see etree is in a shared lib, as is objectify; a quick check shows also objectify is not handled by PyDev. I checked a few other "c-extension" packages (e.g.: import pycurl and from Crypto.Util import strxor) with the same result, so it seems a problem with "C" extensions.

Have you tried to put the offending packages in the forced builtins (as described in http://www.pydev.org/manual_101_interpreter.html)?
If that doesn't work, do you have some error in your error log? (see http://www.pydev.org/faq.html#PyDevFAQ-HowdoIReportaBUG%3F for details on getting it)

Related

Pyenv-installed Python creates virtualenv with empty include dir, no python-config in bin

Note: There are several similar questions, which I have seen and read. None of them are the precise problem I'm having, and none of their answers work for me.
I have installed several Python versions (Python 2.7, 3.8, 3.9, and 3.10) on my macOS Ventura system using Pyenv. The installed directories include a python-config file in /bin and an /include directory with many header files, so I do not need to install python-devel or similar:
$ ls -al /Users/williamsn/.pyenv/versions/3.10.9/bin/*-config
lrwxr-xr-x 1 williamsn staff 17 Dec 19 17:24
/Users/williamsn/.pyenv/versions/3.10.9/bin/python-config -> python3.10-config
lrwxr-xr-x 1 williamsn staff 17 Dec 19 17:24
/Users/williamsn/.pyenv/versions/3.10.9/bin/python3-config -> python3.10-config
-rwxr-xr-x 1 williamsn staff 2073 Dec 19 17:24
/Users/williamsn/.pyenv/versions/3.10.9/bin/python3.10-config
$ ls -al /Users/williamsn/.pyenv/versions/3.10.9/include/python3.10/
total 1192
drwxr-xr-x 86 williamsn staff 2752 Dec 19 17:24 .
drwxr-xr-x 3 williamsn staff 96 Dec 19 17:24 ..
-rw-r--r-- 1 williamsn staff 3224 Dec 19 17:24 Python.h
...
-rw-r--r-- 1 williamsn staff 3026 Dec 19 17:24 import.h
...
-rw-r--r-- 1 williamsn staff 48878 Dec 19 17:24 pyconfig.h
...
-rw-r--r-- 1 williamsn staff 2863 Dec 19 17:24 weakrefobject.h
As a best practice, I work exclusively in virtualenvs to keep the base Python installation clean and pristine. I create my virtualenvs using the venv module built in to Python 3:
$ python3.10 -m venv my_venv
After doing so, there are two problems:
The my_venv/include directory is empty (does not contain the python3.10 directory or any of the header files from the parent Python and does not symlink to the parent Python)
There is no python-config / python3-config / python3.10-config in my_venv/bin.
So, for things requiring the Python headers:
If it tries to find the headers using python-config (which many projects do), it will fail.
If it tries to find the headers using the Python prefix and adding /include to it (which many projects do, notably Boost), it will fail.
If it tries to find the headers by importing sysconfig and calling sysconfig.get_paths(), it will succeed, albeit with paths outside the virtualenv.
Now, on a case-by-case basis, I can work around this, and already have. I can manually copy over python_config and then either export CPATH to add /Users/williamsn/.pyenv/versions/3.10.9/include/python3.10 or modify the virtualenv to symlink to /Users/williamsn/.pyenv/versions/3.10.9/include. These work, but they don't seem right to me. I can't make a global workaround anywhere (such as exporting CPATH in my Bash profile), because I'm working with multiple Python versions, and I'd end up with the wrong headers half the time. It seems to me like a virtualenv that has an empty include directory and is missing python-config when its parent has a full include directory and contains python-config is a broken virtualenv.
Is there an option I'm missing to include these pieces? Is this a bug I need to file against Python/venv?

Serialization error due to different folder structure on local and server machine

I am getting following error in my server tomcat logs:-
ERROR: The serialization policy file '/nos/js/4937711D6147993DCFE47FF203A0FEDC.gwt.rpc' was not found; did you forget to include it in this deployment?
On my local machine the above path is present and tomcat is able to locate it and hence I don't get this error. All the folders are present inside nos folder.
But on server, all the folders are present inside ROOT and nos folder is not present and hence server tomcat is not able to identify this path
I am providing folder structure on my local machine and server below :-
Windows folder structure
Directory: C:\Users\abc.metadata.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\nos
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 06-09-2022 09:07 PM classes
d----- 06-09-2022 09:09 PM com.xyz.nos.Common
d----- 06-09-2022 09:07 PM gxt
d----- 06-09-2022 09:10 PM js
d----- 06-09-2022 09:07 PM maven-status
d----- 07-09-2022 05:04 PM META-INF
d----- 06-09-2022 09:07 PM WEB-INF
Linux server folder structure
Directory
[x_qa#webapp-nos-69d5d4865-mzkxz ROOT]$ pwd
/opt/tomcat/webapps/ROOT
[x_qa#webapp-nos-69d5d4865-mzkxz webapps]$ cd ROOT
[x_qa#webapp-nos-69d5d4865-mzkxz ROOT]$ ls -lrt
total 36
drwxr-xr-x 4 root root 4096 Jan 1 1970 nos-10.0.8-SNAPSHOT
drwxr-xr-x 3 root root 4096 Jan 1 1970 maven-status
drwxr-xr-x 7 root root 4096 Jan 1 1970 js
-rw-r--r-- 1 root root 182 Jan 1 1970 index.html
drwxr-xr-x 8 root root 4096 Jan 1 1970 gxt
drwxr-xr-x 2 root root 4096 Jan 1 1970 gwt-unitCache
drwxr-xr-x 3 root root 4096 Jan 1 1970 classes
drwxr-xr-x 1 root root 4096 Jan 1 1970 WEB-INF
drwxr-xr-x 3 root root 4096 Jan 1 1970 META-INF
Can anyone suggest how can I make consistent folder structure between server and local machine by keeping both of these folders in sync ?
Let me know if my understand is incorrect ?

How to really see the contents of the different perl scripts linking to one file?

I wanted to view the contents of a perl script in our environment which is called dfv_run.pl and specifically check line 245. Line 245 from that script printed the message "Finished checking test result" in my simulation log file. After executing % which dfv_run.pl I am pointed to this location:
drwxr-xr-x 2 dfvmgr dfvadmin 4.0K Jun 22 2017 .SYNC
-r--r--r-- 1 dfvmgr dfvadmin 3.2K Jun 22 2017 loadenv.csh
-r-xr-xr-x 1 dfvmgr dfvadmin 5.2K Jun 22 2017 loadenv
drwxr-xr-x 4 dfvmgr dfvadmin 4.0K Jun 22 2017 ..
drwxr-xr-x 3 dfvmgr dfvadmin 4.0K Jun 22 2017 .
lrwxrwxrwx 1 dfvmgr dfvadmin 7 Jun 22 2017 dfv_comp.pl -> loadenv
lrwxrwxrwx 1 dfvmgr dfvadmin 7 Jun 22 2017 dfv_run.pl -> loadenv
lrwxrwxrwx 1 dfvmgr dfvadmin 7 Jun 22 2017 dfv_sim.pl -> loadenv
/tools/dfv/scripts/v11/bin
However the script is only less than 200 lines and I can see the same content for dfv_comp.pl, dfv_run.pl, and dfv_sim.pl (also diff did not show any difference among the 3 perl scripts). loadenv of course also showed me the same contents.
Any help as to how I can view the real content of each perl script is much appreciated. Additional info which might help:
SHELL=/bin/tcsh
KONSOLE_DBUS_SERVICE=:1.46
KONSOLE_DBUS_WINDOW=/Windows/17
KONSOLE_DBUS_SESSION=/Sessions/30
Kindly let me know if additional info is needed. Thank you in advance!
What you're seeing is the real content. All three *.pl filenames are aliases to the loadenv script - that one script handles all four commands.
If you look at the contents of loadenv (or any of the other three names), you will most likely see that it checks to see which name was used to invoke it and then sets some flags which will cause it to behave differently depending on which name was used.

Perl script that checks rotated log files

There is a web server running multiple websites with configured daily log files rotation.
Task: create a perl script that checks a list of current log files in source directory and compare it with a list of rotated log files in another dir. Script must print a name of log file, if one the yesterday's log was not rotated.
Source dir example:
ls -l /var/log/httpd/logs/*log
-rw-r--r-- 1 root root 0 May 20 00:01 /var/log/httpd/logs/access.log
-rw-r--r-- 1 root root 483652 May 20 12:54 /var/log/httpd/logs/othersite.com_80-access.log
-rw-r--r-- 1 root root 305 May 20 11:51 /var/log/httpd/logs/othersite.com_80-error.log
-rw-r--r-- 1 root root 0 May 20 00:01 /var/log/httpd/logs/error.log
-rw-r--r-- 1 root root 46222 May 20 12:45 /var/log/httpd/logs/www.site.com_8880-access.log
-rw-r--r-- 1 root root 0 May 20 00:01 /var/log/httpd/logs/www.site.com_8880-error.log
dir with a rotated logs:
ls -l /var/log/httpd/logs/completed/|grep 2014-05-19
-rw-r--r-- 1 root root 20 May 20 00:01 access.log.2014-05-19.gz
-rw-r--r-- 1 root root 107244 May 20 00:01 othersite.com_80-access.log.2014-05-19.gz
-rw-r--r-- 1 root root 9991 May 20 00:01 www.site.com_8880-access.log.2014-05-19.gz
-rw-r--r-- 1 root root 20 May 20 00:01 www.site.com_8880-error.log.2014-05-19.gz
In this case two yesterday's log files are absent\were not rotated:
-rw-r--r-- 1 root root 483652 May 20 12:54 /var/log/httpd/logs/othersite.com_80-access.log
-rw-r--r-- 1 root root 305 May 20 11:51 /var/log/httpd/logs/othersite.com_80-error.log
Looking forward to any suggestions!
Do a glob <> on the source directory and put in a hash key. You put the size of the log or date in the value of the hash. Then go to the destination directory and do the same thing - read the files and as you loop, check to see if you have the file in you hash. You can compare the size and date also. if you don't have it, copy it.

GWT + eclipse, which files are part of my source?

i created a GWT project in eclipse, and it's time to put some code back into source control. i'm not sure at this point which files are generated and can be left out of source control,
a. under war/myapp/gwt/... i see many, many files related to the standard GWT themes.
b. under war/myapp,
-rw-r--r-- 1 10102022 1602597546 1876 Jan 24 16:41 0182DE3CC529E42DA72BBD969A44841E.gwt.rpc
-rw-r--r-- 1 10102022 1602597546 1456 Jan 24 14:09 4F701266A6E52E1E409583EA9AEC39E2.gwt.rpc
-rw-r--r-- 1 10102022 1602597546 1876 Jan 25 08:38 D98FD8FE56B70659E9608109BCF8B3C1.gwt.rpc
-rw-r--r-- 1 10102022 1602597546 43 Dec 16 16:01 clear.cache.gif
drwxr-xr-x 6 10102022 1602597546 204 Jan 25 08:26 gwt
-rw-r--r-- 1 10102022 1602597546 11289 Dec 17 01:33 hosted.html
-rw-r--r-- 1 10102022 1602597546 5232 Jan 25 08:31 photodrop_web_gwt.nocache.js
normally i'd just rely on eclipse build > clean to get rid of the build time artifacts. however, i did that, and i still see WEB-INF/classes full of class files, so i know that clean isn't working.
"war/myapp" is by default GWT's output directory. So as long as you haven't saved any files there manually (you shouldn't), you can delete that directory completely.
As always, make a backup first...
I'm using source control for GWT + GAE, and this ignore file has been working great:
syntax: regexp
^war/myapp$
syntax: regexp
^war/WEB-INF/appengine-generated/datastore-indexes-auto\.xml$
syntax: regexp
^war/WEB-INF/appengine-generated/local_db\.bin