Pylint with pyspark throws incorrect warning - pyspark

i would like to use pylint. But when i applied it to my code, which uses pyspark, I got warning "drop_duplicates is not callable". if i change my code from "drop_duplicates" to "dropDuplicates", the warning disappears. How to fix it without changing whole code? I think drop_duplicates is more python-style. I tried to add pyspark-stubs, but it doenst help.

PySpark is a nightmare to deal with when it comes to linting (even with stubs) because of the JVM generated members.
If you plan to ignore those you have two options:
Add the following to .pylintrc file in your project root and make sure your linter is configured to pick it up:
extension-pkg-whitelist=pyspark
generated-members=pyspark.*
ignored-modules=pyspark.sql.functions
Whenever you run lint, run it with the following parameters:
pylint -j 0 --ignored-modules=pyspark.sql.functions --extension-pkg-whitelist=pyspark --generated-members=pyspark.*
Hope this helps.

Related

What is the correct way to add an option to the VSCode Settings Menu (editing VSCode's source)

(not sure if I asked this earlier somewhere, but I can't find it)
Earlier I found out how to create a usable entry on the VSCode settings menu (also usable in settings.json) by editing these VSCode source files:
vscode/src/vs/editor/common/config/editorOptions.ts
Adding a variable in export interface IEditorOptions.
Adding an enum in export const enum EditorOption.
Adding an item in export const EditorOptions.
vscode/src/vs/editor/common/standalone/standaloneEnums.ts
Adding an enum to export enum EditorOption.
I don't know if that's the correct way, I just copied other options.
I hadn't yet figured out how to access the new entry's value from the function I wanted to use it in (I had already implemented the functionality, just without the option).
I then got sidetracked and forgot about this.
Now when I try to build it I'm getting an error (it builds fine without the changes).
running yarn --cwd <mydir> run gulp vscode-linux-x64
gives Error: monaco.d.ts is no longer up to date. Please run gulp watch and commit the new file.
I'm not sure exactly how I'm supposed to run gulp watch here. I don't know anything about yarn or gulp (I'm just using the VSCode contributor instructions), and running yarn --cwd <mydir> run gulp watch doesn't fix the problem.
It sounds like vscode/src/vs/monaco.d.ts being read by the build process before being written/updated, because its timestamp changes. I'm not sure if that's my fault and/or if it's a red herring.
Does anyone know what I should be doing here? Has anyone here modified the settings menu?
Is there any documentation out there for modifying the settings menu? (I had a look, didn't find anything).

vscode linters underlines entire class

I'm using vscode in two different environments - a cloud based VM (using remote-ssh) and a docker container (using remote-container).
I have pylint set up on both.
For some reason, when the linter finds and issue on the VM it underlines the entire scope of that issue. For example if I have a class with an issue, the entire class would be underlined.
On the docker environment however only the first letter of the class is underlined. This is much better.
I tried to understand where the difference comes from... went through the settings.json file of both environments and both workspaces but I can't seem to find any difference.
Anyone have any idea what setting affects this?
In the image below, the top code is from the VM, the bottom is from the docker
There's an open issue in pylint regarding this : https://github.com/PyCQA/pylint/issues/5466. It's scheduled to be in 2.13.0. Downgrading to 2.12.1 solve the issue but also remove other end of line end of column information introduced in https://github.com/PyCQA/pylint/issues/5336
downgrading from pylint==2.12.2 to pylint==2.12.1 seems to have solved the issue

command 'flow.restartClient' not found in vscode

I'm trying to use flow-for-vscode, but it looks like not working, in fact when I try to use it I get:
command 'flow.restartClient' not found
Notice that ./node_modules/.bin/flow works correctly from my project folder.
I also tried to set ${workspaceFolder}/node_modules/.bin/flow as Path to Flow, but it doesn't work.
Also, disabling Use LSP gives the same result.

Netbeans php-cs-fixer end up with error "Files that were not fixed due to errors reported during linting after fixing:"

I am using php-cs-fixer for code formatting in Netbeans 8.2. When I try to format one file, it shows the error
Files that were not fixed due to errors reported during linting after fixing:
I searched for the fix in many websites, but couldn't get this fixed. Is there any way to fix this? I tried with both php-cs-fixer 1 and php-cs-fixer 2.
The error message means that PHP CS Fixer loaded some files from drive, apply changes on them, and then realised that files are not valid anymore (invalid PHP syntax) after those changes, thus it decided to not save it. That's one of the safety mechanism of PHP CS Fixer to not break your project.
This means that you have found an issue in PHP CS Fixer itself.
Please, verify you are using newest release, maybe the bug was already fixed!
If not, please consider to expose your configuration file (if any) and content of files you got listed at https://github.com/FriendsOfPHP/PHP-CS-Fixer/issues/new !
There is also an option to see what happen in progress.
$ ./vendor/bin/php-cs-fixer fix src/ErrorFile.php -vvv
The --verbose option will show the applied rules. When using the txt format it will also display progress notifications.
NOTE: if there is an error like "errors reported during linting after fixing", you can use this to be even more verbose for debugging purpose
-v: verbose
-vv: very verbose
-vvv: debug

Problems using netbeans 7.01 with CMake 2.8.6

so once again, I'm having a little problem I can't figure out myself. Meaning, I'm pretty sure I know what the problem is, I just can't fix it.
I'm developing in C++ using Netbeans 7.01 with CMake 2.8.6. However, when I'm trying to build a project, I get the following error from CMake:
CMake Error: The path to the source directory:
H:/Projects/Netbeans/CppApplication_1/CONF=Release
contains unsupported character '='.
Please use a different source directory name.
Please note, that "CONF=Release" isn't a folder. It seems to be a configuration flag set by Netbeans, as it changes when I'm trying a debug build. So, my guess here is, that theres a space missing, or something similar. Unfortunately, I can't figure out where to change that. There are no options in any related Netbeans window, I can't find anything related in the configuration files for netbeans itself or the project either.
Did anyone here have experienced the same problem and found a solution to it? I would be very glad to hear it. If there's any information missing, let me know, I will add them as fast as possible.
Edit:
Just noticed there is already version 7.1 of netbeans, nice to know. However, even with this version, the same problem occurs.
You must specify the command you ran to allow for the error to be identified. However, it looks like you missed the "source-path" parameter.
The cmake command ends with the path to source and should be something like:
cmake -D CONF=Release <path to source>