Airflow duplicated plugin entries in virtualenv - plugins

Question

Was setting up Airflow (2.1.4) in a virtual environment followed by an install of a third-party plugin "pip install simple-dag-editor"
Plugin installed successfully, however upon checking the plugin list, there were duplicated entries.
(venv) root#test-server:/opt/airflow$ airflow plugins
name | source | flask_blueprints | appbuilder_views
==================+============================================================+=======================================================+=============================================================
simple_dag_editor | simple-dag-editor==0.1.1: | <flask.blueprints.Blueprint object at 0x7f69e5e427b8> | {'category': 'Admin', 'name': 'Simple DAG editor', 'view':
| EntryPoint(name='simple_dag_editor', value='simple_dag_edi | | <simple_dag_editor.app_builder_view.AppBuilderDagEditorView
| tor.simple_dag_editor:SimpleDagEditor', | | object at 0x7f69e5dd1470>}
| group='airflow.plugins') | |
simple_dag_editor | simple-dag-editor==0.1.1: | <flask.blueprints.Blueprint object at 0x7f69e5e427b8> | {'category': 'Admin', 'name': 'Simple DAG editor', 'view':
| EntryPoint(name='simple_dag_editor', value='simple_dag_edi | | <simple_dag_editor.app_builder_view.AppBuilderDagEditorView
| tor.simple_dag_editor:SimpleDagEditor', | | object at 0x7f69e5dd1470>}
| group='airflow.plugins') | |
Airflow portal also resulted in 2 entries in the "Admin" section
Any idea what is happening? I tested the setup again both on a docker container and standalone on the server. Both instances did not result in the duplicated entries therefore I am suspecting it is related to running Airflow in a Python virtual environment. The server is running on CentOS 7.

Answer 1

I believe you might have plugin installed twice in two different places:
In "plugins" folder as simply a python package
Installed as python package
Aiflow Allows for both types of installations, and I think if you have both - it will install both.
If you change the airflow log level to verbose, you should be able to see two entries:
"Loading plugins from entrypoints"
"Loading plugins from directory: <DIRECTORY>"
They should be followed by attempts to import the plugins.
The solution would be to remove the plugings from the "plugins" directory.
It's also possible that you have two packages that have the same entrypoint - for example if you installed it with different package name before, it could also be discovered twice. Airflow checks all pacakges available and if they have appropriate entrypoint declared, it will load it as plugin. But if you enable DEBUG logging level, you should see details. You can easily set airflow logging level by config option (or environment variable):
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging-level