readthedocs "virtualenv: error: unrecognized arguments" during update_repos - virtualenv

Question

I have a local readthedocs installation at /readthedocs.org on a Debian buster container. Python 3.6 is installed along with the pip requirements (including virtualenv). I have a single repository added to readthedocs (Openwhisk) with the following basic config:
# In .readthedocs.yml
mkdocs:
configuration: mkdocs.yml
python:
version: 3.6
system_packages: true
# In mkdocs.yml
site_name: Openwhisk
When I run python3.6 manage.py update_repos I get the following error:
[27/May/2020 19:00:01] readthedocs.doc_builder.environments:146[4964]: INFO Running: 'python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest' [$HOME]
[27/May/2020 19:00:01] readthedocs.doc_builder.environments:294[4964]: DEBUG Post response via JSON encoded data: {'id': 41, 'run_time': 0, 'command': 'python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest', 'description': '', 'output': 'usage: virtualenv [--version] [--with-traceback] [-v | -q] [--app-data APP_DATA] [--clear-app-data] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed]\n [--activators comma_sep_list] [--clear] [--system-site-packages] [--symlinks | --copies] [--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip]\n [--no-setuptools] [--no-wheel] [--symlink-app-data] [--prompt prompt] [-h]\n dest\nvirtualenv: error: unrecognized arguments: /readthedocs.org/user_builds/openwhisk/envs/latest', 'exit_code': 2, 'start_time': '2020-05-27T19:00:01.078364Z', 'end_time': '2020-05-27T19:00:01.326186Z', 'build': 7}
[27/May/2020 19:00:01] readthedocs.vcs_support.utils:129[4964]: DEBUG Lock (openwhisk): Releasing
[27/May/2020 19:00:01] readthedocs.doc_builder.environments:625[4964]: WARNING (Build) [openwhisk:latest] Command python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest failed:
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--app-data APP_DATA] [--clear-app-data] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed]
[--activators comma_sep_list] [--clear] [--system-site-packages] [--symlinks | --copies] [--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip]
[--no-setuptools] [--no-wheel] [--symlink-app-data] [--prompt prompt] [-h]
dest
virtualenv: error: unrecognized arguments: /readthedocs.org/user_builds/openwhisk/envs/latest
Traceback (most recent call last):
File "/readthedocs.org/readthedocs/projects/tasks.py", line 763, in run_build
self.setup_python_environment()
File "/readthedocs.org/readthedocs/projects/tasks.py", line 1139, in setup_python_environment
self.python_env.setup_base()
File "/readthedocs.org/readthedocs/doc_builder/python_environments.py", line 314, in setup_base
cwd='$HOME',
File "/readthedocs.org/readthedocs/doc_builder/environments.py", line 638, in run
return super().run(*cmd, **kwargs)
File "/readthedocs.org/readthedocs/doc_builder/environments.py", line 420, in run
return self.run_command_class(cls=self.command_class, cmd=cmd, **kwargs)
File "/readthedocs.org/readthedocs/doc_builder/environments.py", line 644, in run_command_class
return super().run_command_class(*cmd, **kwargs)
File "/readthedocs.org/readthedocs/doc_builder/environments.py", line 492, in run_command_class
raise BuildEnvironmentWarning(msg)
readthedocs.doc_builder.exceptions.BuildEnvironmentWarning: Command python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest failed:
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--app-data APP_DATA] [--clear-app-data] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed]
[--activators comma_sep_list] [--clear] [--system-site-packages] [--symlinks | --copies] [--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip]
[--no-setuptools] [--no-wheel] [--symlink-app-data] [--prompt prompt] [-h]
dest
virtualenv: error: unrecognized arguments: /readthedocs.org/user_builds/openwhisk/envs/latest
[27/May/2020 19:00:01] readthedocs.doc_builder.environments:581[4964]: INFO (Build) [openwhisk:latest] Build finished
[27/May/2020 19:00:01] celery.app.trace:124[4964]: INFO Task readthedocs.projects.tasks.send_notifications[508c1b76-28bf-4c97-9afd-5824295c378f] succeeded in 0.007888936903327703s: None
[27/May/2020 19:00:01] readthedocs.projects.tasks:1450[4964]: INFO Skipping build files deletetion for version: 1
[27/May/2020 19:00:01] celery.app.trace:124[4964]: INFO Task readthedocs.projects.tasks.update_docs_task[bf8e5976-c70a-47b8-bf9c-69a31fbd9c24] succeeded in 4.51686664391309s: True
The odd thing is when I run the exact same command from the CLI it works fine:
root#38e93376269d:/readthedocs.org# python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest
created virtual environment CPython3.6.10.final.0-64 in 274ms
creator CPython3Posix(dest=/readthedocs.org/user_builds/openwhisk/envs/latest, clear=False, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Any suggestions?

Answer 1

the use_system_site_packages: true key/value config has been removed and doesn't work anymore.
use system_packages: true instead.
python:
version: 3.6
system_packages: true

Answer 2

To answer my own question: There's currently a bug with readthedocs 5.1 (and possibly older versions) where having use_system_site_packages set to false in .readthedocs.yml will give this error.
To fix, put this in .readthedocs.yml:
python:
use_system_site_packages: true

Answer 3

The error is due to the way that Read the Docs invokes python -m virtualenv, which seems to cause issues with the building environment in some cases (but likely not for Docker environments). I can reproduce this with Debian Buster as well, and it is directly reproducible by adding an empty-string argument: python -mvirtualenv "" my_dest/
It is reported upstream, keep an eye out for a fix there: https://github.com/readthedocs/readthedocs.org/issues/7322