Is it possible to configure Azure Windows VMs using Ansible on Azure DevOps Microsoft Hosted Ubuntu agents? - azure-devops

We try to configure an Azure VM using an Azure DevOps pipeline. We first create the machine using Terraform and then we need to configure it. Right now the pipeline is functional when we use a customized Ubuntu Azure DevOps agent (a VM we setup ourselves in Azure).
We prefer to use a Microsoft Hosted Ubuntu Agent. When we try to run our pipeline using the Microsoft Hosted Ubuntu agent we fail with a message "winrm or requests is not installed".
We have done a lot of research and attempts to install the needed components, but none have been fruitful.
All the examples and documentation on the internet we can find don't mention our specific use case. Ansible configuration of Windows VMs in Azure from a Microsoft Hosted Ubuntu agent. Isn't it possible for some reason?
If it is, any pointers in the right direction will be much appreciated!
The error we see in the Azure DevOps pipeline is this:
ansible-playbook -vvvv -i inventory/hosts.cfg main.yml --extra-vars '{"customer_name": "<REMOVED>" }'
ansible-playbook [core 2.12.5]
config file = None
configured module search path = ['/home/vsts/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/vsts/.local/lib/python3.8/site-packages/ansible
ansible collection location = /home/vsts/.ansible/collections:/usr/share/ansible/collections
executable location = /home/vsts/.local/bin/ansible-playbook
python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
jinja version = 2.10.1
libyaml = True
No config file found; using defaults
setting up inventory plugins
host_list declined parsing /home/vsts/work/1/s/ansible/inventory/hosts.cfg as it did not pass its verify_file() method
auto declined parsing /home/vsts/work/1/s/ansible/inventory/hosts.cfg as it did not pass its verify_file() method
yaml declined parsing /home/vsts/work/1/s/ansible/inventory/hosts.cfg as it did not pass its verify_file() method
Parsed /home/vsts/work/1/s/ansible/inventory/hosts.cfg inventory source with ini plugin
Loading collection ansible.windows from /home/vsts/.local/lib/python3.8/site-packages/ansible_collections/ansible/windows
Loading collection community.windows from /home/vsts/.local/lib/python3.8/site-packages/ansible_collections/community/windows
redirecting (type: modules) ansible.builtin.win_service to ansible.windows.win_service
redirecting (type: modules) ansible.builtin.win_service to ansible.windows.win_service
redirecting (type: modules) ansible.builtin.win_service to ansible.windows.win_service
redirecting (type: modules) ansible.builtin.win_service to ansible.windows.win_service
redirecting (type: modules) ansible.builtin.win_service to ansible.windows.win_service
Loading callback plugin default of type stdout, v2.0 from /home/vsts/.local/lib/python3.8/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
PLAYBOOK: main.yml *************************************************************
Positional arguments: main.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/home/vsts/work/1/s/ansible/inventory/hosts.cfg',)
extra_vars: ('{"customer_name": "<REMOVED>"}',)
forks: 5
1 plays in main.yml
PLAY [windows:pro] *********************************************************
TASK [Gathering Facts] *********************************************************
task path: /home/vsts/work/1/s/ansible/main.yml:1
redirecting (type: modules) ansible.builtin.setup to ansible.windows.setup
Using module file /home/vsts/.local/lib/python3.8/site-packages/ansible_collections/ansible/windows/plugins/modules/setup.ps1
Pipelining is enabled.
**fatal: [51.144.125.149]: FAILED! => {
"msg": "winrm or requests is not installed: No module named 'winrm'"
}**
PLAY RECAP *********************************************************************
51.144.125.149 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
We tried to fix the problem by installing various potentially relevant components in the pipeline just before running the ansible-playbook command, for instance this one
pip3 install pywinrm
Later, based on input on this SO question we tried this in the pipeline:
python3 -m pip install --ignore-installed pywinrm
find / -name winrm.py
ansible-playbook -vvv -i inventory/hosts.cfg main.yml
The find command finds winrm.py here:
/opt/pipx/venvs/ansible-core/lib/python3.8/site-packages/ansible/plugins/connection/winrm.py
The ansible-playbook configuration we are using is:
ansible-playbook [core 2.12.5]
config file = None
configured module search path =
['/home/vsts/.ansible/plugins/modules',
'/usr/share/ansible/plugins/modules']
ansible python module location = /opt/pipx/venvs/ansible-
core/lib/python3.8/site-packages/ansible
ansible collection location =
/home/vsts/.ansible/collections:/usr/share/ansible/collections
executable location = /opt/pipx_bin/ansible-playbook
python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC
9.4.0]
jinja version = 3.1.2
libyaml = True
No config file found; using defaults
The error we get is:
task path: /home/vsts/work/1/s/ansible/main.yml:1
redirecting (type: modules) ansible.builtin.setup to
ansible.windows.setup
Using module file /opt/pipx/venvs/ansible-
core/lib/python3.8/site-
packages/ansible_collections/ansible/windows/plugins/modules/
setup.ps1
Pipelining is enabled.
fatal: [13.73.148.141]: FAILED! => {
"msg": "winrm or requests is not installed: No module named
'winrm'"
}

you can try solution in RedHat knowledgebase
https://access.redhat.com/solutions/3356681
Last comment suggestion (replace yum with apt commands)
I was getting this error even if python2-winrm version 0.3.0 is
already installed via yum
yum list installed | grep winrm python2-winrm.noarch
0.3.0-1.el7 #epel
pip install "pywinrm>=0.2.2" only resulted in "Requirement already
satisfied"
I ran this to resolve the error -
yum autoremove python2-winrm.noarch
pip install "pywinrm>=0.2.2"
Then ping: pong worked just fine over https, port=5986
ram#thinkred1cartoon$ ansible all -i hosts.txt -m win_ping
172.16.96.135 | SUCCESS => {
"changed": false,
"ping": "pong" }
conversely, if you don't want to run command 1, then command 2 won't
work for you. In that case, run command 3
3 ) pip install --ignore-installed "pywinrm>=0.2.2"

Related

Installing php extension for SQL Anywhere on server with CentOS, Plesk, php-fpm

we got a new WebServer with CentOS, Plesk and php-fpm.
Now we don't get the php extension running.
What we did:
installed latest SQL Anywhere 17 (ebf29577)
copied php-7.3.0_sqlanywhere_r.so to /opt/plesk/php/7.3/lib64/php/modules/
loading php-7.3.0_sqlanywhere_r.so via /opt/plesk/php/7.3/etc/php.d/sqlanywhere.ini
this leads to an error:
WARNING: [pool plesk-php73-fpm.plesk-service.localdomain] child 11616 said into stderr: "NOTICE: PHP message: PHP Warning: request_startup() for sqlanywhere module failed in Unknown on line 0"
(This leads to php 7.3 fpm is not loading at all)
next try was to switch from FPM to CGI: same error
module is loadable itself within shell if LD_LIBRARY_PATH was enhanced with path /opt/sqlanywhere17/lib64/
some tries getting the cause by strace did not help
Has anybody an idea or maybe even successfully installed php extension.
Thanks
Florian
after a lot of tests we found a solution:
in Plesk Domain php settings we had to add:
[php-fpm-pool-settings]
env[LD_LIBRARY_PATH]="/opt/sqlanywhere_v2/lib64:$LD_LIBRARY_PATH"
Now all libs are found and it works
For me solution was, creating custom handler:
plesk bin php_handler --add -displayname "PHP SQLAnywhere 7.4" -path /opt/php-custom-handlers/7.4/php-sqlanywhere.fcgi -clipath /opt/plesk/php/7.4/bin/php -phpini /opt/plesk/php/7.4/etc/php.ini -type fastcgi
php-sqlanywhere.fcgi contents:
#!/bin/bash
LD_LIBRARY_PATH=/opt/sqlanywhere17/lib64
export LD_LIBRARY_PATH
exec /opt/plesk/php/7.4/bin/php-cgi "$#"
And finally using this handler on domain php settings and adding on additional directives:
extension=sqlanywhere.so

How to stop 'import psycopg2' from causing an Exception when starting an Azure Container?

I am trying to deploy a Django REST API using Azure App Service on Linux. I am using a postgresql Database and deploy via pipeline. Azure has postgresql 9.6. After running my pipeline, the Website shows an Server Error (500).
The AppLogs show, that the Container couldn't be started due an failed import of psycopg2.
[ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/site/wwwroot/antenv/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 25, in
import psycopg2 as Database
File "/home/site/wwwroot/antenv/lib/python3.7/site-packages/psycopg2/__init__.py", line 50, in
from psycopg2._psycopg import ( # noqa
ImportError: /home/site/wwwroot/antenv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PQencryptPasswordConn
In the Build-stage of my pipeline, I set up my environment (python3.7) like this:
- script: |
python -m venv antenv
source antenv/bin/activate
python -m pip install --upgrade pip
pip install setup
pip install -r requirements.txt
Where requirements.txt looks like this:
Django==3.0.2
djangorestframework==3.11.0
psycopg2-binary==2.8.4
pandas==0.25.3
pytest==5.3.5
pytest-django==3.8.0
pytest-mock==2.0.0
python-dateutil==2.8.1
sqlparse==0.3.0
whitenoise==5.0.1
BuildJob and DeploymentJob seem to run flawless. the Build-logs indicate that psycopg2_binary-2.8.4-cp37-cp37m-manylinux1_x86_64.whl was correctly downloaded and installed.
Also the App runs fine on my machine when using the database on azure by configuring in the settings.py:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'databasename',
'USER': 'user#postgresqlserver',
'PASSWORD': 'Password',
'HOST': 'postgresqlserver.postgres.database.azure.com',
'PORT': '',
'OPTIONS': {'sslmode': 'require'}
}
} # Of course the info is actually saved in environment variables
This gives me the feeling, that something with the psycopg2 installation is not working... For others the *psycopg2-binary seemed to do the trick but unfortunateley not for me.
Am I right to assume that on azure I'm nether able to install postgresql10 as suggested here https://github.com/psycopg/psycopg2/issues/983 nor can install from source like suggested here https://github.com/psycopg/psycopg2/issues/1018?
There must be something I am missing, I would be grateful for any advice!
EDIT:
Taking a look at the library (as suggested here https://stackoverflow.com/a/59652816/13183775) I found that I don't have a PQencryptPasswordConn function but only a PQencryptPassword function. I have the feeling that this is expected for Postgresql9.6 (https://github.com/psycopg/psycopg2/blob/cb3353be1f10590cdc2a894ada42c3b4c171feb7/psycopg/psycopgmodule.c#L466).
To check, whether there are multiple versions libpq:
/>find . -name "libpq*"
./var/lib/dpkg/info/libpq5:amd64.symbols
./var/lib/dpkg/info/libpq5:amd64.shlibs
./var/lib/dpkg/info/libpq5:amd64.list
./var/lib/dpkg/info/libpq5:amd64.triggers
./var/lib/dpkg/info/libpq-dev.list
./var/lib/dpkg/info/libpq5:amd64.md5sums
./var/lib/dpkg/info/libpq-dev.md5sums
./usr/share/doc/libpq5
./usr/share/doc/libpq-dev
./usr/share/locale/ko/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/it/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/pl/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/zh_TW/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/tr/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/cs/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/de/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/ru/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/sv/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/pt_BR/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/fr/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/es/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/zh_CN/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/ja/LC_MESSAGES/libpq5-9.6.mo
./usr/lib/x86_64-linux-gnu/pkgconfig/libpq.pc
./usr/lib/x86_64-linux-gnu/libpq.so.5
./usr/lib/x86_64-linux-gnu/libpq.so
./usr/lib/x86_64-linux-gnu/libpq.so.5.9
./usr/lib/x86_64-linux-gnu/libpq.a
./usr/include/postgresql/libpq-events.h
./usr/include/postgresql/libpq-fe.h
./usr/include/postgresql/libpq
./usr/include/postgresql/libpq/libpq-fs.h
./usr/include/postgresql/internal/libpq
./usr/include/postgresql/internal/libpq-int.h>
Sadly I'm not able to see here wether there are multiple libpq versions...

Unable to execute "custom script extension" on Azure VM using azure CLI from linux sub system

I am working on developing post-deployment Pester validation script for my project. I need to push pester scripts into the VM as custom script extension using Azure CLI.
Following is the command I executed:
az vm extension set --resource-group SomeRG--vm-name SimpleVM --name
customScript --publisher Microsoft.Azure.Extensions --settings '{"fileUris":
["https://github.com/myname/DSCConfig/blob/master/pester.ps1"],
"commandToExecute":"powershell -ExecutionPolicy Unrestricted -File
pester.ps1"}' --version 2.0
and I got the below error in the Linux interface after executing above command:
Deployment failed. Correlation ID:
8ba16fc0-fea6-4650-bb0a-2b73c9613dfe. Handler
'Microsoft.Azure.Extensions.customScript' has reported failure for VM
Extension 'customScript' with terminal error code '1007' and error
message: 'Install failed for the plugin (name:
Microsoft.Azure.Extensions.customScript, version 2.0.6) with exception
The specified executable is not a valid application for this OS
platform.'
And while checking the extension on VM saw the status as "Transitioning " and details as Install failed for the plugin (name: Microsoft.Azure.Extensions.customScript, version 2.0.6) with exception The specified executable is not a valid application for this OS platform)
Alternatively I tried with other publishers: Microsoft.Compute and Microsoft.OSTCExtensions
Unfortunately, none of them worked. I have been stuck at this step for past two days. Any help is much appreciated.
I think you might be using the wrong custom script extension (the one you are using I believe is for Linux VMs). I think you should be using the one named "CustomScriptExtension" with a publisher of "Microsoft.Compute" and version set to "1.9" as documented here.
Specifically, try this command instead:
az vm extension set --resource-group SomeRG--vm-name SimpleVM --name CustomScriptExtension --publisher Microsoft.Compute --settings '{"fileUris": ["https://github.com/myname/DSCConfig/blob/master/pester.ps1"], "commandToExecute":"powershell -ExecutionPolicy Unrestricted -File pester.ps1"}' --version 1.9

puppet 4.0 vagrant modules missing

I am trying to use puppet modules in vagrant.
My box is running puppet 4.0
I am installing modules using:
if [ ! -d /etc/puppet/modules/ ]; then
puppet module install puppetlabs-java
fi
in site.pp
I have:
class { 'java':
distribution => 'jdk',
}
I keep getting an error about could not find declared class java
why can't puppet find my module?
/etc/puppet/modules/ is the default path isn't it?
vagrant file
Vagrant.configure(2) do |config|
config.vm.box = "bento/centos-7.2"
config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "8192"
end
config.vm.provision :shell, :path => "upgrade_puppet.sh"
config.vm.provision :shell, :path => "puppet_modules.sh"
config.vm.provision :puppet do |puppet|
puppet.options = '--verbose --debug'
puppet.environment_path = "puppet/environments"
puppet.environment = "production"
end
end
Updated answer now that Vagrantfile has been provided
Locations have changed in puppet 4 and directory environments are now in use by default.
So how you are using the puppet provisioner is correct. However, vagrant will upload all the directories it needs to the guest, based on your Vagrantfile to:
/tmp/vagrant-puppet/environments/production
When Vagrant calls the puppet apply it will be looking for the modules it requires in:
/tmp/vagrant-puppet/environments/production/modules
and that module directory does not exist on your host.
You can change your if block to be:
if [ ! -d /vagrant/puppet/environments/production/modules ]; then
puppet module install puppetlabs-java --modulepath /vagrant/puppet/environments/production/modules
fi
/vagrant is shared between host and guest. This would install the java module and its dependencies on your host machine under:
puppet
|
+--environments
+
-- production
|
+ -- manifests
| +
| -- site.pp
|
+ -- modules
+
-- java
+
-- stdlib
When you do your vagrant up, this content gets uploaded to the host under:
/tmp/vagrant-puppet
Tested and confirmed based on your Vagrantfile.
As Jaxim mentions, it's because the default directory locations have changed in the newer version of Puppet.
If you're interested in installing moduels automatically with Puppet, I'd recommend the R10K vagrant plugin, you can specify versions of modules and make updating them much easier, and allows you to download modules not on the forge, such as git repos.
https://github.com/jantman/vagrant-r10k
A little bit late, but I am switching from Chef over to Puppet (company policy, do not ask! :) ) and ran into the exact same situation and coming from Chef background I was refusing to "pollute" my project folder with so many Puppet specific stuff. In my opinion, I should only need Vagrantfile and nothing else.
I was also getting the "Could not find declared class java at /tmp/vagrant-puppet/environments/production" error message. So, after much messing around I've found that in puppet.options you can provide any arguments that you would normally provide if calling puppet apply at the command line.
So, if anything helps try modifying the puppet.options in your Vagrantfile as follows:
config.vm.provision :puppet do |puppet|
puppet.options = '--verbose --modulepath=/etc/puppetlabs/code/environments/production/modules'
puppet.environment_path = "puppet/environments"
puppet.environment = "production"
end
This will help Puppet find its own nose and not think that everything is available at the /tmp folder, but that the modules have already been installed at its own folder location.

uWSGI + virtualenv 'No module named site'

So this seems to be a really common problem with this setup, but I can't find any solutions that work on SO. I've setup a very new Ubuntu 15.04 server, then installed nginx, virtualenv (and -wrapper), and uWSGI (via apt-get, so globally, not inside the virtualenv).
My virtualenv is located at /root/Env/example. Inside of the virtualenv, I installed Django, then at /srv/www/example/app ran Django's startproject command with the project name example, so I have vaguely this structure:
-root
-Env
-example
-bin
-lib
-srv
-www
-example
-app
-example
manage.py
-example
wsgi.py
...
My example.ini file for uWSGI looks like this:
[uwsgi]
project = example
plugin = python
chdir = /srv/www/example/app/example
home = /root/Env/example
module = example.wsgi:application
master = true
processes = 5
socket = /run/uwsgi/app/example/example.socket
chmod-socket = 664
uid = www-data
gid = www-data
vacuum = true
But no matter whether I run this via uwsgi --ini /etc/uwsgi/apps-enabled/example.ini or via daemon, I get the exact same error:
Python version: 2.7.9 (default, Apr 2 2015, 15:37:21) [GCC 4.9.2]
Set PythonHome to /root/Env/example
ImportError: No module named site
I should note that the Django project works via the built-in development server ./manage.py runserver, and that when I remove home = /root/Env/example the thing works (but is obviously using the global Python and Django rather than the virtualenv versions, which means it's useless for a proper virtualenv setup).
Can anyone see some obvious path error that I'm not seeing? As far as I can tell, home is entirely correct based on my directory structure, and everything else in the ini too, so why is it not working with this ImportError?
In my case, I was seeing this issue because the django app I was trying to run was written in python 3 whereas uwsgi was configured for python 2. I fixed the problem by:
recompiling uwsgi to support both python 2 and python 3 apps
(I followed this guide)
adding this to my mydjangoproject_uwsgi.ini:
plugins = python35 # or whatever you specified while compiling uwsgi
For other folks using Django, you should also make sure you are correctly specifying the following:
# Django dir that contains manage.py
chdir = /var/www/project/myprojectname
# Django wsgi (myprojectname is the name of your top-level project)
module = myprojectname.wsgi:application
# the virtualenv you are using (full path)
home = /home/ubuntu/Env/mydjangovenv
plugins = python35
As #Freek said, site refers to a python module.
The error claims that python cannot find that package, which is because you have specified python_home to the wrong location.
I've encountered with the same problem and my uwsgi.ini is like below:
[uwsgi]
# variable
base = /home/xx/
# project settings
chdir = %(base)/
module = botservice.uwsgi:application
home = %(base)/env/bin
For this configuration uwsgi can find python executable in /env/bin but no packages could be found under this folder. So I changed home to
home = %(base)/env/
and it worked for me.
In your case, I suggest digging into home directive and point it to a location which contains both python executable and packages.
The site module is in the root of django.
First check is to activate the virtualenv manually (source /root/Env/example/bin/activate, start python and import site). If that fails, pip install django.
Assuming that django is correctly installed in the virtualenv, make sure that uWSGI activates the virtualenv. Relevant uWSGI configuration directives:
plugins = python
virtualenv = /root/Env/example
and in case you have error importing example.wsgi:
pythonpath = /srv/www/example/app/example