Can I register event callbacks using the libvirt Python module with a QEMU backend? - virtualization

I would like to write some code to monitor events for domains running under QEMU, managed by libvirt. However, trying to register an event handler yields the following error:
>>> import libvirt
>>> conn = libvirt.openReadOnly('qemu:///system')
>>> conn.domainEventRegister(callback, None)
libvir: Remote error : this function is not supported by the connection driver: no event support
("callback" in this case is a stub function that simply prints its arguments.)
The examples I've been able to find regarding libvirt's event handling don't seem to be specific as to which backend hypervisors support which features. Is this expected to work for QEMU backends?
I'm running a Fedora 16 system, which includes libvirt 0.9.6 and qemu-kvm 0.15.1.
For folks finding themselves here via <searchengine>:
UPDATE 2013-10-04
Many months and a few Fedora releases later, the event-test.py code in the libvirt git repository runs correctly on Fedora 19.

Make sure you have registered in the libvirt event loop (or set up your own) before registering for events.
There is a nice example of event handling shipped with the libvirt source (file is called event-test.py). I'm attaching an example based on that code;
import libvirt
import time
import threading
def callback(conn, dom, event, detail, opaque):
print "EVENT: Domain %s(%s) %s %s" % (dom.name(),
dom.ID(),
event,
detail)
eventLoopThread = None
def virEventLoopNativeRun():
while True:
libvirt.virEventRunDefaultImpl()
def virEventLoopNativeStart():
global eventLoopThread
libvirt.virEventRegisterDefaultImpl()
eventLoopThread = threading.Thread(target=virEventLoopNativeRun,
name="libvirtEventLoop")
eventLoopThread.setDaemon(True)
eventLoopThread.start()
if __name__ == '__main__':
virEventLoopNativeStart()
conn = libvirt.openReadOnly('qemu:///system')
conn.domainEventRegister(callback, None)
conn.setKeepAlive(5, 3)
while conn.isAlive() == 1:
time.sleep(1)
Good luck!
//Seto

Related

JAX pmap with multi-core CPU

What is the correct method for using multiple CPU cores with jax.pmap?
The following example creates an environment variable for SPMD on CPU core backends, tests that JAX recognises the devices, and attempts a device lock.
import os
os.environ["XLA_FLAGS"] = '--xla_force_host_platform_device_count=2'
import jax as jx
import jax.numpy as jnp
jx.local_device_count()
# WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
# 2
jx.devices("cpu")
# [CpuDevice(id=0), CpuDevice(id=1)]
def sfunc(x): while True: pass
jx.pmap(sfunc)(jnp.arange(2))
Executing from a jupyter kernel and observing htop shows that only one core is locked
I receive the same output from htop when omitting the first two lines and running:
$ env XLA_FLAGS=--xla_force_host_platform_device_count=2 python test.py
Replacing sfunc with
def sfunc(x): return 2.0*x
and calling
jx.pmap(sfunc)(jnp.arange(2))
# ShardedDeviceArray([0., 2.], dtype=float32, weak_type=True)
does return a SharedDeviecArray.
Clearly I am not correctly configuring JAX/XLA to use two cores. What am I missing and what can I do to diagnose the problem?
As far as I can tell, you are configuring the cores correctly (see e.g. Issue #2714). The problem lies in your test function:
def sfunc(x): while True: pass
This function gets stuck in an infinite loop at trace-time, not at run-time. Tracing happens in your host Python process on a single CPU (see How to think in JAX for an introduction to the idea of tracing within JAX transformations).
If you want to observe CPU usage at runtime, you'll have to use a function that finishes tracing and begins running. For that you could use any long-running function that actually produces results. Here is a simple example:
def sfunc(x):
for i in range(100):
x = (x # x)
return x
jx.pmap(sfunc)(jnp.zeros((2, 1000, 1000)))

Pydev showing 'undefined variable' for Jython code analysis

I have used Eclipse, Jython & Pydev for a long time. Upgrading as and when new releases arrived. All has worked very well until recently when the Pydev code completion started marking common items such as dir or print as 'undefined variable'. But the program ran correctly.
The error log showed:
'The python client still hasn't connected back to the eclipse java vm (will retry..)'
'Attempt: 2 of 5 failed, trying again...(socket connected still null)'
and more attempts to 5 out of 5
'Error connecting to python process(most likely cause for failure is firewall blocking...misconfigured network)'
Also, attempting to create a live jython console CTRL+ALT+ENTER gives the following error:
'Create interactive Console' has encountered a problem
Error initializing console.
Unexpected error connecting to console.
Failed to receive suitable Hello response from pydevconsole. Last msg received: Console already exited with value: 1 while waiting for an answer.
I have spent a lot of time looking for answers here and elsewhere that have included suggestions to check:
Mixed 32/64 bit installs; Firewall problems; IPV4 preference; localhost entries; path issues and others: all I've checked out with no success so far.
Software is Windows 10, Eclipse 4.21.0, Pydev 9.1.0.2021, Java JDK 11.0.13, Jython 2.7.2
I should be most grateful for any further help on this problem.
Many thanks
Well, you get that message because PyDev does spawn a shell and communicate with it for collecting code-completion results and the same happens for the interactive console.
Now, in both cases it seems like the socket communication is being prevented in your use case (as the messages states, the usual culprit is some firewall -- or possibly an anti-virus -- or some network misconfiguration).
Unfortunately, it's really hard for me to diagnose this as it's pretty much machine-dependent (all I can say is that I checked things here and Jython is working well, so, the issue lies in some misconfiguration on your machine).
I actually have plans to stop requiring that socket communication (for getting the completions from a shell) and using stdin/stdout (https://www.brainwy.com/tracker/PyDev/1183), but this still isn't done.
As Jython does run for you, you can try to create a simple server to verify a connection works.
i.e.: Create an echo_server.py with:
import socket
if __name__ == '__main__':
HOST = '127.0.0.1'
PORT = 65432
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print('Connected by', addr)
data = conn.recv(1024)
conn.sendall(data)
print('Echo server finished')
and a client.py with:
import socket
if __name__ == '__main__':
HOST = '127.0.0.1'
PORT = 65432
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.sendall(b'Hello, world')
data = s.recv(1024)
print('Received %r' % (data,))
Then run the server part and then the client part and check if that does work properly for you.

volttron.platform.vip.agent.core ERROR: Possible conflicting identity

I have been working towards building my agent development skills in Volttron. I am completely new to the platform and trying to understand how to create basic agents that publish and subscribe to the Volttron bus. I'm not alone in this venture and get help from a few other people with experience, but even they are stumped. We are using the same agent files we share through GitHub, but the agent works on their computers and not on mine.
The publishing agent reads from a CSV file that is in the same directory as the agent and is suppose to publish information from the file. I have been careful to map the file directory in my source code to match my setup. I receive the following messages when I start running my publishing agent using eclipse "mars" that's running on Linux Mint 18.1 Serena:
2017-02-02 14:27:22,290 volttron.platform.agent.utils DEBUG: missing file /home/edward/.volttron/keystores/f9d18589-d62b-42b7-bac8-3498a0c37220/keystore.json
2017-02-02 14:27:22,290 volttron.platform.agent.utils INFO: creating file /home/edward/.volttron/keystores/f9d18589-d62b-42b7-bac8-3498a0c37220/keystore.json
2017-02-02 14:27:22,292 volttron.platform.vip.agent.core DEBUG: address: ipc://#/home/edward/.volttron/run/vip.socket
2017-02-02 14:27:22,292 volttron.platform.vip.agent.core DEBUG: identity: None
2017-02-02 14:27:22,292 volttron.platform.vip.agent.core DEBUG: agent_uuid: None
2017-02-02 14:27:22,292 volttron.platform.vip.agent.core DEBUG: severkey: None
2017-02-02 14:27:32,324 volttron.platform.vip.agent.core ERROR: No response to hello message after 10 seconds.
2017-02-02 14:27:32,324 volttron.platform.vip.agent.core ERROR: A common reason for this is a conflicting VIP IDENTITY.
2017-02-02 14:27:32,324 volttron.platform.vip.agent.core ERROR: Shutting down agent.
2017-02-02 14:27:32,324 volttron.platform.vip.agent.core ERROR: Possible conflicting identity is: f9d18589-d62b-42b7-bac8-3498a0c37220
I have done the following:
Created missing file "/home/edward/.volttron/keystores/f9d18589-d62b-42b7-bac8-3498a0c37220/keystore.json". The only thing that happens when I run the agent again is it gives me the same DEBUG message but with a different file name.
I looked into the "volttron.platform.vip.agent.core" file and have no idea what to do in there. I dont want to create more problems for myself.
I have been using the "Volttron's Documentation" to try and trouble shoot, but I always get the same message when ever I try to run any agent. I have had success when testing the platform and running "make-listener" through the terminal, but that's all.
I have been searching the web for the last couple of days and seen similar issues, but when attempting to follow the advise posted to remedy the situation, I have no luck. Error: volttron.platform.web INFO: Web server not started
Reinstalled Volttron, Mint, and Eclipse on my VM a few times to overcome any compatibility issues...
The source code for the agent is as follows:
#testcodeisforpublishingandprinting
import logging
import sys
#import json
from volttron.platform.vip.agent import Agent, Core, PubSub, compat
#from volttron.platform.vip.agent import *
#from volttron.platform.vip.agent import compat
from volttron.platform.agent import utils
from volttron.platform.messaging import headers as headers_mod
from datetime import datetime
#import numpy as NP
#from numpy import linalg as LA
import csv
outdata=open("/home/edward/volttron/testagent/Agent/PredictionfileP.csv","rb")
Pdata=csv.DictReader(outdata)
Price=[]
for row in Pdata:
Price.append(float(row['Price'])*0.01)
#from volttron.platform.agent import BaseAgent, PublishMixin, periodic, matching, utils
#from volttron.platform.agent import BaseAgent, PublishMixin, periodic
utils.setup_logging()
_log = logging.getLogger(__name__)
class testagent1(Agent):
def __init__(self, config_path, **kwargs):
self.config = utils.load_config(config_path)
super(testagent1, self).__init__(**kwargs)
self.step=0
#print('TestAgent example agent start-up function')
#Core.receiver('onsetup')
def onsetup(self, sender, **kwargs):
self._agent_id = self.config['agentid']
#Core.receiver('onstart')
def onstart(self, sender, **kwargs):
pass
#Core.receiver('onstop')
def onstop(self, sender, **kwargs):
pass
#Core.receiver('onfinish')
def onfinish(self, sender, **kwargs):
pass
#Core.periodic(5)
def simulate(self):
self.step=self.step+1#timestep increase
print('Simulationrunning')
now = datetime.utcnow().isoformat(' ')#time now
headers = {
'AgentID': self._agent_id,
headers_mod.CONTENT_TYPE: headers_mod.CONTENT_TYPE.PLAIN_TEXT,
headers_mod.DATE: now,
}
print(self.step)
self.vip.pubsub.publish('pubsub', 'testcase1/Step', headers, self.step)
print('Simulationupdatingloopingindex')
def main(argv=sys.argv):
'''Main method called by the eggsecutable.'''
try:
utils.vip_main(testagent1)
except Exception as e:
_log.exception('unhandled exception')
if __name__ == '__main__':
# Entry point for script
sys.exit(main())
I installed my version of Volttron using the 3.5RC1 manual published Jan. 2017.
I am assuming that you are running this from eclipse and not through the installation process. The installation of the agent would specify an identity that would remain for the life of the agent.
The remaining answer is specific to running within the eclipse environment.
def main(argv=sys.argv):
'''Main method called by the eggsecutable.'''
try:
# This is where the change is.
utils.vip_main(testagent1, identity='Thisidentity')
except Exception as e:
_log.exception('unhandled exception')
You will have to authorize the agent to be able to connect to the message bus by adding the agent publickey through the auth mechanism. Or you can add the wildcard /.*/ to the credentials of an entry through volttron-ctl auth add.
Thank you for asking this question. We will are updating the documentation to highlight this.
You will need to do the following at the command line:
volttron-ctl auth add
domain []:
address []:
user_id []:
capabilities (delimit multiple entries with comma) []:
roles (delimit multiple entries with comma) []:
groups (delimit multiple entries with comma) []:
mechanism [CURVE]:
credentials []: /.*/
comments []:
enabled [True]:
added entry domain=None, address=None, mechanism='CURVE', credentials=u'/.*/', user_id='ff6fea8e-53bd-4506-8237-fbb718aca70d'

Buildbot slaves priority

Problem
I have set up a latent slave in buildbot to help avoid congestion.
I've set up my builds to run either in permanent slave or latent one. The idea is the latent slave is waken up only when needed but the result is that buildbot randomly selectes one slave or the other so sometimes I have to wait for the latent slave to wake even if the permanent one is idle.
Is there a way to prioritize buildbot slaves?
Attempted solutions
1. Custom nextSlave
Following #david-dean suggestion, I've created a nextSlave function as follows (updated to working version):
from twisted.python import log
import traceback
def slave_selector(builder, builders):
try:
host = None
support = None
for builder in builders:
if builder.slave.slavename == 'host-slave':
host = builder
elif builder.slave.slavename == 'support-slave':
support = builder
if host and support and len(support.slave.slave_status.runningBuilds) < len(host.slave.slave_status.runningBuilds):
log.msg('host-slave has many running builds, launching build in support-slave')
return support
if not support:
log.msg('no support slave found, launching build in host-slave')
elif not host:
log.msg('no host slave found, launching build in support-slave')
return support
else:
log.msg('launching build in host-slave')
return host
except Exception as e:
log.err(str(e))
log.err(traceback.format_exc())
log.msg('Selecting random slave')
return random.choice(buildslaves)
And then passed it to BuilderConfig.
The result is that I get this in twistd.log:
2014-04-28 11:01:45+0200 [-] added buildset 4329 to database
But the build never starts, in the web UI it always appear as Pending and none of the logs I've put appear in twistd.log
2. Trying to mimic default behavior
I've having a look to buildbot code, to see how it is done by default.
in file ./master/buildbot/process/buildrequestdistributor.py, class BasicBuildChooser you have:
self.nextSlave = self.bldr.config.nextSlave
if not self.nextSlave:
self.nextSlave = lambda _,slaves: random.choice(slaves) if slaves else None
So I've set exactly that lambda function in my BuilderConfig and I'm getting exactly the same build not starting result.
You can set up a nextSlave function to assign slaves to a builder in a custom manner see: http://docs.buildbot.net/current/manual/cfg-builders.html#builder-configuration

How to delete a queue in rabbit mq

I am using rabbitmctl using pika library.
I use the following code to create a Producer
#!/usr/bin/env python
import pika
import time
import json
import datetime
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
def callback(ch, method, properties, body):
#print " current time: %s " % (str(int((time.time())*1000)))
print body
channel.basic_consume(callback,
queue='hello',
no_ack=True)
channel.start_consuming()
Since I create an existing queue everytime (Over-write the creation of queue in case if queue is not created) The queue has been corrupted due to this.and now I want to delete the queue..how do i do that?
Since this seems to be a maintenance procedure, and not something you'll be doing routinely on your code, you should probably be using the RabbitMQ management plugin and delete the queue from there.
Anyway, you can delete it from pika with:
channel.queue_delete(queue='hello')
https://pika.readthedocs.org/en/latest/modules/channel.html#pika.channel.Channel.queue_delete
The detailed answer is as follows (with reference to above very helpful and useful answer)
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
'localhost'))
channel = connection.channel()
channel.queue_delete(queue='hello')
connection.close()
GUI rabbitMQ mgm't made that easy
$ sudo rabbitmq-plugins enable rabbitmq_management
http://localhost:15672/#/queues
Username : guest
password : guest
inspired by this