How do I "reworker" in buildbot (or, how to fix the missing force button)? - buildbot

Starting with this buildbot master config:
# -*- python -*-
# ex: set filetype=python:
from buildbot.plugins import *
c = BuildmasterConfig = {}
c['workers'] = [worker.Worker("example-worker", "pass")]
c['protocols'] = {'pb': {'port': 9989}}
c['change_source'] = []
c['schedulers'] = []
c['schedulers'].append(schedulers.ForceScheduler(
name="force",
builderNames=["awesomesauce", "runtests"]))
factory = util.BuildFactory()
factory.addStep(steps.ShellCommand(command=['echo', 'hi']))
c['builders'] = []
c['builders'].append(
util.BuilderConfig(name="runtests",
workernames=["example-worker"],
factory=factory))
c['builders'].append(
util.BuilderConfig(name="awesomesauce",
workernames=["example-worker"],
factory=factory))
c['status'] = []
c['title'] = "Cool Beans"
c['titleURL'] = "http://example.com"
c['buildbotURL'] = "http://localhost:8010/"
c['www'] = dict(port=8010,
plugins=dict(waterfall_view={}, console_view={}))
c['db'] = {
'db_url' : "sqlite:///state.sqlite",
}
If I change the ForceScheduler to have the following:
builderNames=['awesomesauce', 'runtests', 'coolbeans']
And then simply copy the builders append block and change the name:
c['builders'].append(
util.BuilderConfig(name="coolbeans",
workernames=["example-worker"],
factory=factory))
Then when I do:
buildbot reconfigure master
It says that it added new builders, however when I go to the builders page it only lists the worker for for my original two builders, and the force button is missing from the builder page. Ruh roh.
Is there a way to fix the worker/builder link without completely restarting my buildbot master?

For the Force button, I'd say you have to also reconfigure the ForceScheduler, which is not part of your description.
For the worker assignment it looks like indeed this is a bug of our reconfiguration algorithm
http://trac.buildbot.net/ticket/3629#ticket
This bug only affects the UI. The builder is still configured with this worker, and should be able to start builds with that worker

Related

`nixos-rebuild switch` gets stuck when using `builtins.fetchGit`

I'm trying to download a package with a version that is not on nixpkgs. To do so, I'm using builtins.fetchGit. Here's a summary of the file where I use fetchGit (/etc/nixos/home/core.nix) for a better idea:
{ pkgs, username, homeDirectory }:
############################
# Custom package snapshots #
############################
let custom-ver-pkgs = {
# Haskell Language Server
hls = let pkgsSnapshot = import (builtins.fetchGit {
name = "custom hls version";
url = "https://github.com/nixos/nixpkgs-channels/";
ref = "refs/heads/nixpkgs-unstable";
rev = "2c162d49cd5b979eb66ff1653aecaeaa01690fcc";
}) {}; in pkgsSnapshot.haskellPackages.haskell-language-server;
};
in
{
# Actual config
}
And here's the point where I use the hls keyword defined above:
# Packages
home.packages = with pkgs; [
... # Normal packages
] ++
# Packages with custom version (See start of file)
(with custom-ver-pkgs; [
hls
]);
As you can see, I also use home-manager. The above-mentioned .../core.nix file is imported directly into /etc/nixos/configuration.nix.
As the title says, if I run sudo nixos-rebuild switch, the terminal freezes (in the sense that the command goes on forever without doing anything). What could my problem be?

Yocto variable not defined but set with _ operator

I'm struggling with something I'm not sure to address correctly.
In a Yocto environment (for STM32MP1 by the way) I have to configure a new target.
Hence I added to meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc this section, that looks like the other already available:
EXTLINUX_BOOTDEVICE_EMMC = "mmc1"
EXTLINUX_BOOTDEVICE_SDCARD = "mmc0"
EXTLINUX_ROOT_EMMC = "${#bb.utils.contains('ST_VENDORFS','1','root=/dev/mmcblk1p4','root=/dev/mmcblk1p3',d)}"
EXTLINUX_ROOT_NAND = "ubi.mtd=UBI rootfstype=ubifs root=ubi0:rootfs"
# Define available targets to use
UBOOT_EXTLINUX_CONFIGURED_TARGETS += "mp151a_sdcard"
UBOOT_EXTLINUX_CONFIGURED_TARGETS += "mp151a_emmc"
# Define bootprefix for each target
UBOOT_EXTLINUX_BOOTPREFIXES_mp151a_sdcard = "${EXTLINUX_BOOTDEVICE_SDCARD}_stm32mp151a_"
UBOOT_EXTLINUX_BOOTPREFIXES_mp151a_emcc = "${EXTLINUX_BOOTDEVICE_EMCC}_stm32mp151a_"
# Define labels for each target
UBOOT_EXTLINUX_LABELS_mp151a_sdcard = "stm32mp151a-sdcard"
UBOOT_EXTLINUX_LABELS_mp151a_emcc = "stm32mp151a-emcc"
# Define default boot config for each target
UBOOT_EXTLINUX_DEFAULT_LABEL_mp151a_sdcard ?= "stm32mp151a-sdcard"
UBOOT_EXTLINUX_DEFAULT_LABEL_mp151a_emcc ?= "stm32mp151a-emcc"
# Define FDT overrides for all labels
UBOOT_EXTLINUX_FDT_stm32mp151a-sdcard = "/stm32mp151a.dtb"
UBOOT_EXTLINUX_FDT_stm32mp151a-emcc = "/stm32mp151a.dtb"
# Define ROOT overrides for all labels
UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard = "${EXTLINUX_ROOT_SDCARD}"
UBOOT_EXTLINUX_ROOT_stm32mp151a-emcc = "${EXTLINUX_ROOT_EMCC}"
But when I bitbake <image> (that includes the file above) I get this output:
DEBUG: Executing python function update_extlinuxconf_targets
NOTE: UBOOT_EXTLINUX_CONFIGURED_TARGETS: mp157a-dk1_sdcard mp157a-dk1_sdcard-optee mp157c-dk2_sdcard mp157c-dk2_sdcard-optee mp157c-ed1_emmc mp157c-ed1_emmc-optee mp157c-ed1_sdcard mp157c-ed1_sdcard-optee mp157c-ev1_emmc mp157c-ev1_emmc-optee mp157c-ev1_nand mp157c-ev1_nor-sdcard mp157c-ev1_nor-emmc mp157c-ev1_sdcard mp157c-ev1_sdcard-optee mp151a_sdcard mp151a_emmc
NOTE: UBOOT_EXTLINUX_CONFIG_FLAGS: emmc sdcard
NOTE: *** Loop for config_label: emmc
NOTE: *** Loop for devicetree: stm32mp151a
NOTE: >>> New target label: mp151a_emmc
NOTE: >>> Append mp151a_emmc to UBOOT_EXTLINUX_TARGETS
NOTE: *** Loop for config_label: sdcard
NOTE: *** Loop for devicetree: stm32mp151a
NOTE: >>> New target label: mp151a_sdcard
NOTE: >>> Append mp151a_sdcard to UBOOT_EXTLINUX_TARGETS
NOTE: >>> UBOOT_EXTLINUX_TARGETS (updated): mp151a_emmc mp151a_sdcard
DEBUG: Python function update_extlinuxconf_targets finished
DEBUG: Executing python function do_create_multiextlinux_config
ERROR: UBOOT_EXTLINUX_ROOT not defined
DEBUG: Python function do_create_multiextlinux_config finished
ERROR: Function failed: do_create_multiextlinux_config
As you can see, the file is actually processed because it added the targets I've defined.
But it doesn't find the UBOOT_EXTLINUX_ROOT even if it's "set" with the _ operator:
UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard = "${EXTLINUX_ROOT_SDCARD}"
UBOOT_EXTLINUX_ROOT_stm32mp151a-emcc = "${EXTLINUX_ROOT_EMCC}"
I also tried to set the main variable to something like:
UBOOT_EXTLINUX_ROOT = ""
or
UBOOT_EXTLINUX_ROOT = "root=/dev/mmcblk1p4"
to see if it was the problem but it doesn't change nothing.
Is this something related to Yocto itself (I mean, something wrong in my syntax) or it's very specific to the SDK (meta-st) ?
The error above should be raised by this file:
root = localdata.getVar('UBOOT_EXTLINUX_ROOT')
if not root:
bb.fatal('UBOOT_EXTLINUX_ROOT not defined')
UPDATE
I checked the (huge) output of bitbake -e and among other targets I see:
# $UBOOT_EXTLINUX_ROOT [41 operations]
[...]
# "${EXTLINUX_ROOT_NOREMMC}"
# override[stm32mp157c-ev1-m4-examples-sdcard]:set /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc:274
# "${EXTLINUX_ROOT_SDCARD}"
# override[stm32mp157c-ev1-m4-examples-sdcard-optee]:set /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc:275
# "${EXTLINUX_ROOT_SDCARD_OPTEE}"
# override[stm32mp151a-sdcard]:set /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc:296
# "${EXTLINUX_ROOT_SDCARD}"
# override[stm32mp151a-emcc]:set /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc:297
[...]
# pre-expansion value:
# ""
UBOOT_EXTLINUX_ROOT=""
# $UBOOT_EXTLINUX_ROOT_cubemx-nor-sdcard
UBOOT_EXTLINUX_ROOT_cubemx-nor-sdcard="root=/dev/mmcblk0p3"
# $UBOOT_EXTLINUX_ROOT_cubemx-sdcard
UBOOT_EXTLINUX_ROOT_cubemx-sdcard="root=/dev/mmcblk0p6"
# $UBOOT_EXTLINUX_ROOT_stm32mp151a-emcc
UBOOT_EXTLINUX_ROOT_stm32mp151a-emcc="\${EXTLINUX_ROOT_EMCC}"
# $UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard
UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard="root=/dev/mmcblk0p6"
So far, if I understand correctly, the override values are correctly assigned (but not the ${EXTLINUX_ROOT_EMCC} - I don't understand where the \ comes from) but the main variable is still empty.
Adding UBOOT_EXTLINUX_ROOT = "root=/dev/mmcblk1p4" at the beginning of the above file, seems to do the trick (even if before I wrote the opposite, perhaps I forgot to clear the cache?) but I don't think it's the right way to do it.
You should specify the wanted name of the machine as a target to build, i.e.:
MACHINE=stm32mp151a-sdcard bitbake <image>
This way, the UBOOT_EXTLINUX_ROOT gets the non-empty value "root=/dev/mmcblk0p6" (from the UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard variant of the variable).

Within a gimp python-fu plug-in can one create/invoke a modal dialog (and/or register a procedure that is ONLY to be added as a temp procedure?)

I am trying to add a procedure to pop-up a modal dialog inside a plug-in.
Its purpose is to query a response at designated steps within the control-flow of the plug-in (not just acquire parameters at its start).
I have tried using gtk - I get a dialog but it is asynchronous - the plugin continues execution. It needs to operate as a synchronous function.
I have tried registering a plugin in order to take advantage of the gimpfu start-up dialogue for same. By itself, it works; it shows up in the procedural db when queried. But I never seem to be able to actually invoke it from within another plug-in - its either an execution error or wrong number of arguments no matter how many permutations I try.
[Reason behind all of this nonsense: I have written a lot of extension Python scripts for PaintShopPro. I have written a App package (with App.Do, App.Constants, Environment and the like that lets me begin to port those scripts to GIMP -- yes it is perverse, and yes sometimes the code just has to be rewritten, but for a lot of what I actual use in the PSP.API it is sufficient.
However, debugging and writing the module rhymes with witch. So. I am trying to add emulation of psp's "SetExecutionMode" (ie interactive). If
set, the intended behavior is that the App.Do() method will "pause" after/before it runs the applicable psp emulation code by popping up a simple message dialog.]
A simple modal dialogue within a gimp python-fu plug-in can be implemented via gtk's Dialog interface, specifically gtk.MessageDialog.
A generic dialog can be created via
queryDialogue = gtk.MessageDialog(None, gtk.DIALOG_DESTROY_WITH_PARENT \
gtk.MESSAGE_QUESTION, \
gtk.BUTTONS_OK_CANCEL, "")
Once the dialog has been shown,
a synchronous response may be obtained from it
queryDialogue.show()
response = queryDialogue.run()
queryDialogue.hide()
The above assumes that the dialog is not created and thence destroyed after each use.
In the use case (mentioned in the question) of a modal dialog to manage single stepping through a pspScript in gimp via an App emulator package, the dialogue message contents need to be customized for each use. [Hence, the "" for the message argument in the Constructor. [more below]]
In addition, the emulator must be able to accept a [cancel] response to 'get out of Dodge' - ie quit the entire plug-in (gracefully). I could not find a gimpfu interface for the latter, (and do not want to kill the app entirely via gimp.exit()). Hence, this is accomplished by raising a custom Exception class [appTerminate] within the App pkg and catching the exception in the outer-most scope of the plugin. When caught, then, the plug-in returns (exits).[App.Do() can not return a value to indicate continue/exit/etc, because the pspScripts are to be included verbatim.]
The following is an abbreviated skeleton of the solution -
a plug-in incorporating (in part) a pspScript
the App.py pkg supplying the environment and App.Do() to support the pspScript
a Map.py pkg supporting how pspScripts use dot-notation for parameters
App.py demonstrates creation, customization and use of a modal dialog - App.doContinue() displays the dialogue illustrating how it can be customized on each use.
App._parse() parses the pspScript (excerpt showing how it determines to start/stop single-step via the dialogue)
App._exec() implements the pspScript commands (excerpt showing how it creates the dialogue, identifies the message widget for later customization, and starts/stops its use)
# App.py (abbreviated)
#
import gimp
import gtk
import Map # see https://stackoverflow.com/questions/2352181/how-to- use-a-dot-to-access-members-of-dictionary
from Map import *
pdb = gimp.pdb
isDialogueAvailable = False
queryDialogue = None
queryMessage = None
Environment = Map({'executionMode' : 1 })
_AutoActionMode = Map({'Match' : 0})
_ExecutionMode = Map({'Default' : 0}, Silent=1, Interactive=2)
Constants = Map({'AutoActionMode' : _AutoActionMode}, ExecutionMode=_ExecutionMode ) # etc...
class appTerminate(Exception): pass
def Do(eNvironment, procedureName, options = {}):
global appTerminate
img = gimp.image_list()[0]
lyr = pdb.gimp_image_get_active_layer(img)
parsed = _parse(img, lyr, procedureName, options)
if eNvironment.executionMode == Constants.ExecutionMode.Interactive:
resp = doContinue(procedureName, parsed.detail)
if resp == -5: # OK
print procedureName # log to stdout
if parsed.valid:
if parsed.isvalid:
_exec(img, lyr, procedureName, options, parsed, eNvironment)
else:
print "invalid args"
else:
print "invalid procedure"
elif resp == -6: # CANCEL
raise appTerminate, "script cancelled"
pass # terminate plugin
else:
print procedureName + " skipped"
pass # skip execution, continue
else:
_exec(img, lyr, procedureName, options, parsed, eNvironment)
return
def doContinue(procedureName, details):
global queryMessage, querySkip, queryDialogue
# - customize the dialog -
if details == "":
msg = "About to execute procedure \n "+procedureName+ "\n\nContinue?"
else:
msg = "About to execute procedure \n "+procedureName+ "\n\nDetails - \n" + details +"\n\nContinue?"
queryMessage.set_text(msg)
queryDialogue.show()
resp = queryDialogue.run() # get modal response
queryDialogue.hide()
return resp
def _parse(img, lyr, procedureName, options):
# validate and interpret App.Do options' semantics vz gimp
if procedureName == "Selection":
isValid=True
# ...
# parsed = Map({'valid' : True}, isvalid=True, start=Start, width=Width, height=Height, channelOP=ChannelOP ...
# /Selection
# ...
elif procedureName == "SetExecutionMode":
generalOptions = options['GeneralSettings']
newMode = generalOptions['ExecutionMode']
if newMode == Constants.ExecutionMode.Interactive:
msg = "set mode interactive/single-step"
else:
msg = "set mode silent/run"
parsed = Map({'valid' : True}, isvalid=True, detail=msg, mode=newMode)
# /SetExecutionMode
else:
parsed = Map({'valid' : False})
return parsed
def _exec(img, lyr, procedureName, options, o, eNvironment):
global isDialogueAvailable, queryMessage, queryDialogue
#
try:
# -------------------------------------------------------------------------------------------------------------------
if procedureName == "Selection":
# pdb.gimp_rect_select(img, o.start[0], o.start[1], o.width, o.height, o.channelOP, ...
# /Selection
# ...
elif procedureName == "SetExecutionMode":
generalOptions = options['GeneralSettings']
eNvironment.executionMode = generalOptions['ExecutionMode']
if eNvironment.executionMode == Constants.ExecutionMode.Interactive:
if isDialogueAvailable:
queryDialogue.destroy() # then clean-up and refresh
isDialogueAvailable = True
queryDialogue = gtk.MessageDialog(None, gtk.DIALOG_DESTROY_WITH_PARENT, gtk.MESSAGE_QUESTION, gtk.BUTTONS_OK_CANCEL, "")
queryDialogue.set_title("psp/APP.Do Emulator")
queryDialogue.set_size_request(450, 180)
aqdContent = queryDialogue.children()[0]
aqdHeader = aqdContent.children()[0]
aqdMsgBox = aqdHeader.children()[1]
aqdMessage = aqdMsgBox.children()[0]
queryMessage = aqdMessage
else:
if isDialogueAvailable:
queryDialogue.destroy()
isDialogueAvailable = False
# /SetExecutionMode
else: # should not get here (should have been screened by parse)
raise AssertionError, "unimplemented PSP procedure: " + procedureName
except:
raise AssertionError, "App.Do("+procedureName+") generated an exception:\n" + sys.exc_info()
return
A skeleton of the plug-in itself. This illustrates incorporating a pspScript which includes a request for single-step/interactive execution mode, and thus the dialogues. It catches the terminate exception raised via the dialogue, and then terminates.
def generateWebImageSet(dasImage, dasLayer, title, mode):
try:
img = dasImage.duplicate()
# ...
bkg = img.layers[-1]
frameWidth = 52
start = bkg.offsets
end = (start[0]+bkg.width, start[1]+frameWidth)
# pspScript: (snippet included verbatim)
# SetExecutionMode / begin interactive single-step through pspScript
App.Do( Environment, 'SetExecutionMode', {
'GeneralSettings': {
'ExecutionMode': App.Constants.ExecutionMode.Interactive
}
})
# Selection
App.Do( Environment, 'Selection', {
'General' : {
'Mode' : 'Replace',
'Antialias' : False,
'Feather' : 0
},
'Start': start,
'End': end
})
# Promote
App.Do( Environment, 'SelectPromote' )
# und_so_weiter ...
except App.appTerminate:
raise AssertionError, "script cancelled"
# /generateWebImageSet
# _generateFloatingCanvasSetWeb.register -----------------------------------------
#
def generateFloatingCanvasSetWeb(dasImage, dasLayer, title):
mode="FCSW"
generateWebImageSet(dasImage, dasLayer, title, mode)
register(
"generateFloatingCanvasSetWeb",
"Generate Floating- Frame GW Canvas Image Set for Web Page",
"Generate Floating- Frame GW Canvas Image Set for Web Page",
"C G",
"C G",
"2019",
"<Image>/Image/Generate Web Imagesets/Floating-Frame Gallery-Wrapped Canvas Imageset...",
"*",
[
( PF_STRING, "title", "title", "")
],
[],
generateFloatingCanvasSetWeb)
main()
I realize that this may seem like a lot of work just to be able to include some pspScripts in a gimp plug-in, and to be able to single-step through the emulation. But we are talking about maybe 10K lines of scripts (and multiple scripts).
However, if any of this helps anyone else with dialogues inside plug-ins, etc., so much the better.

pmrep: [REP_55035] Not deploying DG - Dependent Session in source repository is required to deploy Workflow

In our current Project we are using Deployment Groups to migrate the code from DEV to TEST/PROD servers. We use Repository Manager to Create DG/Deploy the Code. As to reduce the effort on this, we want to perform this using PMREP command.
Issue:
I am deploying a DG which has 8 sessions which belong to a single Workflow ( lets call is wkf_ABC. This workflow contains 20 Sessions in TEST Env). Out of this 8 sessions , one session is new one and rest of the 8 sessions are ones which exist in ENV as well. I have added
a)8 sessions with all dependencies and
b) Workflow with no dependencies
I am deploying it using two methods, one of them is using Repository Manger and other one is using PMREP command.
1) When using Repository Manger, I am able to deploy the DG to TEST env succesfully ( provided I compare the folders - checking the CHECK BOX in Select Compare Folders Option while Deploying the DG - or else its failing with the Error as shown in the screenshot)
2) When using the pmrep deploydeploymentgroup command with the same DG, the DG is not getting deployed showing the below error.
Error: Dependent Session s_123 (id=xxxx) in source repository is required to deploy Workflow wkf_ABC, it is not available in the target repository and is not part of the deployment group.
Copy failed, rolling back changes...
Please help me if I am missing something
More Inputs :
1) s_123 is not part of 8 sessions which are getting deployed and the session s_123 is already in Target repository ( ENV)
2) The Deployment config files which we have used with pmrep command and the one which I have used have same set of attributes.
3) I have seen there are some similar issues when using pmrep command like Object mismatch etc. other than that everything looks good.
Systematic Deployment ( Repository Manager)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE DEPLOYPARAMS SYSTEM "depcntl.dtd">
<DEPLOYPARAMS DEFAULTSERVERNAME = "" COPYPROGRAMINFO = "YES" COPYMAPVARPERVALS = "YES" COPYWFLOWVARPERVALS = "NO" COPYWFLOWSESSLOGS = "NO" COPYDEPENDENCY = "YES" LATESTVERSIONONLY = "NO" RETAINGENERATEDVAL = "YES" RETAINSERVERNETVALS = "YES" RETAINMAPVARPERVALS = "NO">
<DEPLOYGROUP>
<APPLYLABEL SOURCELABELNAME = "" SOURCEMOVELABEL = "NO" TARGETLABELNAME = "" TARGETMOVELABEL = "NO"/>
</DEPLOYGROUP>
</DEPLOYPARAMS>
Deployment Configurant file passed along with pmrep command
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE DEPLOYPARAMS SYSTEM "/dev1/usr/local/informatica/9.1.0/server/bin/depcntl.dtd">
<DEPLOYPARAMS DEFAULTSERVERNAME = "" COPYPROGRAMINFO = "YES" COPYMAPVARPERVALS = "YES" COPYWFLOWVARPERVALS = "NO" COPYWFLOWSESSLOGS = "NO" COPYDEPENDENCY = "YES" LATESTVERSIONONLY = "NO" RETAINGENERATEDVAL = "YES" RETAINSERVERNETVALS = "YES" RETAINMAPVARPERVALS = "NO">
<DEPLOYGROUP CLEARSRCDEPLOYGROUP="NO">
<OVERRIDEFOLDER SOURCEFOLDERNAME = "B_EDW_ODS_ICDS" SOURCEFOLDERTYPE = "LOCAL" TARGETFOLDERNAME = "B_EDW_ODS_ICDS" TARGETFOLDERTYPE = "LOCAL" MODIFIEDMANUALLY = "YES"/>
.
.
.(extra lines for other folders)
<APPLYLABEL SOURCELABELNAME = "" SOURCEMOVELABEL = "NO" TARGETLABELNAME = "" TARGETMOVELABEL = "NO"/>
</DEPLOYGROUP>
</DEPLOYPARAMS>
Informatica(r) PMREP, version [9.1.0 HotFix6], build [496.0111], LINUX 64-bit

How can I export Jira issues to BitBucket

Ive just moved my projects code from java.net to BitBucket. But my jira issue tracking is still hosted on java.net, although BitBucket does have some options for linking to an external issue tracker I don't think I can use it for java.net, not least because I do not have the admin priviledges need to install the DVCS connector.
So I thought an alternative option would be to export and then import the issues into BitBucket issue tracker, is that possible ?
Progress so far
So I tried following the steps in both informative answers using OSX below but I hit a problem - I'm rather confused about what the script would actually be called because in the answers it talks about export.py but no such script exists with that name so I renamed the one I downloaded.
sudo easy_install pip (OSX)
pip install jira
pip install configparser
easy_install -U setuptools
Go to https://bitbucket.org/reece/rcore, select downloads tab, download zip and unzip, and rename to reece ( for some reason git clone https://bitbucket.org/reece/rcore fails with error)
cd reece/rcore
Save script as export.py in rcore subfolder
Replace iteritems with items in import.py
Replace iteritems with types/immutabledict.py
Create .config in rcore folder
Create .config/jira-issues-move-to-bitbucket.conf containing
jira-username=paultaylor
jira-hostname=https://java.net/jira/browse/JAUDIOTAGGER
jira-password=password
Run python export.py --jira-project jaudiotagger
gives
macbook:rcore paul$ python export.py --jira-project jaudiotagger
Traceback (most recent call last):
File "export.py", line 24, in <module>
import configparser
ImportError: No module named configparser
- Run python export.py --jira-project jaudiotagger
I need to run pip insdtall as root so did
sudo pip install configparser
and that worked
but now
python export.py --jira.project jaudiotagger
gives
File "export.py" line 35, in <module?
from jira.client import JIRA
ImportError: No module named jira.client
You can import issues into BitBucket, they just need to be in the appropriate format. Fortunately, Reece Hart has already written a Python script to connect to a Jira instance and export the issues.
To get the script to run I had to install the Jira Python package as well as the latest version of rcore (if you use pip you get an incompatible previous version, so you have to get the source). I also had to replace all instances of iteritems with items in the script and in rcore/types/immutabledict.py to make it work with Python 3. You will also need to fill in the dictionaries (priority_map, person_map, etc) with the values your project uses. Finally, you need a config file to exist with the connection info (see comments at the top of the script).
The basic command line usage is export.py --jira-project <project>
Once you've got the data exported, see the instructions for importing issues to BitBucket
#!/usr/bin/env python
"""extract issues from JIRA and export to a bitbucket archive
See:
https://confluence.atlassian.com/pages/viewpage.action?pageId=330796872
https://confluence.atlassian.com/display/BITBUCKET/Mark+up+comments
https://bitbucket.org/tutorials/markdowndemo/overview
2014-04-12 08:26 Reece Hart <reecehart#gmail.com>
Requires a file ~/.config/jira-issues-move-to-bitbucket.conf
with content like
[default]
jira-username=some.user
jira-hostname=somewhere.jira.com
jira-password=ur$pass
"""
import argparse
import collections
import configparser
import glob
import itertools
import json
import logging
import os
import pprint
import re
import sys
import zipfile
from jira.client import JIRA
from rcore.types.immutabledict import ImmutableDict
priority_map = {
'Critical (P1)': 'critical',
'Major (P2)': 'major',
'Minor (P3)': 'minor',
'Nice (P4)': 'trivial',
}
person_map = {
'reece.hart': 'reece',
# etc
}
issuetype_map = {
'Improvement': 'enhancement',
'New Feature': 'enhancement',
'Bug': 'bug',
'Technical task': 'task',
'Task': 'task',
}
status_map = {
'Closed': 'resolved',
'Duplicate': 'duplicate',
'In Progress': 'open',
'Open': 'new',
'Reopened': 'open',
'Resolved': 'resolved',
}
def parse_args(argv):
def sep_and_flatten(l):
# split comma-sep elements and flatten list
# e.g., ['a','b','c,d'] -> set('a','b','c','d')
return list( itertools.chain.from_iterable(e.split(',') for e in l) )
cf = configparser.ConfigParser()
cf.readfp(open(os.path.expanduser('~/.config/jira-issues-move-to-bitbucket.conf'),'r'))
ap = argparse.ArgumentParser(
description = __doc__
)
ap.add_argument(
'--jira-hostname', '-H',
default = cf.get('default','jira-hostname',fallback=None),
help = 'host name of Jira instances (used for url like https://hostname/, e.g., "instancename.jira.com")',
)
ap.add_argument(
'--jira-username', '-u',
default = cf.get('default','jira-username',fallback=None),
)
ap.add_argument(
'--jira-password', '-p',
default = cf.get('default','jira-password',fallback=None),
)
ap.add_argument(
'--jira-project', '-j',
required = True,
help = 'project key (e.g., JRA)',
)
ap.add_argument(
'--jira-issues', '-i',
action = 'append',
default = [],
help = 'issue id (e.g., JRA-9); multiple and comma-separated okay; default = all in project',
)
ap.add_argument(
'--jira-issues-file', '-I',
help = 'file containing issue ids (e.g., JRA-9)'
)
ap.add_argument(
'--jira-components', '-c',
action = 'append',
default = [],
help = 'components criterion; multiple and comma-separated okay; default = all in project',
)
ap.add_argument(
'--existing', '-e',
action = 'store_true',
default = False,
help = 'read existing archive (from export) and merge new issues'
)
opts = ap.parse_args(argv)
opts.jira_components = sep_and_flatten(opts.jira_components)
opts.jira_issues = sep_and_flatten(opts.jira_issues)
return opts
def link(url,text=None):
return "[{text}]({url})".format(url=url,text=url if text is None else text)
def reformat_to_markdown(desc):
def _indent4(mo):
i = " "
return i + mo.group(1).replace("\n",i)
def _repl_mention(mo):
return "#" + person_map[mo.group(1)]
#desc = desc.replace("\r","")
desc = re.sub("{noformat}(.+?){noformat}",_indent4,desc,flags=re.DOTALL+re.MULTILINE)
desc = re.sub(opts.jira_project+r"-(\d+)",r"issue #\1",desc)
desc = re.sub(r"\[~([^]]+)\]",_repl_mention,desc)
return desc
def fetch_issues(opts,jcl):
jql = [ 'project = ' + opts.jira_project ]
if opts.jira_components:
jql += [ ' OR '.join([ 'component = '+c for c in opts.jira_components ]) ]
if opts.jira_issues:
jql += [ ' OR '.join([ 'issue = '+i for i in opts.jira_issues ]) ]
jql_str = ' AND '.join(["("+q+")" for q in jql])
logging.info('executing query ' + jql_str)
return jcl.search_issues(jql_str,maxResults=500)
def jira_issue_to_bb_issue(opts,jcl,ji):
"""convert a jira issue to a dictionary with values appropriate for
POSTing as a bitbucket issue"""
logger = logging.getLogger(__name__)
content = reformat_to_markdown(ji.fields.description) if ji.fields.description else ''
if ji.fields.assignee is None:
resp = None
else:
resp = person_map[ji.fields.assignee.name]
reporter = person_map[ji.fields.reporter.name]
jiw = jcl.watchers(ji.key)
watchers = [ person_map[u.name] for u in jiw.watchers ] if jiw else []
milestone = None
if ji.fields.fixVersions:
vnames = [ v.name for v in ji.fields.fixVersions ]
milestone = vnames[0]
if len(vnames) > 1:
logger.warn("{ji.key}: bitbucket issues may have only 1 milestone (JIRA fixVersion); using only first ({f}) and ignoring rest ({r})".format(
ji=ji, f=milestone, r=",".join(vnames[1:])))
issue_id = extract_issue_number(ji.key)
bbi = {
'status': status_map[ji.fields.status.name],
'priority': priority_map[ji.fields.priority.name],
'kind': issuetype_map[ji.fields.issuetype.name],
'content_updated_on': ji.fields.created,
'voters': [],
'title': ji.fields.summary,
'reporter': reporter,
'component': None,
'watchers': watchers,
'content': content,
'assignee': resp,
'created_on': ji.fields.created,
'version': None, # ?
'edited_on': None,
'milestone': milestone,
'updated_on': ji.fields.updated,
'id': issue_id,
}
return bbi
def jira_comment_to_bb_comment(opts,jcl,jc):
bbc = {
'content': reformat_to_markdown(jc.body),
'created_on': jc.created,
'id': int(jc.id),
'updated_on': jc.updated,
'user': person_map[jc.author.name],
}
return bbc
def extract_issue_number(jira_issue_key):
return int(jira_issue_key.split('-')[-1])
def jira_key_to_bb_issue_tag(jira_issue_key):
return 'issue #' + str(extract_issue_number(jira_issue_key))
def jira_link_text(jk):
return link("https://invitae.jira.com/browse/"+jk,jk) + " (Invitae access required)"
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
opts = parse_args(sys.argv[1:])
dir_name = opts.jira_project
if opts.jira_components:
dir_name += '-' + ','.join(opts.jira_components)
if opts.jira_issues_file:
issues = [i.strip() for i in open(opts.jira_issues_file,'r')]
logger.info("added {n} issues from {opts.jira_issues_file} to issues list".format(n=len(issues),opts=opts))
opts.jira_issues += issues
opts.dir = os.path.join('/','tmp',dir_name)
opts.att_rel_dir = 'attachments'
opts.att_abs_dir = os.path.join(opts.dir,opts.att_rel_dir)
opts.json_fn = os.path.join(opts.dir,'db-1.0.json')
if not os.path.isdir(opts.att_abs_dir):
os.makedirs(opts.att_abs_dir)
opts.jira_issues = list(set(opts.jira_issues)) # distinctify
jcl = JIRA({'server': 'https://{opts.jira_hostname}/'.format(opts=opts)},
basic_auth=(opts.jira_username,opts.jira_password))
if opts.existing:
issues_db = json.load(open(opts.json_fn,'r'))
existing_ids = [ i['id'] for i in issues_db['issues'] ]
logger.info("read {n} issues from {fn}".format(n=len(existing_ids),fn=opts.json_fn))
else:
issues_db = dict()
issues_db['meta'] = {
'default_milestone': None,
'default_assignee': None,
'default_kind': "bug",
'default_component': None,
'default_version': None,
}
issues_db['attachments'] = []
issues_db['comments'] = []
issues_db['issues'] = []
issues_db['logs'] = []
issues_db['components'] = [ {'name':v.name} for v in jcl.project_components(opts.jira_project) ]
issues_db['milestones'] = [ {'name':v.name} for v in jcl.project_versions(opts.jira_project) ]
issues_db['versions'] = issues_db['milestones']
# bb_issue_map: bb issue # -> bitbucket issue
bb_issue_map = ImmutableDict( (i['id'],i) for i in issues_db['issues'] )
# jk_issue_map: jira key -> bitbucket issue
# contains only items migrated from JIRA (i.e., not preexisting issues with --existing)
jk_issue_map = ImmutableDict()
# issue_links is a dict of dicts of lists, using JIRA keys
# e.g., links['CORE-135']['depends on'] = ['CORE-137']
issue_links = collections.defaultdict(lambda: collections.defaultdict(lambda: []))
issues = fetch_issues(opts,jcl)
logger.info("fetch {n} issues from JIRA".format(n=len(issues)))
for ji in issues:
# Pfft. Need to fetch the issue again due to bug in JIRA.
# See https://bitbucket.org/bspeakmon/jira-python/issue/47/, comment on 2013-10-01 by ssonic
ji = jcl.issue(ji.key,expand="attachments,comments")
# create the issue
bbi = jira_issue_to_bb_issue(opts,jcl,ji)
issues_db['issues'] += [bbi]
bb_issue_map[bbi['id']] = bbi
jk_issue_map[ji.key] = bbi
issue_links[ji.key]['imported from'] = [jira_link_text(ji.key)]
# add comments
for jc in ji.fields.comment.comments:
bbc = jira_comment_to_bb_comment(opts,jcl,jc)
bbc['issue'] = bbi['id']
issues_db['comments'] += [bbc]
# add attachments
for ja in ji.fields.attachment:
att_rel_path = os.path.join(opts.att_rel_dir,ja.id)
att_abs_path = os.path.join(opts.att_abs_dir,ja.id)
if not os.path.exists(att_abs_path):
open(att_abs_path,'w').write(ja.get())
logger.info("Wrote {att_abs_path}".format(att_abs_path=att_abs_path))
bba = {
"path": att_rel_path,
"issue": bbi['id'],
"user": person_map[ja.author.name],
"filename": ja.filename,
}
issues_db['attachments'] += [bba]
# parent-child is task-subtask
if hasattr(ji.fields,'parent'):
issue_links[ji.fields.parent.key]['subtasks'].append(jira_key_to_bb_issue_tag(ji.key))
issue_links[ji.key]['parent task'].append(jira_key_to_bb_issue_tag(ji.fields.parent.key))
# add links
for il in ji.fields.issuelinks:
if hasattr(il,'outwardIssue'):
issue_links[ji.key][il.type.outward].append(jira_key_to_bb_issue_tag(il.outwardIssue.key))
elif hasattr(il,'inwardIssue'):
issue_links[ji.key][il.type.inward].append(jira_key_to_bb_issue_tag(il.inwardIssue.key))
logger.info("migrated issue {ji.key}: {ji.fields.summary} ({components})".format(
ji=ji,components=','.join(c.name for c in ji.fields.components)))
# append links section to content
# this section shows both task-subtask and "issue link" relationships
for src,dstlinks in issue_links.iteritems():
if src not in jk_issue_map:
logger.warn("issue {src}, with issue_links, not in jk_issue_map; skipping".format(src=src))
continue
links_block = "Links\n=====\n"
for desc,dsts in sorted(dstlinks.iteritems()):
links_block += "* **{desc}**: {links} \n".format(desc=desc,links=", ".join(dsts))
if jk_issue_map[src]['content']:
jk_issue_map[src]['content'] += "\n\n" + links_block
else:
jk_issue_map[src]['content'] = links_block
id_counts = collections.Counter(i['id'] for i in issues_db['issues'])
dupes = [ k for k,cnt in id_counts.iteritems() if cnt>1 ]
if dupes:
raise RuntimeError("{n} issue ids appear more than once from existing {opts.json_fn}".format(
n=len(dupes),opts=opts))
json.dump(issues_db,open(opts.json_fn,'w'))
logger.info("wrote {n} issues to {opts.json_fn}".format(n=len(id_counts),opts=opts))
# write zipfile
os.chdir(opts.dir)
with zipfile.ZipFile(opts.dir + '.zip','w') as zf:
for fn in ['db-1.0.json']+glob.glob('attachments/*'):
zf.write(fn)
logger.info("added {fn} to archive".format(fn=fn))
NOTE: I'm writing a new answer because writing this in a comment would be horrible, but most of the credit goes to #Turch's answer.
My steps (in OSX and Debian machines, both worked fine):
apt-get install python-pip (Debian) or sudo easy_install pip (OSX)
pip install jira
pip install configparser
easy_install -U setuptools (not sure if really needed)
Download or clone the source code from https://bitbucket.org/reece/rcore/ in your home folder, for example. Note: don't download using pip, it will get the 0.0.2 version and you need the 0.0.3.
Download the Python script created by Reece, mentioned by #Turch, and place it inside of the rcore folder.
Follow the instructions by #Turch: I also had to replace all instances of iteritems with items in the script and in rcore/types/immutabledict.py to make it work with Python 3. You will also need to fill in the dictionaries (priority_map, person_map, etc) with the values your project uses. Finally, you need a config file to exist with the connection info (see comments at the top of the script). Note: I used hostname like jira.domain.com (no http or https).
(This change did the trick for me) I had to change part of the line 250 from 'https://{opts.jira_hostname}/' to 'http://{opts.jira_hostname}/'
To finish, run the script like #Turch mentioned: The basic command line usage is export.py --jira-project <project>
The file was placed in /tmp/.zip for me.
The file was perfectly accepted in the BitBucket importer today.
Hooray for Reece and Turch! Thanks guys!