object not found when creating targets list programmatically - targets-r-package

I'm trying to generate a {targets} list programmatically, via a function in an R package.
get_pipeline <- function(which_countries) {
countries <- NULL # avoid R CMD CHECK warning
print(which_countries) # Shows that which_countries is available
list(
targets::tar_target(
name = countries,
command = which_countries # But here, which_countries is not found
)
)
}
The _targets.R file looks like this:
library(targets)
couns <- c("USA", "GBR")
TargetsQuestions::get_pipeline(couns)
I see the following error:
> tar_make()
[1] "USA" "GBR"
Error in enexpr(expr) : object 'which_countries' not found
Error in `tar_throw_run()`:
! callr subprocess failed: object 'which_countries' not found
Note that the which_countries variable is printable, but not found in the call to tar_target.
How can I get create the countries target successfully so that it contains the vector c("USA", "GBR")?
This code is in a GitHub repository at https://github.com/MatthewHeun/TargetsQuestions. To reproduce:
git clone https://github.com/MatthewHeun/TargetsQuestions
Build the package in RStudio.
targets::tar_make() at the Console.
Thanks in advance for any suggestions!

Thanks to #landau for pointing to https://wlandau.github.io/targetopia/contributing.html#target-factories which in turn points to the metaprogramming section of Advanced R at https://adv-r.hadley.nz/metaprogramming.html.
The solution turned out to be:
get_pipeline <- function(which_countries) {
list(
targets::tar_target_raw(
name = "countries",
# command = which_countries # which_countries must have length 1
# command = !!which_countries # invalid argument type
command = rlang::enexpr(which_countries) # Works
)
)
}
With _targets.R like this:
library(targets)
couns <- c("USA", "GBR")
TargetsQuestions::get_pipeline(couns)
the commands tar_make() and tar_read(countries), give
[1] "USA" "GBR"
as expected!

Related

Yocto variable not defined but set with _ operator

I'm struggling with something I'm not sure to address correctly.
In a Yocto environment (for STM32MP1 by the way) I have to configure a new target.
Hence I added to meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc this section, that looks like the other already available:
EXTLINUX_BOOTDEVICE_EMMC = "mmc1"
EXTLINUX_BOOTDEVICE_SDCARD = "mmc0"
EXTLINUX_ROOT_EMMC = "${#bb.utils.contains('ST_VENDORFS','1','root=/dev/mmcblk1p4','root=/dev/mmcblk1p3',d)}"
EXTLINUX_ROOT_NAND = "ubi.mtd=UBI rootfstype=ubifs root=ubi0:rootfs"
# Define available targets to use
UBOOT_EXTLINUX_CONFIGURED_TARGETS += "mp151a_sdcard"
UBOOT_EXTLINUX_CONFIGURED_TARGETS += "mp151a_emmc"
# Define bootprefix for each target
UBOOT_EXTLINUX_BOOTPREFIXES_mp151a_sdcard = "${EXTLINUX_BOOTDEVICE_SDCARD}_stm32mp151a_"
UBOOT_EXTLINUX_BOOTPREFIXES_mp151a_emcc = "${EXTLINUX_BOOTDEVICE_EMCC}_stm32mp151a_"
# Define labels for each target
UBOOT_EXTLINUX_LABELS_mp151a_sdcard = "stm32mp151a-sdcard"
UBOOT_EXTLINUX_LABELS_mp151a_emcc = "stm32mp151a-emcc"
# Define default boot config for each target
UBOOT_EXTLINUX_DEFAULT_LABEL_mp151a_sdcard ?= "stm32mp151a-sdcard"
UBOOT_EXTLINUX_DEFAULT_LABEL_mp151a_emcc ?= "stm32mp151a-emcc"
# Define FDT overrides for all labels
UBOOT_EXTLINUX_FDT_stm32mp151a-sdcard = "/stm32mp151a.dtb"
UBOOT_EXTLINUX_FDT_stm32mp151a-emcc = "/stm32mp151a.dtb"
# Define ROOT overrides for all labels
UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard = "${EXTLINUX_ROOT_SDCARD}"
UBOOT_EXTLINUX_ROOT_stm32mp151a-emcc = "${EXTLINUX_ROOT_EMCC}"
But when I bitbake <image> (that includes the file above) I get this output:
DEBUG: Executing python function update_extlinuxconf_targets
NOTE: UBOOT_EXTLINUX_CONFIGURED_TARGETS: mp157a-dk1_sdcard mp157a-dk1_sdcard-optee mp157c-dk2_sdcard mp157c-dk2_sdcard-optee mp157c-ed1_emmc mp157c-ed1_emmc-optee mp157c-ed1_sdcard mp157c-ed1_sdcard-optee mp157c-ev1_emmc mp157c-ev1_emmc-optee mp157c-ev1_nand mp157c-ev1_nor-sdcard mp157c-ev1_nor-emmc mp157c-ev1_sdcard mp157c-ev1_sdcard-optee mp151a_sdcard mp151a_emmc
NOTE: UBOOT_EXTLINUX_CONFIG_FLAGS: emmc sdcard
NOTE: *** Loop for config_label: emmc
NOTE: *** Loop for devicetree: stm32mp151a
NOTE: >>> New target label: mp151a_emmc
NOTE: >>> Append mp151a_emmc to UBOOT_EXTLINUX_TARGETS
NOTE: *** Loop for config_label: sdcard
NOTE: *** Loop for devicetree: stm32mp151a
NOTE: >>> New target label: mp151a_sdcard
NOTE: >>> Append mp151a_sdcard to UBOOT_EXTLINUX_TARGETS
NOTE: >>> UBOOT_EXTLINUX_TARGETS (updated): mp151a_emmc mp151a_sdcard
DEBUG: Python function update_extlinuxconf_targets finished
DEBUG: Executing python function do_create_multiextlinux_config
ERROR: UBOOT_EXTLINUX_ROOT not defined
DEBUG: Python function do_create_multiextlinux_config finished
ERROR: Function failed: do_create_multiextlinux_config
As you can see, the file is actually processed because it added the targets I've defined.
But it doesn't find the UBOOT_EXTLINUX_ROOT even if it's "set" with the _ operator:
UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard = "${EXTLINUX_ROOT_SDCARD}"
UBOOT_EXTLINUX_ROOT_stm32mp151a-emcc = "${EXTLINUX_ROOT_EMCC}"
I also tried to set the main variable to something like:
UBOOT_EXTLINUX_ROOT = ""
or
UBOOT_EXTLINUX_ROOT = "root=/dev/mmcblk1p4"
to see if it was the problem but it doesn't change nothing.
Is this something related to Yocto itself (I mean, something wrong in my syntax) or it's very specific to the SDK (meta-st) ?
The error above should be raised by this file:
root = localdata.getVar('UBOOT_EXTLINUX_ROOT')
if not root:
bb.fatal('UBOOT_EXTLINUX_ROOT not defined')
UPDATE
I checked the (huge) output of bitbake -e and among other targets I see:
# $UBOOT_EXTLINUX_ROOT [41 operations]
[...]
# "${EXTLINUX_ROOT_NOREMMC}"
# override[stm32mp157c-ev1-m4-examples-sdcard]:set /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc:274
# "${EXTLINUX_ROOT_SDCARD}"
# override[stm32mp157c-ev1-m4-examples-sdcard-optee]:set /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc:275
# "${EXTLINUX_ROOT_SDCARD_OPTEE}"
# override[stm32mp151a-sdcard]:set /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc:296
# "${EXTLINUX_ROOT_SDCARD}"
# override[stm32mp151a-emcc]:set /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/meta-st/meta-st-stm32mp/conf/machine/include/st-machine-extlinux-config-stm32mp.inc:297
[...]
# pre-expansion value:
# ""
UBOOT_EXTLINUX_ROOT=""
# $UBOOT_EXTLINUX_ROOT_cubemx-nor-sdcard
UBOOT_EXTLINUX_ROOT_cubemx-nor-sdcard="root=/dev/mmcblk0p3"
# $UBOOT_EXTLINUX_ROOT_cubemx-sdcard
UBOOT_EXTLINUX_ROOT_cubemx-sdcard="root=/dev/mmcblk0p6"
# $UBOOT_EXTLINUX_ROOT_stm32mp151a-emcc
UBOOT_EXTLINUX_ROOT_stm32mp151a-emcc="\${EXTLINUX_ROOT_EMCC}"
# $UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard
UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard="root=/dev/mmcblk0p6"
So far, if I understand correctly, the override values are correctly assigned (but not the ${EXTLINUX_ROOT_EMCC} - I don't understand where the \ comes from) but the main variable is still empty.
Adding UBOOT_EXTLINUX_ROOT = "root=/dev/mmcblk1p4" at the beginning of the above file, seems to do the trick (even if before I wrote the opposite, perhaps I forgot to clear the cache?) but I don't think it's the right way to do it.
You should specify the wanted name of the machine as a target to build, i.e.:
MACHINE=stm32mp151a-sdcard bitbake <image>
This way, the UBOOT_EXTLINUX_ROOT gets the non-empty value "root=/dev/mmcblk0p6" (from the UBOOT_EXTLINUX_ROOT_stm32mp151a-sdcard variant of the variable).

Using input function with remote files in snakemake

I want to use a function to read inputs file paths from a dataframe and send them to my snakemake rule. I also have a helper function to select the remote from which to pull the files.
from snakemake.remote.GS import RemoteProvider as GSRemoteProvider
from snakemake.remote.SFTP import RemoteProvider as SFTPRemoteProvider
from os.path import join
import pandas as pd
configfile: "config.yaml"
units = pd.read_csv(config["units"]).set_index(["library", "unit"], drop=False)
TMP= join('data', 'tmp')
def access_remote(local_path):
""" Connnects to remote as defined in config file"""
provider = config['provider']
if provider == 'GS':
GS = GSRemoteProvider()
remote_path = GS.remote(join("gs://" + config['bucket'], local_path))
elif provider == 'SFTP':
SFTP = SFTPRemoteProvider(
username=config['user'],
private_key=config['ssh_key']
)
remote_path = SFTP.remote(
config['host'] + ":22" + join(base_path, local_path)
)
else:
remote_path = local_path
return remote_path
def get_fastqs(wc):
"""
Get fastq files (units) of a particular library - sample
combination from the unit sheet.
"""
fqs = units.loc[
(units.library == wc.library) &
(units.libtype == wc.libtype),
"fq1"
]
return {
"r1": list(map(access_remote, fqs.fq1.values)),
}
# Combine all fastq files from the same sample / library type combination
rule combine_units:
input: unpack(get_fastqs)
output:
r1 = join(TMP, "reads", "{library}_{libtype}.end1.fq.gz")
threads: 12
run:
shell("cat {i1} > {o1}".format(i1=input['r1'], o1=output['r1']))
My config file contains the bucket name and provider, which are passed to the function. This works as expected when running simply snakemake.
However, I would like to use the kubernetes integration, which requires passing the provider and bucket name in the command line. But when I run:
snakemake -n --kubernetes --default-remote-provider GS --default-remote-prefix bucket-name
I get this error:
ERROR :: MissingInputException in line 19 of Snakefile:
Missing input files for rule combine_units:
bucket-name/['bucket-name/lib1-unit1.end1.fastq.gz', 'bucket-name/lib1-unit2.end1.fastq.gz', 'bucket-name/lib1-unit3.end1.fastq.gz']
The bucket is applied twice (once mapped correctly to each element, and once before the whole list (which gets converted to a string). Did I miss something ? Is there a good way to work around this ?

Examples of using SCons with knitr

Are there minimal, or even larger, working examples of using SCons and knitr to generate reports from .Rmd files?
kniting an cleaning_session.Rmd file from the command line (bash shell) to derive an .html file, may be done via:
Rscript -e "library(knitr); knit('cleaning_session.Rmd')".
In this example, Rscript and instructions are fed to a Makefile:
RMDFILE=test
html :
Rscript -e "require(knitr); require(markdown); knit('$(RMDFILE).rmd', '$(RMDFILE).md'); markdownToHTML('$(RMDFILE).md', '$(RMDFILE).html', options=c('use_xhtml', 'base64_images')); browseURL(paste('file://', file.path(getwd(),'$(RMDFILE).html'), sep=''
In this answer https://stackoverflow.com/a/10945832/1172302, there is reportedly a solution using SCons. Yet, I did not test enough to make it work for me. Essentially, it would be awesome to have something like the example presented at https://tex.stackexchange.com/a/26573/8272.
[Updated] One working example is an Sconstruct file:
import os
environment = Environment(ENV=os.environ)
# define a `knitr` builder
builder = Builder(action = '/usr/local/bin/knit $SOURCE -o $TARGET',
src_suffix='Rmd')
# add builders as "Knit", "RMD"
environment.Append( BUILDERS = {'Knit' : builder} )
# define an `rmarkdown::render()` builder
builder = Builder(action = '/usr/bin/Rscript -e "rmarkdown::render(input=\'$SOURCE\', output_file=\'$TARGET\')"',
src_suffix='Rmd')
environment.Append( BUILDERS = {'RMD' : builder} )
# define source (and target files -- currently useless, since not defined above!)
# main cleaning session code
environment.RMD(source='cleaning_session.Rmd', target='cleaning_session.html')
# documentation of the Cleaning Process
environment.Knit(source='Cleaning_Process.Rmd', target='Cleaning_Process.html')
# documentation of data
environment.Knit(source='Code_Book.Rmd', target='Code_Book.html')
The first builder calls the custom script called knit. Which, in turn, takes care of the target file/extension, here being cleaning_session.html. Likely the suffix parameter is not needed altogether, in this very example.
The second builder added is Rscript -e "rmarkdown::render(\'$SOURCE\')"'.
The existence of $TARGETs (as in the example at Command wrapper) ensures SCons won't repeat work if a target file already exists.
The custom script (whose source I can't retrieve currently) is:
#!/usr/bin/env Rscript
local({
p = commandArgs(TRUE)
if (length(p) == 0L || any(c('-h', '--help') %in% p)) {
message('usage: knit input [input2 input3] [-n] [-o output output2 output3]
-h, --help to print help messages
-n, --no-convert do not convert tex to pdf, markdown to html, etc
-o output filename(s) for knit()')
q('no')
}
library(knitr)
o = match('-o', p)
if (is.na(o)) output = NA else {
output = tail(p, length(p) - o)
p = head(p, o - 1L)
}
nc = c('-n', '--no-convert')
knit_fun = if (any(nc %in% p)) {
p = setdiff(p, nc)
knit
} else {
if (length(p) == 0L) stop('no input file provided')
if (grepl('\\.(R|S)(nw|tex)$', p[1])) {
function(x, ...) knit2pdf(x, ..., clean = TRUE)
} else {
if (grepl('\\.R(md|markdown)$', p[1])) knit2html else knit
}
}
mapply(knit_fun, p, output = output, MoreArgs = list(envir = globalenv()))
})
The only thing, now, necessary is to run scons.

How can I export Jira issues to BitBucket

Ive just moved my projects code from java.net to BitBucket. But my jira issue tracking is still hosted on java.net, although BitBucket does have some options for linking to an external issue tracker I don't think I can use it for java.net, not least because I do not have the admin priviledges need to install the DVCS connector.
So I thought an alternative option would be to export and then import the issues into BitBucket issue tracker, is that possible ?
Progress so far
So I tried following the steps in both informative answers using OSX below but I hit a problem - I'm rather confused about what the script would actually be called because in the answers it talks about export.py but no such script exists with that name so I renamed the one I downloaded.
sudo easy_install pip (OSX)
pip install jira
pip install configparser
easy_install -U setuptools
Go to https://bitbucket.org/reece/rcore, select downloads tab, download zip and unzip, and rename to reece ( for some reason git clone https://bitbucket.org/reece/rcore fails with error)
cd reece/rcore
Save script as export.py in rcore subfolder
Replace iteritems with items in import.py
Replace iteritems with types/immutabledict.py
Create .config in rcore folder
Create .config/jira-issues-move-to-bitbucket.conf containing
jira-username=paultaylor
jira-hostname=https://java.net/jira/browse/JAUDIOTAGGER
jira-password=password
Run python export.py --jira-project jaudiotagger
gives
macbook:rcore paul$ python export.py --jira-project jaudiotagger
Traceback (most recent call last):
File "export.py", line 24, in <module>
import configparser
ImportError: No module named configparser
- Run python export.py --jira-project jaudiotagger
I need to run pip insdtall as root so did
sudo pip install configparser
and that worked
but now
python export.py --jira.project jaudiotagger
gives
File "export.py" line 35, in <module?
from jira.client import JIRA
ImportError: No module named jira.client
You can import issues into BitBucket, they just need to be in the appropriate format. Fortunately, Reece Hart has already written a Python script to connect to a Jira instance and export the issues.
To get the script to run I had to install the Jira Python package as well as the latest version of rcore (if you use pip you get an incompatible previous version, so you have to get the source). I also had to replace all instances of iteritems with items in the script and in rcore/types/immutabledict.py to make it work with Python 3. You will also need to fill in the dictionaries (priority_map, person_map, etc) with the values your project uses. Finally, you need a config file to exist with the connection info (see comments at the top of the script).
The basic command line usage is export.py --jira-project <project>
Once you've got the data exported, see the instructions for importing issues to BitBucket
#!/usr/bin/env python
"""extract issues from JIRA and export to a bitbucket archive
See:
https://confluence.atlassian.com/pages/viewpage.action?pageId=330796872
https://confluence.atlassian.com/display/BITBUCKET/Mark+up+comments
https://bitbucket.org/tutorials/markdowndemo/overview
2014-04-12 08:26 Reece Hart <reecehart#gmail.com>
Requires a file ~/.config/jira-issues-move-to-bitbucket.conf
with content like
[default]
jira-username=some.user
jira-hostname=somewhere.jira.com
jira-password=ur$pass
"""
import argparse
import collections
import configparser
import glob
import itertools
import json
import logging
import os
import pprint
import re
import sys
import zipfile
from jira.client import JIRA
from rcore.types.immutabledict import ImmutableDict
priority_map = {
'Critical (P1)': 'critical',
'Major (P2)': 'major',
'Minor (P3)': 'minor',
'Nice (P4)': 'trivial',
}
person_map = {
'reece.hart': 'reece',
# etc
}
issuetype_map = {
'Improvement': 'enhancement',
'New Feature': 'enhancement',
'Bug': 'bug',
'Technical task': 'task',
'Task': 'task',
}
status_map = {
'Closed': 'resolved',
'Duplicate': 'duplicate',
'In Progress': 'open',
'Open': 'new',
'Reopened': 'open',
'Resolved': 'resolved',
}
def parse_args(argv):
def sep_and_flatten(l):
# split comma-sep elements and flatten list
# e.g., ['a','b','c,d'] -> set('a','b','c','d')
return list( itertools.chain.from_iterable(e.split(',') for e in l) )
cf = configparser.ConfigParser()
cf.readfp(open(os.path.expanduser('~/.config/jira-issues-move-to-bitbucket.conf'),'r'))
ap = argparse.ArgumentParser(
description = __doc__
)
ap.add_argument(
'--jira-hostname', '-H',
default = cf.get('default','jira-hostname',fallback=None),
help = 'host name of Jira instances (used for url like https://hostname/, e.g., "instancename.jira.com")',
)
ap.add_argument(
'--jira-username', '-u',
default = cf.get('default','jira-username',fallback=None),
)
ap.add_argument(
'--jira-password', '-p',
default = cf.get('default','jira-password',fallback=None),
)
ap.add_argument(
'--jira-project', '-j',
required = True,
help = 'project key (e.g., JRA)',
)
ap.add_argument(
'--jira-issues', '-i',
action = 'append',
default = [],
help = 'issue id (e.g., JRA-9); multiple and comma-separated okay; default = all in project',
)
ap.add_argument(
'--jira-issues-file', '-I',
help = 'file containing issue ids (e.g., JRA-9)'
)
ap.add_argument(
'--jira-components', '-c',
action = 'append',
default = [],
help = 'components criterion; multiple and comma-separated okay; default = all in project',
)
ap.add_argument(
'--existing', '-e',
action = 'store_true',
default = False,
help = 'read existing archive (from export) and merge new issues'
)
opts = ap.parse_args(argv)
opts.jira_components = sep_and_flatten(opts.jira_components)
opts.jira_issues = sep_and_flatten(opts.jira_issues)
return opts
def link(url,text=None):
return "[{text}]({url})".format(url=url,text=url if text is None else text)
def reformat_to_markdown(desc):
def _indent4(mo):
i = " "
return i + mo.group(1).replace("\n",i)
def _repl_mention(mo):
return "#" + person_map[mo.group(1)]
#desc = desc.replace("\r","")
desc = re.sub("{noformat}(.+?){noformat}",_indent4,desc,flags=re.DOTALL+re.MULTILINE)
desc = re.sub(opts.jira_project+r"-(\d+)",r"issue #\1",desc)
desc = re.sub(r"\[~([^]]+)\]",_repl_mention,desc)
return desc
def fetch_issues(opts,jcl):
jql = [ 'project = ' + opts.jira_project ]
if opts.jira_components:
jql += [ ' OR '.join([ 'component = '+c for c in opts.jira_components ]) ]
if opts.jira_issues:
jql += [ ' OR '.join([ 'issue = '+i for i in opts.jira_issues ]) ]
jql_str = ' AND '.join(["("+q+")" for q in jql])
logging.info('executing query ' + jql_str)
return jcl.search_issues(jql_str,maxResults=500)
def jira_issue_to_bb_issue(opts,jcl,ji):
"""convert a jira issue to a dictionary with values appropriate for
POSTing as a bitbucket issue"""
logger = logging.getLogger(__name__)
content = reformat_to_markdown(ji.fields.description) if ji.fields.description else ''
if ji.fields.assignee is None:
resp = None
else:
resp = person_map[ji.fields.assignee.name]
reporter = person_map[ji.fields.reporter.name]
jiw = jcl.watchers(ji.key)
watchers = [ person_map[u.name] for u in jiw.watchers ] if jiw else []
milestone = None
if ji.fields.fixVersions:
vnames = [ v.name for v in ji.fields.fixVersions ]
milestone = vnames[0]
if len(vnames) > 1:
logger.warn("{ji.key}: bitbucket issues may have only 1 milestone (JIRA fixVersion); using only first ({f}) and ignoring rest ({r})".format(
ji=ji, f=milestone, r=",".join(vnames[1:])))
issue_id = extract_issue_number(ji.key)
bbi = {
'status': status_map[ji.fields.status.name],
'priority': priority_map[ji.fields.priority.name],
'kind': issuetype_map[ji.fields.issuetype.name],
'content_updated_on': ji.fields.created,
'voters': [],
'title': ji.fields.summary,
'reporter': reporter,
'component': None,
'watchers': watchers,
'content': content,
'assignee': resp,
'created_on': ji.fields.created,
'version': None, # ?
'edited_on': None,
'milestone': milestone,
'updated_on': ji.fields.updated,
'id': issue_id,
}
return bbi
def jira_comment_to_bb_comment(opts,jcl,jc):
bbc = {
'content': reformat_to_markdown(jc.body),
'created_on': jc.created,
'id': int(jc.id),
'updated_on': jc.updated,
'user': person_map[jc.author.name],
}
return bbc
def extract_issue_number(jira_issue_key):
return int(jira_issue_key.split('-')[-1])
def jira_key_to_bb_issue_tag(jira_issue_key):
return 'issue #' + str(extract_issue_number(jira_issue_key))
def jira_link_text(jk):
return link("https://invitae.jira.com/browse/"+jk,jk) + " (Invitae access required)"
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
opts = parse_args(sys.argv[1:])
dir_name = opts.jira_project
if opts.jira_components:
dir_name += '-' + ','.join(opts.jira_components)
if opts.jira_issues_file:
issues = [i.strip() for i in open(opts.jira_issues_file,'r')]
logger.info("added {n} issues from {opts.jira_issues_file} to issues list".format(n=len(issues),opts=opts))
opts.jira_issues += issues
opts.dir = os.path.join('/','tmp',dir_name)
opts.att_rel_dir = 'attachments'
opts.att_abs_dir = os.path.join(opts.dir,opts.att_rel_dir)
opts.json_fn = os.path.join(opts.dir,'db-1.0.json')
if not os.path.isdir(opts.att_abs_dir):
os.makedirs(opts.att_abs_dir)
opts.jira_issues = list(set(opts.jira_issues)) # distinctify
jcl = JIRA({'server': 'https://{opts.jira_hostname}/'.format(opts=opts)},
basic_auth=(opts.jira_username,opts.jira_password))
if opts.existing:
issues_db = json.load(open(opts.json_fn,'r'))
existing_ids = [ i['id'] for i in issues_db['issues'] ]
logger.info("read {n} issues from {fn}".format(n=len(existing_ids),fn=opts.json_fn))
else:
issues_db = dict()
issues_db['meta'] = {
'default_milestone': None,
'default_assignee': None,
'default_kind': "bug",
'default_component': None,
'default_version': None,
}
issues_db['attachments'] = []
issues_db['comments'] = []
issues_db['issues'] = []
issues_db['logs'] = []
issues_db['components'] = [ {'name':v.name} for v in jcl.project_components(opts.jira_project) ]
issues_db['milestones'] = [ {'name':v.name} for v in jcl.project_versions(opts.jira_project) ]
issues_db['versions'] = issues_db['milestones']
# bb_issue_map: bb issue # -> bitbucket issue
bb_issue_map = ImmutableDict( (i['id'],i) for i in issues_db['issues'] )
# jk_issue_map: jira key -> bitbucket issue
# contains only items migrated from JIRA (i.e., not preexisting issues with --existing)
jk_issue_map = ImmutableDict()
# issue_links is a dict of dicts of lists, using JIRA keys
# e.g., links['CORE-135']['depends on'] = ['CORE-137']
issue_links = collections.defaultdict(lambda: collections.defaultdict(lambda: []))
issues = fetch_issues(opts,jcl)
logger.info("fetch {n} issues from JIRA".format(n=len(issues)))
for ji in issues:
# Pfft. Need to fetch the issue again due to bug in JIRA.
# See https://bitbucket.org/bspeakmon/jira-python/issue/47/, comment on 2013-10-01 by ssonic
ji = jcl.issue(ji.key,expand="attachments,comments")
# create the issue
bbi = jira_issue_to_bb_issue(opts,jcl,ji)
issues_db['issues'] += [bbi]
bb_issue_map[bbi['id']] = bbi
jk_issue_map[ji.key] = bbi
issue_links[ji.key]['imported from'] = [jira_link_text(ji.key)]
# add comments
for jc in ji.fields.comment.comments:
bbc = jira_comment_to_bb_comment(opts,jcl,jc)
bbc['issue'] = bbi['id']
issues_db['comments'] += [bbc]
# add attachments
for ja in ji.fields.attachment:
att_rel_path = os.path.join(opts.att_rel_dir,ja.id)
att_abs_path = os.path.join(opts.att_abs_dir,ja.id)
if not os.path.exists(att_abs_path):
open(att_abs_path,'w').write(ja.get())
logger.info("Wrote {att_abs_path}".format(att_abs_path=att_abs_path))
bba = {
"path": att_rel_path,
"issue": bbi['id'],
"user": person_map[ja.author.name],
"filename": ja.filename,
}
issues_db['attachments'] += [bba]
# parent-child is task-subtask
if hasattr(ji.fields,'parent'):
issue_links[ji.fields.parent.key]['subtasks'].append(jira_key_to_bb_issue_tag(ji.key))
issue_links[ji.key]['parent task'].append(jira_key_to_bb_issue_tag(ji.fields.parent.key))
# add links
for il in ji.fields.issuelinks:
if hasattr(il,'outwardIssue'):
issue_links[ji.key][il.type.outward].append(jira_key_to_bb_issue_tag(il.outwardIssue.key))
elif hasattr(il,'inwardIssue'):
issue_links[ji.key][il.type.inward].append(jira_key_to_bb_issue_tag(il.inwardIssue.key))
logger.info("migrated issue {ji.key}: {ji.fields.summary} ({components})".format(
ji=ji,components=','.join(c.name for c in ji.fields.components)))
# append links section to content
# this section shows both task-subtask and "issue link" relationships
for src,dstlinks in issue_links.iteritems():
if src not in jk_issue_map:
logger.warn("issue {src}, with issue_links, not in jk_issue_map; skipping".format(src=src))
continue
links_block = "Links\n=====\n"
for desc,dsts in sorted(dstlinks.iteritems()):
links_block += "* **{desc}**: {links} \n".format(desc=desc,links=", ".join(dsts))
if jk_issue_map[src]['content']:
jk_issue_map[src]['content'] += "\n\n" + links_block
else:
jk_issue_map[src]['content'] = links_block
id_counts = collections.Counter(i['id'] for i in issues_db['issues'])
dupes = [ k for k,cnt in id_counts.iteritems() if cnt>1 ]
if dupes:
raise RuntimeError("{n} issue ids appear more than once from existing {opts.json_fn}".format(
n=len(dupes),opts=opts))
json.dump(issues_db,open(opts.json_fn,'w'))
logger.info("wrote {n} issues to {opts.json_fn}".format(n=len(id_counts),opts=opts))
# write zipfile
os.chdir(opts.dir)
with zipfile.ZipFile(opts.dir + '.zip','w') as zf:
for fn in ['db-1.0.json']+glob.glob('attachments/*'):
zf.write(fn)
logger.info("added {fn} to archive".format(fn=fn))
NOTE: I'm writing a new answer because writing this in a comment would be horrible, but most of the credit goes to #Turch's answer.
My steps (in OSX and Debian machines, both worked fine):
apt-get install python-pip (Debian) or sudo easy_install pip (OSX)
pip install jira
pip install configparser
easy_install -U setuptools (not sure if really needed)
Download or clone the source code from https://bitbucket.org/reece/rcore/ in your home folder, for example. Note: don't download using pip, it will get the 0.0.2 version and you need the 0.0.3.
Download the Python script created by Reece, mentioned by #Turch, and place it inside of the rcore folder.
Follow the instructions by #Turch: I also had to replace all instances of iteritems with items in the script and in rcore/types/immutabledict.py to make it work with Python 3. You will also need to fill in the dictionaries (priority_map, person_map, etc) with the values your project uses. Finally, you need a config file to exist with the connection info (see comments at the top of the script). Note: I used hostname like jira.domain.com (no http or https).
(This change did the trick for me) I had to change part of the line 250 from 'https://{opts.jira_hostname}/' to 'http://{opts.jira_hostname}/'
To finish, run the script like #Turch mentioned: The basic command line usage is export.py --jira-project <project>
The file was placed in /tmp/.zip for me.
The file was perfectly accepted in the BitBucket importer today.
Hooray for Reece and Turch! Thanks guys!

Scala command line parser using Scallop

I'm fairly new to Scala and need to build a really simple command line parser which provides something like the following which I created using JRuby in a few minutes:-
java -jar demo.jar --help
Command Line Example Application
Example: java -jar demo.jar --dn "CN=Test" --nde-url "http://www.example.com" --password "password"
For usage see below:
-n http://www.example.com
-p, --password set the password
-c, --capi set add to Windows key-store
-h, --help Show this message
-v, --version Print version
Scallop looks like it will do the trick, but I can't seem to find a simple example that works! All of the examples I've found seem to be fragmented and don't work for some reason or other.
UPDATE
I found this example which works, but I'm not sure how to bind it into the actual args within the main method.
import org.rogach.scallop._;
object cmdlinetest {
def main(args: Array[String])
val opts = Scallop(List("-d","--num-limbs","1"))
.version("test 1.2.3 (c) 2012 Mr Placeholder")
.banner("""Usage: test [OPTION]... [pet-name]
|test is an awesome program, which does something funny
|Options:
|""".stripMargin)
.footer("\nFor all other tricks, consult the documentation!")
.opt[Boolean]("donkey", descr = "use donkey mode")
.opt("monkeys", default = Some(2), short = 'm')
.opt[Int]("num-limbs", 'k',
"number of libms", required = true)
.opt[List[Double]]("params")
.opt[String]("debug", hidden = true)
.props[String]('D',"some key-value pairs")
// you can add parameters a bit later
.args(List("-Dalpha=1","-D","betta=2","gamma=3", "Pigeon"))
.trailArg[String]("pet name")
.verify
println(opts.help)
}
}
Well, I'll try to add more examples :)
In this case, it would be much better to use ScallopConf:
import org.rogach.scallop._
object Main extends App {
val opts = new ScallopConf(args) {
banner("""
NDE/SCEP Certificate enrollment prototype
Example: java -jar demo.jar --dn CN=Test --nde-url http://www.example.com --password password
For usage see below:
""")
val ndeUrl = opt[String]("nde-url")
val password = opt[String]("password", descr = "set the password")
val capi = toggle("capi", prefix = "no-", descrYes = "enable adding to Windows key-store", descrNo = "disable adding to Windows key-store")
val version = opt[Boolean]("version", noshort = true, descr = "Print version")
val help = opt[Boolean]("help", noshort = true, descr = "Show this message")
}
println(opts.password())
}
It prints:
$ java -jar demo.jar --help
NDE/SCEP Certificate enrollment prototype
Example: java -jar demo.jar --dn CN=Test --nde-url http://www.example.com --password password
For usage see below:
-c, --capi enable adding to Windows key-store
--no-capi disable adding to Windows key-store
--help Show this message
-n, --nde-url <arg>
-p, --password <arg> set the password
--version Print version
Did you read the documentation? It looks like all you have to do is call get for each option you want:
def get [A] (name: String)(implicit m: Manifest[A]): Option[A]
It looks like you might need to provide the expected return type in the method call. Try something like this:
val donkey = opts.get[Boolean]("donkey")
val numLimbs = opts.get[Int]("num-limbs")
If you're just looking for a quick and dirty way to parse command line arguments, you can use pirate, an extremely barebones way to parse arguments. Here is what it would look like to handle the usage you describe above:
import com.mosesn.pirate.Pirate
object Main {
def main(commandLineArgs: Array[String]) {
val args = Pirate("[ -n string ] [ -p string ] [ -chv ]")("-n whatever -c".split(" "))
val c = args.flags.contains('c')
val v = args.flags.contains('v')
val h = args.flags.contains('h')
val n = args.strings.get("n")
val p = args.strings.get("p")
println(Seq(c, v, h, n, p))
}
}
Of course, for your program, you would pass commandLineArgs instead of "-n whatever -c".
Unfortunately, pirate does not yet support GNU style arguments, nor the version or help text options.