I am working on a Python project and recently started using poetry. I was originally working on the project using macOS 11.0, but as I near completion, I wanted to test it on a Linux workstation. I use Github for my repository, and so, I am able to clone the repo easily. However, that is where the simple part ends. I have used both pyenv and conda for the virtual environment on macOS, but when I clone the repo onto the workstation, I set up an environment and then try the following command:
poetry shell
poetry install
I receive the following error after poetry install:
ParseVersionError
Unable to parse "at20RC5+54.g5702a232fe.dirty".
at ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/semver/version.py:211 in parse
207│ except TypeError:
208│ match = None
209│
210│ if match is None:
→ 211│ raise ParseVersionError('Unable to parse "{}".'.format(text))
212│
213│ text = text.rstrip(".")
214│
215│ major = int(match.group(1))
I have tried poetry lock with the same result, and I have even deleted poetry.lock and attempted poetry lock with no success.
I intend to build and publish it when all is said and done, but because I eventually want to add features that my Mac does not have (e.g., CUDA), I want to code on the workstation as well.
Any help will be appreciated.
Update -- 03/27/21
pyproject.toml
[tool.poetry]
name = "qaa"
version = "0.1.0"
description = "Quasi-Anharmonic Analysis"
authors = ["Timothy H. Click <tclick#okstate.edu>"]
license = "BSD-3-Clause"
readme = "README.rst"
homepage = "https://github.com/tclick/qaa"
repository = "https://github.com/tclick/qaa"
documentation = "https://qaa.readthedocs.io"
classifiers = [
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
]
[tool.poetry.urls]
Changelog = "https://github.com/tclick/qaa/releases"
[tool.poetry.dependencies]
python = "^3.8"
click = "^7.0"
numpy = "^1.20.1"
scipy = "^1.6.1"
matplotlib = "^3.3.4"
seaborn = "^0.11.1"
scikit-learn = "^0.24.1"
pandas = "^1.2.3"
netCDF4 = "^1.5.6"
mdtraj = "^1.9.5"
[tool.poetry.dev-dependencies]
pytest = "^6.2.2"
pytest-cache = "^1.0"
pytest-click = "^1.0.2"
pytest-console-scripts = "^1.1.0"
pytest-cov = "^2.11.1"
pytest-flake8 = "^1.0.7"
pytest-mock = "^3.5.1"
pytest-pep8 = "^1.0.6"
pytest-randomly = "^3.5.0"
coverage = {extras = ["toml"], version = "^5.3"}
safety = "^1.9.0"
mypy = "^0.812"
typeguard = "^2.9.1"
xdoctest = {extras = ["colors"], version = "^0.15.0"}
sphinx = "^3.3.1"
sphinx-autobuild = "^2020.9.1"
pre-commit = "^2.8.2"
flake8 = "^3.8.4"
black = "^20.8b1"
flake8-bandit = "^2.1.2"
flake8-bugbear = "^21.3.2"
flake8-docstrings = "^1.5.0"
flake8-rst-docstrings = "^0.0.14"
pep8-naming = "^0.11.1"
darglint = "^1.5.5"
reorder-python-imports = "^2.3.6"
pre-commit-hooks = "^3.3.0"
sphinx-rtd-theme = "^0.5.0"
sphinx-click = "^2.5.0"
Pygments = "^2.7.2"
ipython = "^7.21.0"
isort = "^5.7.0"
towncrier = "^19.2.0"
nox = "^2020.12.31"
pytest-coverage = "^0.0"
nox-poetry = "^0.8.4"
numpydoc = "^1.1.0"
codecov = "^2.1.11"
flake8-black = "^0.2.1"
flake8-import-order = "^0.18.1"
[tool.poetry.scripts]
qaa = "qaa.__main__:main"
update -- 03/29/21
This is the error received when running poetry lock -vvv
Using virtualenv: /home/tclick/.cache/pypoetry/virtualenvs/qaa-VNW0yB_S-py3.8
Stack trace:
10 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py:131 in run
129│ parsed_args = resolved_command.args
130│
→ 131│ status_code = command.handle(parsed_args, io)
132│ except KeyboardInterrupt:
133│ status_code = 1
9 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:120 in handle
118│ def handle(self, args, io): # type: (Args, IO) -> int
119│ try:
→ 120│ status_code = self._do_handle(args, io)
121│ except KeyboardInterrupt:
122│ if io.is_debug():
8 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:163 in _do_handle
161│ if self._dispatcher and self._dispatcher.has_listeners(PRE_HANDLE):
162│ event = PreHandleEvent(args, io, self)
→ 163│ self._dispatcher.dispatch(PRE_HANDLE, event)
164│
165│ if event.is_handled():
7 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/event/event_dispatcher.py:22 in dispatch
20│
21│ if listeners:
→ 22│ self._do_dispatch(listeners, event_name, event)
23│
24│ return event
6 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/event/event_dispatcher.py:89 in _do_dispatch
87│ break
88│
→ 89│ listener(event, event_name, self)
90│
91│ def _sort_listeners(self, event_name): # type: (str) -> None
5 ~/.poetry/lib/poetry/console/config/application_config.py:141 in set_installer
139│
140│ poetry = command.poetry
→ 141│ installer = Installer(
142│ event.io,
143│ command.env,
4 ~/.poetry/lib/poetry/installation/installer.py:65 in __init__
63│ self._installer = self._get_installer()
64│ if installed is None:
→ 65│ installed = self._get_installed()
66│
67│ self._installed_repository = installed
3 ~/.poetry/lib/poetry/installation/installer.py:561 in _get_installed
559│
560│ def _get_installed(self): # type: () -> InstalledRepository
→ 561│ return InstalledRepository.load(self._env)
562│
2 ~/.poetry/lib/poetry/repositories/installed_repository.py:118 in load
116│ path = Path(str(distribution._path))
117│ version = distribution.metadata["version"]
→ 118│ package = Package(name, version, version)
119│ package.description = distribution.metadata.get("summary", "")
120│
1 ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/packages/package.py:61 in __init__
59│
60│ if not isinstance(version, Version):
→ 61│ self._version = Version.parse(version)
62│ self._pretty_version = pretty_version or version
63│ else:
ParseVersionError
Unable to parse "at20RC5+54.g5702a232fe.dirty".
at ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/semver/version.py:206 in parse
202│ except TypeError:
203│ match = None
204│
205│ if match is None:
→ 206│ raise ParseVersionError('Unable to parse "{}".'.format(text))
207│
208│ text = text.rstrip(".")
209│
210│ major = int(match.group(1))
I think you should NOT run poetry shell first. It's used only when poetry project and its dependencies are installed.
poetry install should work nice in the project directory with pyproject.toml
Possible help:
ensure that poetry installed and path to poetry executable in PATH
environment variable;
check that python 3.8+ is installed (from your dependencies). If
you use pyenv check that path to this version is available for
terminal.
I'm having trouble getting VSCode to load build targets for a Java project. The error message that I get is the following:
Command failed: bazel --output_base=/var/folders/cj/1fv063dx1772zbpnfpzvj_c00000gn/T/5b71fd39758e7fd8e710c281fbbdda92 query ...:* --output=package Loading: 0 packages loaded ERROR: error loading package 'bazel-test/external/rules_jvm_external': cannot load '//:private/versions.bzl': no such file Loading: 12 packages loaded currently loading: bazel-test/external/local_config_cc ... (6 packages) Loading: 12 packages loaded currently loading: bazel-test/external/local_config_cc ... (6 packages)
My WORKSPACE file is as follows:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "io_bazel_rules_docker",
sha256 = "4521794f0fba2e20f3bf15846ab5e01d5332e587e9ce81629c7f96c793bb7036",
strip_prefix = "rules_docker-0.14.4",
urls = ["https://github.com/bazelbuild/rules_docker/releases/download/v0.14.4/rules_docker-v0.14.4.tar.gz"],
)
load(
"#io_bazel_rules_docker//repositories:repositories.bzl",
container_repositories = "repositories",
)
container_repositories()
load("#io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")
container_deps()
load("#io_bazel_rules_docker//repositories:pip_repositories.bzl", "pip_deps")
pip_deps()
load(
"#io_bazel_rules_docker//java:image.bzl",
_java_image_repos = "repositories",
)
_java_image_repos()
load(
"#io_bazel_rules_docker//container:container.bzl",
"container_pull",
)
container_pull(
name = "openjdk11_slim",
registry = "index.docker.io",
repository = "library/openjdk",
tag = "11-slim"
)
RULES_JVM_EXTERNAL_TAG = "3.3"
RULES_JVM_EXTERNAL_SHA = "d85951a92c0908c80bd8551002d66cb23c3434409c814179c0ff026b53544dab"
http_archive(
name = "rules_jvm_external",
strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG,
sha256 = RULES_JVM_EXTERNAL_SHA,
url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG,
)
load("#rules_jvm_external//:defs.bzl", "maven_install")
maven_install(
artifacts = [
# Accessed as dependency: #maven://io_vertx_vertx_core
"io.vertx:vertx-core:3.9.2",
# Accessed as dependency: #maven://io_vertx_vertx_web
"io.vertx:vertx-web:3.9.2",
# Accessed as dependency: #maven://io_vertx_vertx_rx_java2
"io.vertx:vertx-rx-java2:3.9.2",
],
repositories = [
"https://repo1.maven.org/maven2",
]
)
I don't see anything obviously wrong with my WORKSPACE file, so I'm tempted to assume the problem is with VSCode or in the BUILD files in one of my external dependencies. Other than that the project contains a single Hello.java and its accompanying java_binary rule. If I run myself the same query for all packages I get the following:
% bazel query ...:\* --output=package
src/main/java
Loading: 0 packages loaded
In other words, there's no error. Any ideas how I could try to debug or fix this? I just installed everything on the laptop, so the software versions are:
Bazel version: 3.5.0
VSCode versio: 1.48.2
Following the comment ignoring the convenience symlink solved it for me:
echo bazel-`basename ${PWD}` >> .bazelignore
I got around this issue by changing the Query Expression in Bazel extension Settings from
...:*
to
//< directory name >/...:*
I am creating a bazel python project which depends on apache_beam sdk. In the WORKSPACE file of my project, I have downloaded apache_beam zip file via http_archive.
Whenever I am trying to access apache_beam.testing sub-package in my project, I am getting an error
ModuleNotFoundError: No module named 'apache_beam.testing'
import apache_beam is working fine though.
WORKSPACE:
http_archive(
name = "apache_beam",
urls = ["https://github.com/apache/beam/archive/master.zip"],
strip_prefix = 'beam-master/sdks/python',
build_file = str(Label("//third_party:apache_beam.BUILD")),
)
apache_beam.BUILD:
py_library(
name = "apache_beam",
srcs = glob(["apache_beam/**/*.py"]),
visibility = ["//visibility:public"],
)
project's BUILD file:
py_test(
name = "project",
srcs = ["project.py"],
deps = [
"#apache_beam//:apache_beam",
],
project.py file:
from apache_beam.testing import util
In apache_beam.BUILD, you need to set the imports attribute:
py_library(
name = "apache_beam",
srcs = glob(["apache_beam/**/*.py"]),
visibility = ["//visibility:public"],
imports = "apache_beam",
)
I have a pytest testing project running selenium tests that has a structure like:
ProjRoot
|
|_Pytest.ini
|_____________TestFolderA
| |
| |_test_folderA_tests1.py
| |_test_folderA_tests2.py
|
|____________TestFolderB
| |
| |_test_folderB_test1.py
| |_test_folderA_tests2.py
|
|
|___________TestHelperModules
| |
| |_VariousTestHelperModules
|
|____________DriversAndTools
|___(contains chromedriver.exe, firefox profile folder etc)
I have a confTest.py file which I currently run in the ProjRoot, which I use as a setup and tear down for establishing the browser session for each test that is run. It runs each test twice. Once for Chrome and once for Firefox. In my tests I just utilise the resulting driver fixture. The conftest file is as below:
#conftest.py
import pytest
import os
import rootdir_ref
from selenium.webdriver.common.keys import Keys
import time
from webdriverwrapper.pytest import *
from webdriverwrapper import Chrome
from webdriverwrapper import DesiredCapabilities
from webdriverwrapper import Firefox
from webdriverwrapper import FirefoxProfile
#when running tests from command line we should be able to pass --url=www..... for a different website, check what order these definitions need to be in
def pytest_addoption(parser):
parser.addoption('--url', default='https://test1.testsite.com.au')
#pytest.fixture(scope='function')
def url(request):
return request.config.option.url
browsers = {
'firefox': Firefox,
'chrome': Chrome,
}
#pytest.fixture(scope='function',
params=browsers.keys())
def browser(request):
if request.param == 'firefox':
firefox_capabilities = DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
firefox_capabilities['handleAlerts'] = True
theRootDir = os.path.dirname(rootdir_ref.__file__)
ffProfilePath = os.path.join(theRootDir, 'DriversAndTools', 'FirefoxSeleniumProfile')
geckoDriverPath = os.path.join(theRootDir, 'DriversAndTools', 'geckodriver.exe')
profile = FirefoxProfile(profile_directory=ffProfilePath)
print (ffProfilePath)
print (geckoDriverPath)
b = browsers[request.param](firefox_profile=profile, capabilities=firefox_capabilities, executable_path=geckoDriverPath)
elif request.param == 'chrome':
desired_cap = DesiredCapabilities.CHROME
desired_cap['chromeOptions'] = {}
desired_cap['chromeOptions']['args'] = ['--disable-plugins', '--disable-extensions']
theRootDir = os.path.dirname(rootdir_ref.__file__)
chromeDriverPath = os.path.join(theRootDir, 'DriversAndTools', 'chromedriver.exe')
b = browsers[request.param](chromeDriverPath)
else:
b = browsers[request.param]()
request.addfinalizer(lambda *args: b.quit())
return b
#pytest.fixture(scope='function')
def driver(browser, url):
driver = browser
driver.maximize_window()
driver.get(url)
return driver
What I’d like to do is have a conftest file in each Test Folder instead of the ProjRoot. But if I take this existing conftest file and put it in each test folder and then run pytest from the project root using
python –m pytest
letting pytest pickup the test directories from pytest.ini (expecting the test folders to run with their respectively contained conftest files) I have issues with the parser.addoption --url already having been added. The end of the error message is:
ClientScripts\conftest.py:19: in pytest_addoption
parser.addoption('--url', default='https://test1.coreplus.com.au/coreplus01')
..\..\..\VirtEnv\VirtEnv\lib\site-packages\_pytest\config.py:521: in addoption
self._anonymous.addoption(*opts, **attrs)
..\..\..\VirtEnv\VirtEnv\lib\site-packages\_pytest\config.py:746: in addoption
raise ValueError("option names %s already added" % conflict)
E ValueError: option names {'--url'} already added
The purpose of the --url addoption is so I can override the defaults in the conftest file at commandline if I want to point them all to a different url at the same time, but otherwise let them default to running to different url's as specified in their conftest files.
I had a similar issue.
Error was gone after removing all cached files and venv.
Ive just moved my projects code from java.net to BitBucket. But my jira issue tracking is still hosted on java.net, although BitBucket does have some options for linking to an external issue tracker I don't think I can use it for java.net, not least because I do not have the admin priviledges need to install the DVCS connector.
So I thought an alternative option would be to export and then import the issues into BitBucket issue tracker, is that possible ?
Progress so far
So I tried following the steps in both informative answers using OSX below but I hit a problem - I'm rather confused about what the script would actually be called because in the answers it talks about export.py but no such script exists with that name so I renamed the one I downloaded.
sudo easy_install pip (OSX)
pip install jira
pip install configparser
easy_install -U setuptools
Go to https://bitbucket.org/reece/rcore, select downloads tab, download zip and unzip, and rename to reece ( for some reason git clone https://bitbucket.org/reece/rcore fails with error)
cd reece/rcore
Save script as export.py in rcore subfolder
Replace iteritems with items in import.py
Replace iteritems with types/immutabledict.py
Create .config in rcore folder
Create .config/jira-issues-move-to-bitbucket.conf containing
jira-username=paultaylor
jira-hostname=https://java.net/jira/browse/JAUDIOTAGGER
jira-password=password
Run python export.py --jira-project jaudiotagger
gives
macbook:rcore paul$ python export.py --jira-project jaudiotagger
Traceback (most recent call last):
File "export.py", line 24, in <module>
import configparser
ImportError: No module named configparser
- Run python export.py --jira-project jaudiotagger
I need to run pip insdtall as root so did
sudo pip install configparser
and that worked
but now
python export.py --jira.project jaudiotagger
gives
File "export.py" line 35, in <module?
from jira.client import JIRA
ImportError: No module named jira.client
You can import issues into BitBucket, they just need to be in the appropriate format. Fortunately, Reece Hart has already written a Python script to connect to a Jira instance and export the issues.
To get the script to run I had to install the Jira Python package as well as the latest version of rcore (if you use pip you get an incompatible previous version, so you have to get the source). I also had to replace all instances of iteritems with items in the script and in rcore/types/immutabledict.py to make it work with Python 3. You will also need to fill in the dictionaries (priority_map, person_map, etc) with the values your project uses. Finally, you need a config file to exist with the connection info (see comments at the top of the script).
The basic command line usage is export.py --jira-project <project>
Once you've got the data exported, see the instructions for importing issues to BitBucket
#!/usr/bin/env python
"""extract issues from JIRA and export to a bitbucket archive
See:
https://confluence.atlassian.com/pages/viewpage.action?pageId=330796872
https://confluence.atlassian.com/display/BITBUCKET/Mark+up+comments
https://bitbucket.org/tutorials/markdowndemo/overview
2014-04-12 08:26 Reece Hart <reecehart#gmail.com>
Requires a file ~/.config/jira-issues-move-to-bitbucket.conf
with content like
[default]
jira-username=some.user
jira-hostname=somewhere.jira.com
jira-password=ur$pass
"""
import argparse
import collections
import configparser
import glob
import itertools
import json
import logging
import os
import pprint
import re
import sys
import zipfile
from jira.client import JIRA
from rcore.types.immutabledict import ImmutableDict
priority_map = {
'Critical (P1)': 'critical',
'Major (P2)': 'major',
'Minor (P3)': 'minor',
'Nice (P4)': 'trivial',
}
person_map = {
'reece.hart': 'reece',
# etc
}
issuetype_map = {
'Improvement': 'enhancement',
'New Feature': 'enhancement',
'Bug': 'bug',
'Technical task': 'task',
'Task': 'task',
}
status_map = {
'Closed': 'resolved',
'Duplicate': 'duplicate',
'In Progress': 'open',
'Open': 'new',
'Reopened': 'open',
'Resolved': 'resolved',
}
def parse_args(argv):
def sep_and_flatten(l):
# split comma-sep elements and flatten list
# e.g., ['a','b','c,d'] -> set('a','b','c','d')
return list( itertools.chain.from_iterable(e.split(',') for e in l) )
cf = configparser.ConfigParser()
cf.readfp(open(os.path.expanduser('~/.config/jira-issues-move-to-bitbucket.conf'),'r'))
ap = argparse.ArgumentParser(
description = __doc__
)
ap.add_argument(
'--jira-hostname', '-H',
default = cf.get('default','jira-hostname',fallback=None),
help = 'host name of Jira instances (used for url like https://hostname/, e.g., "instancename.jira.com")',
)
ap.add_argument(
'--jira-username', '-u',
default = cf.get('default','jira-username',fallback=None),
)
ap.add_argument(
'--jira-password', '-p',
default = cf.get('default','jira-password',fallback=None),
)
ap.add_argument(
'--jira-project', '-j',
required = True,
help = 'project key (e.g., JRA)',
)
ap.add_argument(
'--jira-issues', '-i',
action = 'append',
default = [],
help = 'issue id (e.g., JRA-9); multiple and comma-separated okay; default = all in project',
)
ap.add_argument(
'--jira-issues-file', '-I',
help = 'file containing issue ids (e.g., JRA-9)'
)
ap.add_argument(
'--jira-components', '-c',
action = 'append',
default = [],
help = 'components criterion; multiple and comma-separated okay; default = all in project',
)
ap.add_argument(
'--existing', '-e',
action = 'store_true',
default = False,
help = 'read existing archive (from export) and merge new issues'
)
opts = ap.parse_args(argv)
opts.jira_components = sep_and_flatten(opts.jira_components)
opts.jira_issues = sep_and_flatten(opts.jira_issues)
return opts
def link(url,text=None):
return "[{text}]({url})".format(url=url,text=url if text is None else text)
def reformat_to_markdown(desc):
def _indent4(mo):
i = " "
return i + mo.group(1).replace("\n",i)
def _repl_mention(mo):
return "#" + person_map[mo.group(1)]
#desc = desc.replace("\r","")
desc = re.sub("{noformat}(.+?){noformat}",_indent4,desc,flags=re.DOTALL+re.MULTILINE)
desc = re.sub(opts.jira_project+r"-(\d+)",r"issue #\1",desc)
desc = re.sub(r"\[~([^]]+)\]",_repl_mention,desc)
return desc
def fetch_issues(opts,jcl):
jql = [ 'project = ' + opts.jira_project ]
if opts.jira_components:
jql += [ ' OR '.join([ 'component = '+c for c in opts.jira_components ]) ]
if opts.jira_issues:
jql += [ ' OR '.join([ 'issue = '+i for i in opts.jira_issues ]) ]
jql_str = ' AND '.join(["("+q+")" for q in jql])
logging.info('executing query ' + jql_str)
return jcl.search_issues(jql_str,maxResults=500)
def jira_issue_to_bb_issue(opts,jcl,ji):
"""convert a jira issue to a dictionary with values appropriate for
POSTing as a bitbucket issue"""
logger = logging.getLogger(__name__)
content = reformat_to_markdown(ji.fields.description) if ji.fields.description else ''
if ji.fields.assignee is None:
resp = None
else:
resp = person_map[ji.fields.assignee.name]
reporter = person_map[ji.fields.reporter.name]
jiw = jcl.watchers(ji.key)
watchers = [ person_map[u.name] for u in jiw.watchers ] if jiw else []
milestone = None
if ji.fields.fixVersions:
vnames = [ v.name for v in ji.fields.fixVersions ]
milestone = vnames[0]
if len(vnames) > 1:
logger.warn("{ji.key}: bitbucket issues may have only 1 milestone (JIRA fixVersion); using only first ({f}) and ignoring rest ({r})".format(
ji=ji, f=milestone, r=",".join(vnames[1:])))
issue_id = extract_issue_number(ji.key)
bbi = {
'status': status_map[ji.fields.status.name],
'priority': priority_map[ji.fields.priority.name],
'kind': issuetype_map[ji.fields.issuetype.name],
'content_updated_on': ji.fields.created,
'voters': [],
'title': ji.fields.summary,
'reporter': reporter,
'component': None,
'watchers': watchers,
'content': content,
'assignee': resp,
'created_on': ji.fields.created,
'version': None, # ?
'edited_on': None,
'milestone': milestone,
'updated_on': ji.fields.updated,
'id': issue_id,
}
return bbi
def jira_comment_to_bb_comment(opts,jcl,jc):
bbc = {
'content': reformat_to_markdown(jc.body),
'created_on': jc.created,
'id': int(jc.id),
'updated_on': jc.updated,
'user': person_map[jc.author.name],
}
return bbc
def extract_issue_number(jira_issue_key):
return int(jira_issue_key.split('-')[-1])
def jira_key_to_bb_issue_tag(jira_issue_key):
return 'issue #' + str(extract_issue_number(jira_issue_key))
def jira_link_text(jk):
return link("https://invitae.jira.com/browse/"+jk,jk) + " (Invitae access required)"
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
opts = parse_args(sys.argv[1:])
dir_name = opts.jira_project
if opts.jira_components:
dir_name += '-' + ','.join(opts.jira_components)
if opts.jira_issues_file:
issues = [i.strip() for i in open(opts.jira_issues_file,'r')]
logger.info("added {n} issues from {opts.jira_issues_file} to issues list".format(n=len(issues),opts=opts))
opts.jira_issues += issues
opts.dir = os.path.join('/','tmp',dir_name)
opts.att_rel_dir = 'attachments'
opts.att_abs_dir = os.path.join(opts.dir,opts.att_rel_dir)
opts.json_fn = os.path.join(opts.dir,'db-1.0.json')
if not os.path.isdir(opts.att_abs_dir):
os.makedirs(opts.att_abs_dir)
opts.jira_issues = list(set(opts.jira_issues)) # distinctify
jcl = JIRA({'server': 'https://{opts.jira_hostname}/'.format(opts=opts)},
basic_auth=(opts.jira_username,opts.jira_password))
if opts.existing:
issues_db = json.load(open(opts.json_fn,'r'))
existing_ids = [ i['id'] for i in issues_db['issues'] ]
logger.info("read {n} issues from {fn}".format(n=len(existing_ids),fn=opts.json_fn))
else:
issues_db = dict()
issues_db['meta'] = {
'default_milestone': None,
'default_assignee': None,
'default_kind': "bug",
'default_component': None,
'default_version': None,
}
issues_db['attachments'] = []
issues_db['comments'] = []
issues_db['issues'] = []
issues_db['logs'] = []
issues_db['components'] = [ {'name':v.name} for v in jcl.project_components(opts.jira_project) ]
issues_db['milestones'] = [ {'name':v.name} for v in jcl.project_versions(opts.jira_project) ]
issues_db['versions'] = issues_db['milestones']
# bb_issue_map: bb issue # -> bitbucket issue
bb_issue_map = ImmutableDict( (i['id'],i) for i in issues_db['issues'] )
# jk_issue_map: jira key -> bitbucket issue
# contains only items migrated from JIRA (i.e., not preexisting issues with --existing)
jk_issue_map = ImmutableDict()
# issue_links is a dict of dicts of lists, using JIRA keys
# e.g., links['CORE-135']['depends on'] = ['CORE-137']
issue_links = collections.defaultdict(lambda: collections.defaultdict(lambda: []))
issues = fetch_issues(opts,jcl)
logger.info("fetch {n} issues from JIRA".format(n=len(issues)))
for ji in issues:
# Pfft. Need to fetch the issue again due to bug in JIRA.
# See https://bitbucket.org/bspeakmon/jira-python/issue/47/, comment on 2013-10-01 by ssonic
ji = jcl.issue(ji.key,expand="attachments,comments")
# create the issue
bbi = jira_issue_to_bb_issue(opts,jcl,ji)
issues_db['issues'] += [bbi]
bb_issue_map[bbi['id']] = bbi
jk_issue_map[ji.key] = bbi
issue_links[ji.key]['imported from'] = [jira_link_text(ji.key)]
# add comments
for jc in ji.fields.comment.comments:
bbc = jira_comment_to_bb_comment(opts,jcl,jc)
bbc['issue'] = bbi['id']
issues_db['comments'] += [bbc]
# add attachments
for ja in ji.fields.attachment:
att_rel_path = os.path.join(opts.att_rel_dir,ja.id)
att_abs_path = os.path.join(opts.att_abs_dir,ja.id)
if not os.path.exists(att_abs_path):
open(att_abs_path,'w').write(ja.get())
logger.info("Wrote {att_abs_path}".format(att_abs_path=att_abs_path))
bba = {
"path": att_rel_path,
"issue": bbi['id'],
"user": person_map[ja.author.name],
"filename": ja.filename,
}
issues_db['attachments'] += [bba]
# parent-child is task-subtask
if hasattr(ji.fields,'parent'):
issue_links[ji.fields.parent.key]['subtasks'].append(jira_key_to_bb_issue_tag(ji.key))
issue_links[ji.key]['parent task'].append(jira_key_to_bb_issue_tag(ji.fields.parent.key))
# add links
for il in ji.fields.issuelinks:
if hasattr(il,'outwardIssue'):
issue_links[ji.key][il.type.outward].append(jira_key_to_bb_issue_tag(il.outwardIssue.key))
elif hasattr(il,'inwardIssue'):
issue_links[ji.key][il.type.inward].append(jira_key_to_bb_issue_tag(il.inwardIssue.key))
logger.info("migrated issue {ji.key}: {ji.fields.summary} ({components})".format(
ji=ji,components=','.join(c.name for c in ji.fields.components)))
# append links section to content
# this section shows both task-subtask and "issue link" relationships
for src,dstlinks in issue_links.iteritems():
if src not in jk_issue_map:
logger.warn("issue {src}, with issue_links, not in jk_issue_map; skipping".format(src=src))
continue
links_block = "Links\n=====\n"
for desc,dsts in sorted(dstlinks.iteritems()):
links_block += "* **{desc}**: {links} \n".format(desc=desc,links=", ".join(dsts))
if jk_issue_map[src]['content']:
jk_issue_map[src]['content'] += "\n\n" + links_block
else:
jk_issue_map[src]['content'] = links_block
id_counts = collections.Counter(i['id'] for i in issues_db['issues'])
dupes = [ k for k,cnt in id_counts.iteritems() if cnt>1 ]
if dupes:
raise RuntimeError("{n} issue ids appear more than once from existing {opts.json_fn}".format(
n=len(dupes),opts=opts))
json.dump(issues_db,open(opts.json_fn,'w'))
logger.info("wrote {n} issues to {opts.json_fn}".format(n=len(id_counts),opts=opts))
# write zipfile
os.chdir(opts.dir)
with zipfile.ZipFile(opts.dir + '.zip','w') as zf:
for fn in ['db-1.0.json']+glob.glob('attachments/*'):
zf.write(fn)
logger.info("added {fn} to archive".format(fn=fn))
NOTE: I'm writing a new answer because writing this in a comment would be horrible, but most of the credit goes to #Turch's answer.
My steps (in OSX and Debian machines, both worked fine):
apt-get install python-pip (Debian) or sudo easy_install pip (OSX)
pip install jira
pip install configparser
easy_install -U setuptools (not sure if really needed)
Download or clone the source code from https://bitbucket.org/reece/rcore/ in your home folder, for example. Note: don't download using pip, it will get the 0.0.2 version and you need the 0.0.3.
Download the Python script created by Reece, mentioned by #Turch, and place it inside of the rcore folder.
Follow the instructions by #Turch: I also had to replace all instances of iteritems with items in the script and in rcore/types/immutabledict.py to make it work with Python 3. You will also need to fill in the dictionaries (priority_map, person_map, etc) with the values your project uses. Finally, you need a config file to exist with the connection info (see comments at the top of the script). Note: I used hostname like jira.domain.com (no http or https).
(This change did the trick for me) I had to change part of the line 250 from 'https://{opts.jira_hostname}/' to 'http://{opts.jira_hostname}/'
To finish, run the script like #Turch mentioned: The basic command line usage is export.py --jira-project <project>
The file was placed in /tmp/.zip for me.
The file was perfectly accepted in the BitBucket importer today.
Hooray for Reece and Turch! Thanks guys!