VSCode problem retrieving Bazel build targets - visual-studio-code

I'm having trouble getting VSCode to load build targets for a Java project. The error message that I get is the following:
Command failed: bazel --output_base=/var/folders/cj/1fv063dx1772zbpnfpzvj_c00000gn/T/5b71fd39758e7fd8e710c281fbbdda92 query ...:* --output=package Loading: 0 packages loaded ERROR: error loading package 'bazel-test/external/rules_jvm_external': cannot load '//:private/versions.bzl': no such file Loading: 12 packages loaded currently loading: bazel-test/external/local_config_cc ... (6 packages) Loading: 12 packages loaded currently loading: bazel-test/external/local_config_cc ... (6 packages)
My WORKSPACE file is as follows:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "io_bazel_rules_docker",
sha256 = "4521794f0fba2e20f3bf15846ab5e01d5332e587e9ce81629c7f96c793bb7036",
strip_prefix = "rules_docker-0.14.4",
urls = ["https://github.com/bazelbuild/rules_docker/releases/download/v0.14.4/rules_docker-v0.14.4.tar.gz"],
)
load(
"#io_bazel_rules_docker//repositories:repositories.bzl",
container_repositories = "repositories",
)
container_repositories()
load("#io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")
container_deps()
load("#io_bazel_rules_docker//repositories:pip_repositories.bzl", "pip_deps")
pip_deps()
load(
"#io_bazel_rules_docker//java:image.bzl",
_java_image_repos = "repositories",
)
_java_image_repos()
load(
"#io_bazel_rules_docker//container:container.bzl",
"container_pull",
)
container_pull(
name = "openjdk11_slim",
registry = "index.docker.io",
repository = "library/openjdk",
tag = "11-slim"
)
RULES_JVM_EXTERNAL_TAG = "3.3"
RULES_JVM_EXTERNAL_SHA = "d85951a92c0908c80bd8551002d66cb23c3434409c814179c0ff026b53544dab"
http_archive(
name = "rules_jvm_external",
strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG,
sha256 = RULES_JVM_EXTERNAL_SHA,
url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG,
)
load("#rules_jvm_external//:defs.bzl", "maven_install")
maven_install(
artifacts = [
# Accessed as dependency: #maven://io_vertx_vertx_core
"io.vertx:vertx-core:3.9.2",
# Accessed as dependency: #maven://io_vertx_vertx_web
"io.vertx:vertx-web:3.9.2",
# Accessed as dependency: #maven://io_vertx_vertx_rx_java2
"io.vertx:vertx-rx-java2:3.9.2",
],
repositories = [
"https://repo1.maven.org/maven2",
]
)
I don't see anything obviously wrong with my WORKSPACE file, so I'm tempted to assume the problem is with VSCode or in the BUILD files in one of my external dependencies. Other than that the project contains a single Hello.java and its accompanying java_binary rule. If I run myself the same query for all packages I get the following:
% bazel query ...:\* --output=package
src/main/java
Loading: 0 packages loaded
In other words, there's no error. Any ideas how I could try to debug or fix this? I just installed everything on the laptop, so the software versions are:
Bazel version: 3.5.0
VSCode versio: 1.48.2

Following the comment ignoring the convenience symlink solved it for me:
echo bazel-`basename ${PWD}` >> .bazelignore

I got around this issue by changing the Query Expression in Bazel extension Settings from
...:*
to
//< directory name >/...:*

Related

How to remove toaster from yocto Project

I am using yocto to build an image for the Up-Board following this guide: https://github.com/AaeonCM/meta-up-board
I tried to use Toaster for this project but later found out that the Up-Board isn't supported by toaster.
After I ran source toaster start I was no longer able to build form the command line with bitbake.
I get an error when I try to add an existing project by build directory in toaster, I think this is because the board is not supported.
The error I get when I try to use the 'Import command line project' feature in Toaster web browser is:
Environment:
Request Method: POST
Request URL: http://127.0.0.1:8000/toastergui/newproject/
Django Version: 2.2.27
Python Version: 3.6.9
Installed Applications:
('django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.messages',
'django.contrib.sessions',
'django.contrib.admin',
'django.contrib.staticfiles',
'django.contrib.humanize',
'bldcollector',
'toastermain',
'bldcontrol',
'toastergui',
'orm')
Installed Middleware:
['django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware']
Traceback:
File "/home/dave/.local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
34. response = get_response(request)
File "/home/dave/.local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
115. response = self.process_exception_by_middleware(e, request)
File "/home/dave/.local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
113. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/media/drive2/yocto2/poky/bitbake/lib/toaster/toastergui/views.py" in newproject
1407. management.call_command('buildimport', '--command=import', '--name=%s' % request.POST['projectname'], '--path=%s' % request.POST['importdir'], interactive=False)
File "/home/dave/.local/lib/python3.6/site-packages/django/core/management/__init__.py" in call_command
140. ', '.join(sorted(valid_options)),
Exception Type: TypeError at /toastergui/newproject/
Exception Value: Unknown option(s) for buildimport command: interactive. Valid options are: callback, command, delete_project, force_color, help, name, no_color, path, pythonpath, release, settings, skip_checks, stderr, stdout, traceback, verbosity, version.
The error I get when I try to build my project again from the command line is:
/poky/build$ MACHINE=up-board bitbake upboard-image-sato
Loading cache: 100% |#####################################################################################################################################################################| Time: 0:00:00
Loaded 3393 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.46.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "universal"
TARGET_SYS = "x86_64-poky-linux"
MACHINE = "up-board"
DISTRO = "poky"
DISTRO_VERSION = "3.1.14"
TUNE_FEATURES = "m64 corei7"
TARGET_FPU = ""
meta
meta-poky
meta-yocto-bsp = "dunfell:c8987e7bca6ab22a166ca13c5d2fe8e02fbb6e23"
meta-intel = "dunfell:8eec569734cb1ff9c0905f4a88f9b4bfc89ed9fc"
meta-up-board = "dunfell:23b0e460c068db2c69d844681a23b8febe9b95a0"
meta-oe
meta-python
meta-networking
meta-filesystems = "dunfell:ec978232732edbdd875ac367b5a9c04b881f2e19"
meta-virtualization = "dunfell:c5f61e547b90aa8058cf816f00902afed9c96f72"
Initialising tasks: 100% |################################################################################################################################################################| Time: 0:00:04
Sstate summary: Wanted 2 Found 0 Missed 2 Current 3705 (0% match, 99% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 9002 tasks of which 9002 didn't need to be rerun and all succeeded.
ERROR: Execution of event handler 'toaster_buildhistory_dump' failed
Traceback (most recent call last):
File "/media/drive2/yocto2/poky/meta/classes/toaster.bbclass", line 273, in toaster_buildhistory_dump(e=<bb.event.BuildCompleted object at 0x7fe8e830f6d8>):
files[target]['files'] = []
> with open("%s/installed-package-sizes.txt" % installed_img_path, "r") as fin:
for line in fin:
FileNotFoundError: [Errno 2] No such file or directory: '/media/drive2/yocto2/poky/build/buildhistory/images/up_board/glibc/upboard-image-sato/installed-package-sizes.txt'
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Is there any way to un-do whatever changes toaster did to the project? Uninstall toaster? Or should I start the project over again?

Bazel error: type 'java_common' has no method create_provider()

I've tried to create a hello world for scala project on bazel.
But got an error:
ERROR: /private/var/tmp/_bazel_user/4c70a1cb51cd18a4cc09dc9aa5e6a71b/external/scalac_rules_commons_io/BUILD:9:1: in scala_import rule #scalac_rules_commons_io//:scalac_rules_commons_io:
Traceback (most recent call last):
File "/private/var/tmp/_bazel_user/4c70a1cb51cd18a4cc09dc9aa5e6a71b/external/scalac_rules_commons_io/BUILD", line 9
scala_import(name = 'scalac_rules_commons_io')
File "/private/var/tmp/_bazel_user/4c70a1cb51cd18a4cc09dc9aa5e6a71b/external/io_bazel_rules_scala/scala/scala_import.bzl", line 27, in _scala_import_impl
struct(<2 more arguments>)
File "/private/var/tmp/_bazel_user/4c70a1cb51cd18a4cc09dc9aa5e6a71b/external/io_bazel_rules_scala/scala/scala_import.bzl", line 32, in struct
_create_provider(current_jars, <6 more arguments>)
File "/private/var/tmp/_bazel_user/4c70a1cb51cd18a4cc09dc9aa5e6a71b/external/io_bazel_rules_scala/scala/scala_import.bzl", line 73, in _create_provider
java_common.create_provider(<5 more arguments>)
type 'java_common' has no method create_provider()
Project structure:
WORKSPACE file was copied from here
rules_scala_version="69d3c5b5d9b51537231746e93b4383384c9ebcf4" # update this as needed
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "io_bazel_rules_scala",
strip_prefix = "rules_scala-%s" % rules_scala_version,
type = "zip",
url = "https://github.com/bazelbuild/rules_scala/archive/%s.zip" % rules_scala_version,
)
load("#io_bazel_rules_scala//scala:toolchains.bzl", "scala_register_toolchains")
scala_register_toolchains()
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()
protobuf_version="09745575a923640154bcf307fba8aedff47f240a"
protobuf_version_sha256="416212e14481cff8fd4849b1c1c1200a7f34808a54377e22d7447efdf54ad758"
http_archive(
name = "com_google_protobuf",
url = "https://github.com/protocolbuffers/protobuf/archive/%s.tar.gz" % protobuf_version,
strip_prefix = "protobuf-%s" % protobuf_version,
sha256 = protobuf_version_sha256,
)
# bazel-skylib 0.8.0 released 2019.03.20 (https://github.com/bazelbuild/bazel-skylib/releases/tag/0.8.0)
skylib_version = "0.8.0"
http_archive(
name = "bazel_skylib",
type = "tar.gz",
url = "https://github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib.{}.tar.gz".format (skylib_version, skylib_version),
sha256 = "2ef429f5d7ce7111263289644d233707dba35e39696377ebab8b0bc701f7818e",
)
BUILD.bazel file:
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_binary", "scala_test")
scala_library(
name = "pkg1",
srcs = [
"Main.scala"
]
)
bazel version:
$ bazel version
Bazelisk version: v1.0
Build label: 1.1.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Oct 21 08:47:13 2019 (1571647633)
Build timestamp: 1571647633
Build timestamp as int: 1571647633
That version of rules_scala appears appears to be designed for the prior bazel version. rules_scala#876 is about this specific issue, and mentions another commit to try: 3404be0e3b3afda99a12724171d80283eaf2b1e7. You might also try 26cf9b74fc46f1e9a970c97837447549ed7257b6, which is master when I looked just now. You would replace the rules_scala_version value in the first line of the WORKSPACE with one of those commit SHA1 values.
For reference, the java_common v0.29.1 docs mention create_provider, but the java_common v1.0.0 docs don't.

Depending on apache_beam from a bazel project

I am creating a bazel python project which depends on apache_beam sdk. In the WORKSPACE file of my project, I have downloaded apache_beam zip file via http_archive.
Whenever I am trying to access apache_beam.testing sub-package in my project, I am getting an error
ModuleNotFoundError: No module named 'apache_beam.testing'
import apache_beam is working fine though.
WORKSPACE:
http_archive(
name = "apache_beam",
urls = ["https://github.com/apache/beam/archive/master.zip"],
strip_prefix = 'beam-master/sdks/python',
build_file = str(Label("//third_party:apache_beam.BUILD")),
)
apache_beam.BUILD:
py_library(
name = "apache_beam",
srcs = glob(["apache_beam/**/*.py"]),
visibility = ["//visibility:public"],
)
project's BUILD file:
py_test(
name = "project",
srcs = ["project.py"],
deps = [
"#apache_beam//:apache_beam",
],
project.py file:
from apache_beam.testing import util
In apache_beam.BUILD, you need to set the imports attribute:
py_library(
name = "apache_beam",
srcs = glob(["apache_beam/**/*.py"]),
visibility = ["//visibility:public"],
imports = "apache_beam",
)

compiles .py to .exe inluding it Classes

I have 4 .py files. Below is a list of files what is required to run the programme. Any of them missing will fail the programme to run.
How my code works:
) GUIss.py imports demonstrator.py
) Demonstrator.py imports filereader.py and process.py
) To run the programm I just need to click GUIss.py.
My cx-freeze code below:
from cx_Freeze import setup,Executable
import os
includefiles = ['filereader.py','demonstrator.py','logo.gif','thebrighterchoice.gif']
#bin_includes= ['process.py','demonstrator.py','filereader.py'] ..... 'bin_includes':bin_includesincludes = ['process']
includes = ['process','tkinter']
excludes = ['tkinter']
packages = ['os','xlrd']
setup(
name = "Process",
version = "0.1",
description = "description",
author = "Raitis Kupce",
options = {'build_exe' :{'excludes': excludes,'includes':includes,'packages':packages,'include_files':includefiles}},
executables = [Executable("GUIss.py")]
)
When I run compiled file I get an error message:
I then tried to write in setup.py (cx-freeze file)
excludes = ['tkinter']
Then includes = ['tkinter']
Afterwards packages = ['tkinter']
Despite numerous attempt, no luck, same message all the time.
P.S
My python source code can be downloaded from https://github.com/Dragnets/Administration
I did studied hours from here and here and modifying endless times but no luck.

How can I export Jira issues to BitBucket

Ive just moved my projects code from java.net to BitBucket. But my jira issue tracking is still hosted on java.net, although BitBucket does have some options for linking to an external issue tracker I don't think I can use it for java.net, not least because I do not have the admin priviledges need to install the DVCS connector.
So I thought an alternative option would be to export and then import the issues into BitBucket issue tracker, is that possible ?
Progress so far
So I tried following the steps in both informative answers using OSX below but I hit a problem - I'm rather confused about what the script would actually be called because in the answers it talks about export.py but no such script exists with that name so I renamed the one I downloaded.
sudo easy_install pip (OSX)
pip install jira
pip install configparser
easy_install -U setuptools
Go to https://bitbucket.org/reece/rcore, select downloads tab, download zip and unzip, and rename to reece ( for some reason git clone https://bitbucket.org/reece/rcore fails with error)
cd reece/rcore
Save script as export.py in rcore subfolder
Replace iteritems with items in import.py
Replace iteritems with types/immutabledict.py
Create .config in rcore folder
Create .config/jira-issues-move-to-bitbucket.conf containing
jira-username=paultaylor
jira-hostname=https://java.net/jira/browse/JAUDIOTAGGER
jira-password=password
Run python export.py --jira-project jaudiotagger
gives
macbook:rcore paul$ python export.py --jira-project jaudiotagger
Traceback (most recent call last):
File "export.py", line 24, in <module>
import configparser
ImportError: No module named configparser
- Run python export.py --jira-project jaudiotagger
I need to run pip insdtall as root so did
sudo pip install configparser
and that worked
but now
python export.py --jira.project jaudiotagger
gives
File "export.py" line 35, in <module?
from jira.client import JIRA
ImportError: No module named jira.client
You can import issues into BitBucket, they just need to be in the appropriate format. Fortunately, Reece Hart has already written a Python script to connect to a Jira instance and export the issues.
To get the script to run I had to install the Jira Python package as well as the latest version of rcore (if you use pip you get an incompatible previous version, so you have to get the source). I also had to replace all instances of iteritems with items in the script and in rcore/types/immutabledict.py to make it work with Python 3. You will also need to fill in the dictionaries (priority_map, person_map, etc) with the values your project uses. Finally, you need a config file to exist with the connection info (see comments at the top of the script).
The basic command line usage is export.py --jira-project <project>
Once you've got the data exported, see the instructions for importing issues to BitBucket
#!/usr/bin/env python
"""extract issues from JIRA and export to a bitbucket archive
See:
https://confluence.atlassian.com/pages/viewpage.action?pageId=330796872
https://confluence.atlassian.com/display/BITBUCKET/Mark+up+comments
https://bitbucket.org/tutorials/markdowndemo/overview
2014-04-12 08:26 Reece Hart <reecehart#gmail.com>
Requires a file ~/.config/jira-issues-move-to-bitbucket.conf
with content like
[default]
jira-username=some.user
jira-hostname=somewhere.jira.com
jira-password=ur$pass
"""
import argparse
import collections
import configparser
import glob
import itertools
import json
import logging
import os
import pprint
import re
import sys
import zipfile
from jira.client import JIRA
from rcore.types.immutabledict import ImmutableDict
priority_map = {
'Critical (P1)': 'critical',
'Major (P2)': 'major',
'Minor (P3)': 'minor',
'Nice (P4)': 'trivial',
}
person_map = {
'reece.hart': 'reece',
# etc
}
issuetype_map = {
'Improvement': 'enhancement',
'New Feature': 'enhancement',
'Bug': 'bug',
'Technical task': 'task',
'Task': 'task',
}
status_map = {
'Closed': 'resolved',
'Duplicate': 'duplicate',
'In Progress': 'open',
'Open': 'new',
'Reopened': 'open',
'Resolved': 'resolved',
}
def parse_args(argv):
def sep_and_flatten(l):
# split comma-sep elements and flatten list
# e.g., ['a','b','c,d'] -> set('a','b','c','d')
return list( itertools.chain.from_iterable(e.split(',') for e in l) )
cf = configparser.ConfigParser()
cf.readfp(open(os.path.expanduser('~/.config/jira-issues-move-to-bitbucket.conf'),'r'))
ap = argparse.ArgumentParser(
description = __doc__
)
ap.add_argument(
'--jira-hostname', '-H',
default = cf.get('default','jira-hostname',fallback=None),
help = 'host name of Jira instances (used for url like https://hostname/, e.g., "instancename.jira.com")',
)
ap.add_argument(
'--jira-username', '-u',
default = cf.get('default','jira-username',fallback=None),
)
ap.add_argument(
'--jira-password', '-p',
default = cf.get('default','jira-password',fallback=None),
)
ap.add_argument(
'--jira-project', '-j',
required = True,
help = 'project key (e.g., JRA)',
)
ap.add_argument(
'--jira-issues', '-i',
action = 'append',
default = [],
help = 'issue id (e.g., JRA-9); multiple and comma-separated okay; default = all in project',
)
ap.add_argument(
'--jira-issues-file', '-I',
help = 'file containing issue ids (e.g., JRA-9)'
)
ap.add_argument(
'--jira-components', '-c',
action = 'append',
default = [],
help = 'components criterion; multiple and comma-separated okay; default = all in project',
)
ap.add_argument(
'--existing', '-e',
action = 'store_true',
default = False,
help = 'read existing archive (from export) and merge new issues'
)
opts = ap.parse_args(argv)
opts.jira_components = sep_and_flatten(opts.jira_components)
opts.jira_issues = sep_and_flatten(opts.jira_issues)
return opts
def link(url,text=None):
return "[{text}]({url})".format(url=url,text=url if text is None else text)
def reformat_to_markdown(desc):
def _indent4(mo):
i = " "
return i + mo.group(1).replace("\n",i)
def _repl_mention(mo):
return "#" + person_map[mo.group(1)]
#desc = desc.replace("\r","")
desc = re.sub("{noformat}(.+?){noformat}",_indent4,desc,flags=re.DOTALL+re.MULTILINE)
desc = re.sub(opts.jira_project+r"-(\d+)",r"issue #\1",desc)
desc = re.sub(r"\[~([^]]+)\]",_repl_mention,desc)
return desc
def fetch_issues(opts,jcl):
jql = [ 'project = ' + opts.jira_project ]
if opts.jira_components:
jql += [ ' OR '.join([ 'component = '+c for c in opts.jira_components ]) ]
if opts.jira_issues:
jql += [ ' OR '.join([ 'issue = '+i for i in opts.jira_issues ]) ]
jql_str = ' AND '.join(["("+q+")" for q in jql])
logging.info('executing query ' + jql_str)
return jcl.search_issues(jql_str,maxResults=500)
def jira_issue_to_bb_issue(opts,jcl,ji):
"""convert a jira issue to a dictionary with values appropriate for
POSTing as a bitbucket issue"""
logger = logging.getLogger(__name__)
content = reformat_to_markdown(ji.fields.description) if ji.fields.description else ''
if ji.fields.assignee is None:
resp = None
else:
resp = person_map[ji.fields.assignee.name]
reporter = person_map[ji.fields.reporter.name]
jiw = jcl.watchers(ji.key)
watchers = [ person_map[u.name] for u in jiw.watchers ] if jiw else []
milestone = None
if ji.fields.fixVersions:
vnames = [ v.name for v in ji.fields.fixVersions ]
milestone = vnames[0]
if len(vnames) > 1:
logger.warn("{ji.key}: bitbucket issues may have only 1 milestone (JIRA fixVersion); using only first ({f}) and ignoring rest ({r})".format(
ji=ji, f=milestone, r=",".join(vnames[1:])))
issue_id = extract_issue_number(ji.key)
bbi = {
'status': status_map[ji.fields.status.name],
'priority': priority_map[ji.fields.priority.name],
'kind': issuetype_map[ji.fields.issuetype.name],
'content_updated_on': ji.fields.created,
'voters': [],
'title': ji.fields.summary,
'reporter': reporter,
'component': None,
'watchers': watchers,
'content': content,
'assignee': resp,
'created_on': ji.fields.created,
'version': None, # ?
'edited_on': None,
'milestone': milestone,
'updated_on': ji.fields.updated,
'id': issue_id,
}
return bbi
def jira_comment_to_bb_comment(opts,jcl,jc):
bbc = {
'content': reformat_to_markdown(jc.body),
'created_on': jc.created,
'id': int(jc.id),
'updated_on': jc.updated,
'user': person_map[jc.author.name],
}
return bbc
def extract_issue_number(jira_issue_key):
return int(jira_issue_key.split('-')[-1])
def jira_key_to_bb_issue_tag(jira_issue_key):
return 'issue #' + str(extract_issue_number(jira_issue_key))
def jira_link_text(jk):
return link("https://invitae.jira.com/browse/"+jk,jk) + " (Invitae access required)"
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
opts = parse_args(sys.argv[1:])
dir_name = opts.jira_project
if opts.jira_components:
dir_name += '-' + ','.join(opts.jira_components)
if opts.jira_issues_file:
issues = [i.strip() for i in open(opts.jira_issues_file,'r')]
logger.info("added {n} issues from {opts.jira_issues_file} to issues list".format(n=len(issues),opts=opts))
opts.jira_issues += issues
opts.dir = os.path.join('/','tmp',dir_name)
opts.att_rel_dir = 'attachments'
opts.att_abs_dir = os.path.join(opts.dir,opts.att_rel_dir)
opts.json_fn = os.path.join(opts.dir,'db-1.0.json')
if not os.path.isdir(opts.att_abs_dir):
os.makedirs(opts.att_abs_dir)
opts.jira_issues = list(set(opts.jira_issues)) # distinctify
jcl = JIRA({'server': 'https://{opts.jira_hostname}/'.format(opts=opts)},
basic_auth=(opts.jira_username,opts.jira_password))
if opts.existing:
issues_db = json.load(open(opts.json_fn,'r'))
existing_ids = [ i['id'] for i in issues_db['issues'] ]
logger.info("read {n} issues from {fn}".format(n=len(existing_ids),fn=opts.json_fn))
else:
issues_db = dict()
issues_db['meta'] = {
'default_milestone': None,
'default_assignee': None,
'default_kind': "bug",
'default_component': None,
'default_version': None,
}
issues_db['attachments'] = []
issues_db['comments'] = []
issues_db['issues'] = []
issues_db['logs'] = []
issues_db['components'] = [ {'name':v.name} for v in jcl.project_components(opts.jira_project) ]
issues_db['milestones'] = [ {'name':v.name} for v in jcl.project_versions(opts.jira_project) ]
issues_db['versions'] = issues_db['milestones']
# bb_issue_map: bb issue # -> bitbucket issue
bb_issue_map = ImmutableDict( (i['id'],i) for i in issues_db['issues'] )
# jk_issue_map: jira key -> bitbucket issue
# contains only items migrated from JIRA (i.e., not preexisting issues with --existing)
jk_issue_map = ImmutableDict()
# issue_links is a dict of dicts of lists, using JIRA keys
# e.g., links['CORE-135']['depends on'] = ['CORE-137']
issue_links = collections.defaultdict(lambda: collections.defaultdict(lambda: []))
issues = fetch_issues(opts,jcl)
logger.info("fetch {n} issues from JIRA".format(n=len(issues)))
for ji in issues:
# Pfft. Need to fetch the issue again due to bug in JIRA.
# See https://bitbucket.org/bspeakmon/jira-python/issue/47/, comment on 2013-10-01 by ssonic
ji = jcl.issue(ji.key,expand="attachments,comments")
# create the issue
bbi = jira_issue_to_bb_issue(opts,jcl,ji)
issues_db['issues'] += [bbi]
bb_issue_map[bbi['id']] = bbi
jk_issue_map[ji.key] = bbi
issue_links[ji.key]['imported from'] = [jira_link_text(ji.key)]
# add comments
for jc in ji.fields.comment.comments:
bbc = jira_comment_to_bb_comment(opts,jcl,jc)
bbc['issue'] = bbi['id']
issues_db['comments'] += [bbc]
# add attachments
for ja in ji.fields.attachment:
att_rel_path = os.path.join(opts.att_rel_dir,ja.id)
att_abs_path = os.path.join(opts.att_abs_dir,ja.id)
if not os.path.exists(att_abs_path):
open(att_abs_path,'w').write(ja.get())
logger.info("Wrote {att_abs_path}".format(att_abs_path=att_abs_path))
bba = {
"path": att_rel_path,
"issue": bbi['id'],
"user": person_map[ja.author.name],
"filename": ja.filename,
}
issues_db['attachments'] += [bba]
# parent-child is task-subtask
if hasattr(ji.fields,'parent'):
issue_links[ji.fields.parent.key]['subtasks'].append(jira_key_to_bb_issue_tag(ji.key))
issue_links[ji.key]['parent task'].append(jira_key_to_bb_issue_tag(ji.fields.parent.key))
# add links
for il in ji.fields.issuelinks:
if hasattr(il,'outwardIssue'):
issue_links[ji.key][il.type.outward].append(jira_key_to_bb_issue_tag(il.outwardIssue.key))
elif hasattr(il,'inwardIssue'):
issue_links[ji.key][il.type.inward].append(jira_key_to_bb_issue_tag(il.inwardIssue.key))
logger.info("migrated issue {ji.key}: {ji.fields.summary} ({components})".format(
ji=ji,components=','.join(c.name for c in ji.fields.components)))
# append links section to content
# this section shows both task-subtask and "issue link" relationships
for src,dstlinks in issue_links.iteritems():
if src not in jk_issue_map:
logger.warn("issue {src}, with issue_links, not in jk_issue_map; skipping".format(src=src))
continue
links_block = "Links\n=====\n"
for desc,dsts in sorted(dstlinks.iteritems()):
links_block += "* **{desc}**: {links} \n".format(desc=desc,links=", ".join(dsts))
if jk_issue_map[src]['content']:
jk_issue_map[src]['content'] += "\n\n" + links_block
else:
jk_issue_map[src]['content'] = links_block
id_counts = collections.Counter(i['id'] for i in issues_db['issues'])
dupes = [ k for k,cnt in id_counts.iteritems() if cnt>1 ]
if dupes:
raise RuntimeError("{n} issue ids appear more than once from existing {opts.json_fn}".format(
n=len(dupes),opts=opts))
json.dump(issues_db,open(opts.json_fn,'w'))
logger.info("wrote {n} issues to {opts.json_fn}".format(n=len(id_counts),opts=opts))
# write zipfile
os.chdir(opts.dir)
with zipfile.ZipFile(opts.dir + '.zip','w') as zf:
for fn in ['db-1.0.json']+glob.glob('attachments/*'):
zf.write(fn)
logger.info("added {fn} to archive".format(fn=fn))
NOTE: I'm writing a new answer because writing this in a comment would be horrible, but most of the credit goes to #Turch's answer.
My steps (in OSX and Debian machines, both worked fine):
apt-get install python-pip (Debian) or sudo easy_install pip (OSX)
pip install jira
pip install configparser
easy_install -U setuptools (not sure if really needed)
Download or clone the source code from https://bitbucket.org/reece/rcore/ in your home folder, for example. Note: don't download using pip, it will get the 0.0.2 version and you need the 0.0.3.
Download the Python script created by Reece, mentioned by #Turch, and place it inside of the rcore folder.
Follow the instructions by #Turch: I also had to replace all instances of iteritems with items in the script and in rcore/types/immutabledict.py to make it work with Python 3. You will also need to fill in the dictionaries (priority_map, person_map, etc) with the values your project uses. Finally, you need a config file to exist with the connection info (see comments at the top of the script). Note: I used hostname like jira.domain.com (no http or https).
(This change did the trick for me) I had to change part of the line 250 from 'https://{opts.jira_hostname}/' to 'http://{opts.jira_hostname}/'
To finish, run the script like #Turch mentioned: The basic command line usage is export.py --jira-project <project>
The file was placed in /tmp/.zip for me.
The file was perfectly accepted in the BitBucket importer today.
Hooray for Reece and Turch! Thanks guys!