I'm trying to install Spark on my local machine. I have been following this guide. I have installed JDK-7 (also have JDK-8) and Scala 2.11.7. A problem occurs when I try to use sbt to build Spark 1.4.1. I get the following exception.
NOTE: The sbt/sbt script has been relocated to build/sbt.
Please update references to point to the new location.
Invoking 'build/sbt assembly' now ...
Attempting to fetch sbt
Launching sbt from build/sbt-launch-0.13.7.jar
Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar
I have searched for a solution to this problem. I have found a nice guide https://stackoverflow.com/a/31597283/2771315 which uses a pre-built version. Other than using the pre-built version, is there a way to install Spark using sbt? Further, is there a reason as to why the Invalid or corrupt jarfile error occurs?
I met the same problem. I have fixed it now.
This probably because sbt-launch-0.13.7.jar has a unsuccessful download, although you can see the file is exist, but it's not correct file. The file is about 1.2MB in size. If less than that, you can get into the build/ , use "vim sbt-launch-0.13.7.jar" or other tools to open sbt-launch-0.13.7.jar file.
If the file have the content like this:
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
It implys that sbt-launch-0.13.7.jar is not downloaded.
Then open sbt-launch-lib.bash in the same directory,check the line 41 and 42, it gives two urls. Open it to check if they work well.
If url1 doesn't work，download the sbt-launch.jar manually(you can use url2, it may works,or you can download from sbt official website), put it in the same directory, rename it to sbt-launch-0.13.7.jar, then you shoud comment lines in relation to the downloading(may be between line 47 and 68)， avoid the script download it again. Like this:
acquire_sbt_jar () {
SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}' ./project/build.properties`
URL1=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
URL2=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
JAR=build/sbt-launch-${SBT_VERSION}.jar
sbt_jar=$JAR
# if [[ ! -f "$sbt_jar" ]]; then
# # Download sbt launch jar if it hasn't been downloaded yet
# if [ ! -f "${JAR}" ]; then
# # Download
# printf "Attempting to fetch sbt\n"
# JAR_DL="${JAR}.part"
# if [ $(command -v curl) ]; then
# (curl --silent ${URL1} > "${JAR_DL}" || curl --silent ${URL2} > "${JAR_DL}") && mv "${JAR_DL}" "${JAR}"
# elif [ $(command -v wget) ]; then
# (wget --quiet ${URL1} -O "${JAR_DL}" || wget --quiet ${URL2} -O "${JAR_DL}") && mv "${JAR_DL}" "${JAR}"
# else
# printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
# exit -1
# fi
# fi
# if [ ! -f "${JAR}" ]; then
# # We failed to download
# printf "Our attempt to download sbt locally to ${JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
# exit -1
# fi
# printf "Launching sbt from ${JAR}\n"
# fi
}
Then use "build/sbt assembly" to build the spark again.
Hope you will succeed.
If I didn't express clearly, the following links may be helpful.
https://www.mail-archive.com/user#spark.apache.org/msg34358.html
Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.5.jar the answer by prabeesh
https://groups.google.com/forum/#!topic/predictionio-user/fllCh8n-0d4
Download the sbt-launch.jar file manually (you can use url2, it may work, or you can download from the sbt official website), put it in the same directory, rename it to sbt-launch-0.13.7.jar, then run the sbt/sbt assembly command.
Related
I tried to build the Android SDK Addon System Image of halogenOS 13 (AOSP 13, nothing was changed in the AOSP source code yet).
The steps to build are as usual:
source build/envsetup.sh
lunch aosp_sdk_phone_x86_64-eng
m sdk_addon
At 100%, the build fails with following error:
[100% 12483/12483] Packaging SDK Addon System-Image: out/host/linux-x86/sdk_addon/custom-eng.simao--
FAILED: out/host/linux-x86/sdk_addon/custom-eng.simao--img.zip
/bin/bash -c "(cp -R out/target/product/emulator_x86_64/data out/host/linux-x86/obj/SDK_ADDON/custom_intermediates/custom-eng.simao--img/images/x86_64/data ) && (out/host/linux-x86/bin/soong_zip -o out/host/linux-x86/sdk_addon/custom-eng.simao--img.zip -C out/host/linux-x86/obj/SDK_ADDON/custom_intermediates/custom-eng.simao--img/images/ -D out/host/linux-x86/obj/SDK_ADDON/custom_intermediates/custom-eng.simao--img/images/x86_64 )"
cp: bad 'out/target/product/emulator_x86_64/data': No such file or directory
13:17:13 ninja failed with: exit status 1
If I just mkdir out/target/product/emulator_x86_64/data, of course, that just solves the build error but the SDK addon does not actually boot in the emulator due to encryption issues with the userdata partition so I think this is related. This makes me guess that in the data directory there should be some files in there but for some reason the are not created.
EDIT:
What's really odd here is that the file device/generic/goldfish/vendor.mk explicitly adds some data files to PRODUCT_COPY_FILES, notably:
PRODUCT_COPY_FILES += \
device/generic/goldfish/data/etc/dtb.img:dtb.img \
device/generic/goldfish/emulator-info.txt:data/misc/emulator/version.txt \
device/generic/goldfish/data/etc/apns-conf.xml:data/misc/apns/apns-conf.xml \
device/generic/goldfish/radio/RadioConfig/radioconfig.xml:data/misc/emulator/config/radioconfig.xml \
device/generic/goldfish/data/etc/iccprofile_for_sim0.xml:data/misc/modem_simulator/iccprofile_for_sim0.xml \
If I manually build them using, for example, m out/target/product/emulator_x86_64/data/misc/emulator/version.txt, the file is created at the correct location, as expected. Which leads me to wonder when entries PRODUCT_COPY_FILES are considered targets to be built and when they aren't.
EDIT2:
I got the emulator to boot but the data directory is still not being created. (Creating manually or building one target in the data dir is a workaorund).
I am trying to create a TPM-based unlock script using tpm2-tools with instructions from Tevora Secure boot tpm2. I have set up the key, loaded it with cryptsetup luksAddKey secret.bin, then tested it using tpm2_unlock -c 0x81000000 --auth pci:sha1:0,2,3,7 and returns the value of secret.bin. For extra measures, to make sure it works, I loaded secret.bin into "/etc/crypttab", ran # update-initramfs -u -k all, and rebooted. Upon reboot, the system unlocked.
I copied over the following code into "/etc/initramfs-tools/hooks/tpm2"
#!/bin/sh -e
if [ "$1" = "prereqs" ]; then exit 0; fi
. /usr/share/initramfs-tools/hook-functions
copy_exec /usr/local/bin/tpm2_unseal
copy_exec /usr/local/lib/libtss2-tcti-device.so
I appended my etc/crypttab from cryptname UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks to cryptname UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks,keyscript=/usr/local/bin/passphrase-from-tpm
I rewrote the following script because the tpm2-tools command was outdated, edited in the new command, and stored it in /usr/local/bin/passphrase-from-tpm:
#!/bin/sh
set -e
echo "Unlocking via TPM" >&2
export TPM2TOOLS_TCTI="device:/dev/tpm0"
/usr/local/bin/tpm2_unseal -c 0x81000000 --auth pcr:sha1:0,2,3,7
if [ $? -eq 0 ]; then
exit
fi
/lib/cryptsetup/askpass "Unlocking the disk fallback $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)\nEnter passphrase: "
I ran # update-initramfs -u -k all then rebooted. In reboot, I get the following error: /lib/cryptsetup/scripts/passphrase-from-tpm: line 5: /usr/local/bin/tpm2_unseal: not found
I have tried many times to edit passphrase-from-tpm unsuccessfully, including:
Moving both passphrase-from-tpm into "/boot/efi/EFI/BOOT/" and referencing crypttab to that file
Modifying passphrase-from-tpm to use a relative file path to tpm_unseal
Before I figured out how to create a backup linux boot using:
objcopy \
--add-section .osrel=/etc/os-release --change-section-vma .osrel=0x20000 \
--add-section .cmdline=cmdline.txt --change-section-vma .cmdline=0x30000 \
--add-section .linux="/boot/vmlinuz" --change-section-vma .linux=0x40000 \
--add-section .initrd="/boot/initrd.img" --change-section-vma .initrd=0x3000000 \
/usr/lib/systemd/boot/efi/linuxx64.efi.stub /boot/EFI/BOOT/BOOT_RECX64.EFI
I would be locked out of the system completely because of the error and had to reinstall Ubuntu about 40 times. I have suffered a lot and want to quit but I am too stubborn to throw in the flag.
just copy tpm2_unseal to /usr/local/bin/
I'm trying to make a working setup following basically those instructions, plus a few others I have found. While not working 100% yet, check that both /etc/initramfs-tools/hooks/tpm2 and /usr/local/bin/passphrase-from-tpm are marked executable (sudo chmod ug+x $filename).
After making the initramfs, you can run the following to ensure that the TPM related files are actually in the image. Replace the path in the filename by whatever update-initramfs said it was generating:
$ lsinitramfs /boot/initrd.img-5.0.0-37-generic | egrep "(tpm|libtss)"
lib/cryptsetup/scripts/passphrase-from-tpm
lib/modules/5.0.0-37-generic/kernel/crypto/asymmetric_keys/tpm_key_parser.ko
lib/modules/5.0.0-37-generic/kernel/crypto/asymmetric_keys/asym_tpm.ko
lib/udev/rules.d/tpm-udev.rules
usr/local/lib/libtss2-sys.so.0
usr/local/lib/libtss2-mu.so.0
usr/local/lib/libtss2-sys.so.0.0.0
usr/local/lib/libtss2-tcti-device.so
usr/local/lib/libtss2-tcti-device.so.0
usr/local/lib/libtss2-tcti-device.so.0.0.0
usr/local/lib/libtss2-mu.so.0.0.0
usr/local/bin/tpm2_unseal
Additionally, I have modified /usr/local/bin/passphrase-from-tpm to the following:
#!/bin/sh
TPM_DEVICE=/dev/tpm0
TPM_REGISTER=0x81000001
TPM_SEAL_POLICY=sha256:0,2,4,7
export TPM2TOOLS_TCTI="device:$TPM_DEVICE"
if [ "$CRYPTTAB_TRIED" -eq 0 ]; then
echo "Unlocking via TPM" >&2
/usr/local/bin/tpm2_unseal -H $TPM_REGISTER -L $TPM_SEAL_POLICY
UNSEAL_STATUS=$?
echo "Unseal status $UNSEAL_STATUS" >&2
if [ $UNSEAL_STATUS -eq 0 ]; then
exit
fi
else
echo "TPM unlocking previously failed for $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)" >&2
/lib/cryptsetup/askpass "Enter passphrase for $CRYPTTAB_SOURCE ($CRYPTTAB_NAME): "
fi
Note that the command line options to tpm2_unseal are for the 3.x versions of tpm2-tools. If you're using another version, you might need to update the options.
I pulled out various bits into variables at the top of the file. Modify TPM_REGISTER and TPM_SEAL_POLICY to match how you created the TPM object. set -e was removed since if any command failed, the whole script would exit, preventing the askpass fallback from ever running if tpm2_unseal failed.
Additionally, I noticed that if the script fails for some reason, systemd will attempt to run it again. If the secret in the TPM doesn't match the LUKS key, this will render the system unbootable, since the unseal succeeds, but unlocking fails, and systemd will run the script again.
Looking at the man page for crypttab, I discovered that one of the environment variables provided to the keyscript is CRYPTTAB_TRIED which is the number of tries it has attempted to unlock the volume. If CRYPTTAB_TRIED is 0, it'll attempt to use the TPM, as shown by this test (Running as non-root, so accessing the TPM device fails):
$ export CRYPTTAB_SOURCE=some_device
$ export CRYPTTAB_NAME=some_device_name
$ export CRYPTTAB_TRIED=0
$ ./passphrase-from-tpm
Unlocking via TPM
ERROR:tcti:src/tss2-tcti/tcti-device.c:440:Tss2_Tcti_Device_Init() Failed to open device file /dev/tpm0: Permission denied
ERROR: tcti init allocation routine failed for library: "device" options: "/dev/tpm0"
ERROR: Could not load tcti, got: "device"
Unseal status 1
When it tries running the script again, CRYPTTAB_TRIED will be greater than 0, making it display the password prompt instead:
$ export CRYPTTAB_TRIED=1
$ ./passphrase-from-tpm
TPM unlocking previously failed for some_device (some_device_name)
Enter passphrase for some_device (some_device_name):
Hopefully this is still of use to you, and helpful to anyone else trying to get the house of cards that is disk encryption with a TPM on Linux working.
I wrote Yocto recipe for library and try to install to image.
When I run command "bitbake myproject" I get follow error:
ERROR: sxe-0.1-r0 do_package: QA Issue: sxe: Files/directories were installed but not shipped in any package:
/sxe
/usr/share
/usr/share/sxm
/usr/share/sxm/baselines
/usr/share/sxm/baselines/resources.psv
/usr/share/sxm/baselines/resources_bcatl.psv
/usr/share/sxm/baselines/version.txt
/usr/share/sxm/baselines/resources_bcatp.psv
/usr/share/sxm/baselines/cgs
/usr/share/sxm/baselines/alerts
/usr/share/sxm/baselines/IntegratedSportsLogos
/usr/share/sxm/baselines/states
/usr/share/sxm/baselines/IntegratedLogos
/usr/share/sxm/baselines/tabweather
/usr/share/sxm/baselines/trafficplus
/usr/share/sxm/baselines/stocks
/usr/share/sxm/baselines/ev
/usr/share/sxm/baselines/contentAssociatedImage/Album020.zip
/usr/share/sxm/baselines/gmd/GEMBase_1.15.zip
/usr/share/sxm/baselines/parking/ParkingLocation2.zip
/usr/share/sxm/baselines/safeview/sxm_safeviewcameralocations_v120.zip
/usr/share/sxm/baselines/safeview/sxm_safeviewcameratypes_v120.zip
/usr/share/sxm/baselines/IntegratedBackgroundLogos/IntPortBack160816.zip
/usr/share/sxm/baselines/IntegratedBackgroundLogos/IntLandBack160816.zip
/usr/share/sxm/baselines/fuel/sxm_fuelbaseline23.pvn1.zip
/usr/share/sxm/baselines/fuel/sxm_canfuelbaseline3.pvn1.zip
/usr/share/sxm/baselines/phonetics/T005V054.psv.zip
/usr/share/sxm/baselines/phonetics/T000V054.psv.zip
/usr/share/sxm/baselines/phonetics/T001V011.psv.zip
/usr/share/sxm/baselines/phonetics/T006V011.psv.zip
/usr/share/sxm/baselines/phonetics/T003V003.psv.zip
/usr/share/sxm/baselines/phonetics/T004V000.psv.zip
/usr/share/sxm/baselines/phonetics/T002V000.psv.zip
/usr/share/sxm/baselines/movies/sxm_theatersbaseline23.pvn1.zip
/usr/share/sxm/baselines/movies/sxm_movieratingsv0.pvn1.zip
Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.
sxe: 55 installed and not shipped files. [installed-vs-shipped]
ERROR: sxe-0.1-r0 do_package: Fatal QA errors found, failing task.
ERROR: sxe-0.1-r0 do_package: Function failed: do_package
ERROR: Logfile of failure stored in: /home/local/SWTECNN/rkhairulin/imx-yocto-bspIII/build-wayland-imx8qmmekII/tmp/work/aarch64-poky-linux/sxe/0.1-r0/temp/log.do_package.32253
ERROR: Task (/home/local/SWTECNN/rkhairulin/imx-yocto-bspIII/sources/meta-sxe/recipes-example/example/sxe_0.1.bb:do_package) failed with exit code '1'
How can I fix the issue? What should I add to the ricpe?
By default, package with recipe name contains following files (see here):
FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
${sysconfdir} ${sharedstatedir} ${localstatedir} \
${base_bindir}/* ${base_sbindir}/* \
${base_libdir}/*${SOLIBS} \
${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \
${datadir}/${BPN} ${libdir}/${BPN}/* \
${datadir}/pixmaps ${datadir}/applications \
${datadir}/idl ${datadir}/omf ${datadir}/sounds \
${libdir}/bonobo/servers"
So files in /usr/share/sxm and /sxe are created in ${D} deploy folder, but they are not added to any package.
You have to add this line to your recipe (see here):
FILES_${PN} += "/sxe ${datadir}/sxm"
Note /usr/share is defined as ${datadir}
If you don't need those files you can remove them from ${D}:
do_install_append(){
rm -r ${D}/sxe
rm -r ${D}/${datadir}/sxm
}
You have to tell bitbake into which package the files shall go. Probably something like
FILES_sxe += " \
{root_prefix}/sxe \
{datadir}/sxe \
"
Having said that, placing something directly under / is strongly discouraged by the FHS. For more information on the predefined prefixes and directories, look at poky/meta/conf/bitbake.conf
I have been going through the documentation of OrientDB about Stress tester. But I couldn't find the script for it in my setup's bin folder.
the script for the stress test was present in version older than 3.x
You can take stresstester.sh from, for example, OrientDB 2.2 and adapt it for your version.
Here is /orientdb/bin/stresstester.sh from 2.2.37:
#!/usr/bin/env bash
# resolve links - $0 may be a softlink
PRG="$0"
while [ -h "$PRG" ]; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG=`dirname "$PRG"`/"$link"
fi
done
# Get standard environment variables
PRGDIR=`dirname "$PRG"`
# Only set ORIENTDB_HOME if not already set
[ -f "$ORIENTDB_HOME"/lib/orientdb-tools-2.2.37.jar ] || ORIENTDB_HOME=`cd "$PRGDIR/.." ; pwd`
export ORIENTDB_HOME
# Set JavaHome if it exists
if [ -f "${JAVA_HOME}/bin/java" ]; then
JAVA=${JAVA_HOME}/bin/java
else
JAVA=java
fi
export JAVA
exec "$JAVA" -XX:MaxDirectMemorySize=512g -cp "$ORIENTDB_HOME/lib/*" \
com.orientechnologies.orient.stresstest.OStressTester $*
In my case I only had to change orientdb-tools-2.2.37.jar to orientdb-tools-3.0.33.jar in order to run this script with OrientDB 3.0.33 I have installed.
I have a user that just has access to pull from github. In my Dockerfile I have added the plugins for Jenkins, such as github:1.22.4, but I want to configure the plugins as some of the people that will build the image won't know how to do the configuration, and don't care to learn.
So, I have some plugins for Jenkins and I want to be able to configure them using the Dockerfile. How can I do that?
My Dockerfile is pretty basic right now:
FROM jenkins
COPY plugins.txt /plugins.txt
RUN /usr/local/bin/plugins.sh /plugins.txt
and I have several plugins in plugins.txt, but the one I want to configure is to pull the code from github.
Did you check this git repository?
lets say you have plugins.txt like:
github:1.22.4
maven-plugin:2.7.1
ant:1.3
and Dockerfile like in your question.
You can take a look into example of plugins.sh and here is part for installing plugins. since you want do configure some plugins you can add configuration when you are installing plugin:
if ! grep -q "${plugin[0]}:${plugin[1]}" "$TEMP_ALREADY_INSTALLED"
then
echo "Downloading ${plugin[0]}:${plugin[1]}"
curl --retry 3 --retry-delay 5 -sSL -f "${JENKINS_UC_DOWNLOAD}/plugins/${plugin[0]}/${plugin[1]}/${plugin[0]}.hpi" -o "$REF/${plugin[0]}.jpi"
unzip -qqt "$REF/${plugin[0]}.jpi"
# if [ some plugin ] then
# here your configuration
# fi
(( COUNT_PLUGINS_INSTALLED += 1 ))
else
echo " ... skipping already installed: ${plugin[0]}:${plugin[1]}"
fi