I have a spark streaming project in pyspark. I have packaged it using pypi so that I can upload to a nexus repository. How can I run the spark submit job if I download the tar.gz file from nexus repository?
Related
Trying to deploy an MULE API and as CI/CD I am using Azure Build Pipeline, but we are getting following error :
Failed to execute goal on project mule-fts-sap-eapi: Could not resolve dependencies for project 1674d8b0-3a4f-4bfb-8c70-89641e023735:mule-fts-sap-eapi:mule-application:1.0.0: The following artifacts could not be resolved: com.sap.conn.jco:com.sap.conn.jco.sapjco3:jar:3.0.19, com.sap.conn.idoc:com.sap.conn.idoc.sapidoc3:jar:3.0.13, com.sap.conn.jco:libsapjco3:dll:3.0.19: Could not find artifact com.sap.conn.jco:com.sap.conn.jco.sapjco3:jar:3.0.19 in Central (https://repo1.maven.org/maven2) ->
We found this dependency not in Maven repo, so what could be the possible way to download this dependency from azure pipeline?
We tried to put the .zip file of dependency in Azure library and but not sure how to pull it, is it possible to store the .zip file of dependency to store somewhere in azure pipeline and unzip and use it when required.
Have you tried to add the Maven Authenticate task before you resolve the artifacts?
- task: MavenAuthenticate#0
inputs:
artifactsFeeds: '**'
I see that the tarball present in http://packages.confluent.io/archive/5.1/confluent-5.1.0-2.11.tar.gz
contains dependency jars arranged in different folder structure ,with dependent jar files distributed according to categories in share/java/ folder
However , when i clone the git of confluent kafka v_5.1 and build the project , all the dependent jar files are present in lib/ folder instead .
Is this because of a different gradle definition ? where can i obtain the gradle file for publishing http://packages.confluent.io/archive/5.1/confluent-5.1.0-2.11.tar.gz
?
Kafka is built with Gradle.
Confluent primarily builds with Maven, and the build scripts are located on private Jenkins servers, with most of the build artifacts uploaded to S3.
The bin/ package for individual projects is just copied as-is (which can be done with some Gradle copy task), or sometimes using the maven-assembly-plugin, (as shown here with the Schema Registry)
The lib/ & share/java folders are done similarly, and could be done with a maven-dependency-plugin
After each individual repo is done, it mostly is up to shell-scripts to move around the folders and re-package everything.
Need some help with writing the gradle dependices to download a single file from github repository
We have a scenario where we need to download .yaml file from the github repository so that we can use it further to autogenerarte it . How do I download a single file form github report via gradle
The best option, if you have freedom to change the source project, is for the yaml to be built, versioned and published to a repository. Eg packed inside a zip/jar in nexus. You could then get the zip/jar in Gradle similar to any other dependency then unpack the yaml file.
Failing that there's a couple of options with github
Download via http (see download plugin)
Get using git (see gradle-git plugin)
I am trying to deploy a war file into WAS8.5 with GitHub. at first I am specifying "EAR Path" with the war file in "Websphere Deployment" option. Then I made some changes in eclipse workspace and pushed into GitHub. Jenkins is checking for latest build but unable to deploy the same in Server. I have used copy artifact plugin, but still nothing happening. I may be wrong in configurations or something. I am getting the following exception:
Copied 0 artifacts from "simplewebapp" build number 20
ERROR: Failed to copy artifacts from simplewebapp with filter: *.war
Archiving artifacts
Finished: FAILURE
and in \jobs\simplewebapp\builds\lastSuccessfulBuild\archive\SimpleWebApp, am unable to see any archive file. How can i achieve it without Maven?
Kindly show me with some examples.
jenkinone
jenkintwo
As a final answer, I recommend to host your project code in GitHub, to use Jenkins/Maven to build your project and to publish the EAR/WAR file to WebSphere using this deployer plugin.
I am trying to upload a play 2.1.3 application to our internal maven repository. I would like the zip file generated by play dist to be uploaded to the repo.
I tried following the steps mentioned in the this Google group thread but the method dependsOn does not seem to exist on TaskKey anymore!. but with no luck; publish continues to publish only the jar file.
How can I make sure the the artifact published by the publish task is the zip file generated by dist?
I ended up using the sbt-aether-deploy sbt plugin to deploy the play application to the nexus repository.