Good Day, I have an Issue uploading Jobs to Flink API using Scala
All Get request seem to work
import scalaj.http._
val url: String = "http://127.0.0.1:8081"
val response: HttpResponse[String] = Http(url+"/config").asString
return response
When I try Uploading a JAR file through CURL (works)
curl -vvv -X POST -H "Expect:" -F "jarfile=#/home/Downloads/myJob.jar" http://127.0.0.1:8081/jars/upload
Now I would Like to upload using SCALA
The documentation does not provide a working example and I am fairly new to this type of post: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs
Currently my code is (Does not Work):
Taken from : https://github.com/Guru107/flinkjobuploadplugin/tree/master/src/main/java/com/github/guru107 - Edited to my needs
// Ideal Case is to upload a Jar File as a multipart in Scala
import java.io.IOException
import org.apache.http.client.methods.HttpPost
import org.apache.http.entity.mime.MultipartEntityBuilder
import org.apache.http.impl.client.{HttpClients, LaxRedirectStrategy}
import org.apache.http.message.BasicHeader
import org.apache.http.util.EntityUtils
val requestUrl = "http://localhost:8081/jars/upload"
val jarPath = "#/home/Downloads/myJob.jar"
val httpClient: CloseableHttpClient = HttpClients.custom.setRedirectStrategy(new LaxRedirectStrategy).build
val fileToUpload: File = new File(jarPath)
val uploadFileUrl: HttpPost = new HttpPost(requestUrl)
val builder: MultipartEntityBuilder = MultipartEntityBuilder.create
builder.addBinaryBody("jarfile", fileToUpload)
val multipart: HttpEntity = builder.build
var jobUploadResponse: JSONObject = null
uploadFileUrl.setEntity(multipart)
var response: CloseableHttpResponse = null
try {
response = httpClient.execute(uploadFileUrl)
println("response: " + response)
response.setHeader(new BasicHeader("Expect", ""))
response.setHeader(new BasicHeader("content-type", "application/x-java-archive"))
val bodyAsString = EntityUtils.toString(response.getEntity, "UTF-8")
println("bodyAsString: " + bodyAsString)
jobUploadResponse = new JSONObject(bodyAsString)
println("jobUploadResponse: " + jobUploadResponse)
}
It fails to upload file.
Please provide a working example or link of scala example to upload a job/jar file to flink in scala
Thanks in Advance
You can use the client code from com.github.mjreid.flinkwrapper
And upload jar file with scala code:
val apiEndpoint: String = as.settings.config.getString("flink.url") //http://<flink_web_host>:<flink_web_port>
val client = FlinkRestClient(apiEndpoint, as)
client.runProgram(<jarId>)
Related
I am trying to connect to Neo4j from Spark using neo4j-spark-connector. I am facing an authentication issue when I try to connect to the Neo4j org.neo4j.driver.v1.exceptions.AuthenticationException: Unsupported authentication token, scheme='none' only allowed when auth is disabled: { scheme='none' }
I have checked and the credentials I am passing are correct. Not sure why is it failing.
import org.neo4j.spark._
import org.apache.spark._
import org.graphframes._
import org.apache.spark.sql.SparkSession
import org.neo4j.driver.v1.GraphDatabase
import org.neo4j.driver.v1.AuthTokens
val config = new SparkConf()
config.set(Neo4jConfig.prefix + "url", "bolt://localhost")
config.set(Neo4jConfig.prefix + "user", "neo4j")
config.set(Neo4jConfig.prefix + "password", "root")
val sparkSession :SparkSession = SparkSession.builder.config(config).getOrCreate()
val neo = Neo4j(sparkSession.sparkContext)
val graphFrame = neo.pattern(("Person","id"),("KNOWS","null"), ("Employee","id")).partitions(3).rows(1000).loadGraphFrame
println("**********Graphframe Vertices Count************")
graphFrame.vertices.count
println("**********Graphframe Edges Count************")
graphFrame.edges.count
val pageRankFrame = graphFrame.pageRank.maxIter(5).run()
val ranked = pageRankFrame.vertices
ranked.printSchema()
val top3 = ranked.orderBy(ranked.col("pagerank").desc).take(3)
Can someone please have a look and let me know the reason for the same?
It might be a configuration issue with your neo4j.conf file. Is this line commented out:
dbms.security.auth_enabled=false
I had a similar problem, creating the following spring beans fixed the issue.
#Bean
public org.neo4j.ogm.config.Configuration getConfiguration() {
return new org.neo4j.ogm.config.Configuration.Builder()
.credentials("neo4j", "secret")
.uri("bolt://localhost:7687").build();
}
#Bean
public SessionFactory sessionFactory(org.neo4j.ogm.config.Configuration configuration) {
return new SessionFactory(configuration,
"<your base package>");
}
I'm trying to use gatling and I have a problem.
1- I have one scenario that exec POST request for getting a list of tokens and save all tokens in csv
2- I create another scenario that exec GET request but I need a token for auth each request
My problem is before executing my first scenario my file doesn't exist and I have this following error:
Could not locate feeder file: Resource user-files/resources/token.csv not found
My code :
Scenario 1 :
val auth_app = scenario("App authentication")
.exec(http("App Authentication")
.post("/token")
.header("Content-Type", "application/x-www-form-urlencoded")
.formParamSeq(Seq(("grant_type", "password"), ("client_id", clientID), ("client_secret", clientSecret)))
.check(jsonPath("$.token").saveAs("token")))
.exec(session => {
val token_data = new File(token_file_path)
if(token_data.exists()){
val writer = new PrintWriter(new FileOutputStream(new File(token_file_path), true))
writer.write(session("access_token").as[String].trim)
writer.write("\n")
writer.close()
}
else {
val writer = new PrintWriter(new FileOutputStream(new File(token_file_path), true))
writer.println("AccessToken")
writer.write(session("access_token").as[String].trim)
writer.write("\n")
writer.close()
}
session
})
Scenario 2 :
val load_catalog = scenario("Load catalog")
.exec(http("Load catalog")
.get("/list")
.headers(Map("Content-Type" -> "application/json", "Authorization Bearer" -> "${AccessToken}")))
.feed(csv(token_file_path).random)
My setup :
setUp(
auth_app.inject(atOnceUsers(10)).protocols(httpProtocol),
load_catalog.inject(nothingFor(120 seconds), atOnceUsers(10)).protocols(httpProtocol)
)
Is it possible to have a dynamic feeder with gatling ?
I'm creating a Kotlin/Jvm (without Android Sdk) application that interacts with a instance of a Parse Server (Back4App). Unfortunately, parse doesn't provide a Sdk implementation to use with Java/Kotlin without Android.
So I'm using the rest Api. Now I trying to upload a image from my disk into Back4App file server. In the doc there is snippet using curl. But I wasn't able to translate into a Retrofit service:
curl -X POST \
-H "X-Parse-Application-Id: 4MGgDJ0ZiQloXoSTE2I9VM6YUYIz8EwCKF4pK7zr" \
-H "X-Parse-REST-API-Key: ${REST_API_KEY}" \
-H "Content-Type: image/jpeg" \
--data-binary '#myPicture.jpg' \
https://YOUR.PARSE-SERVER.HERE/parse/files/pic.jpg
So I based my implementation in this article and other snippets from GitHub and created a retrofit service for it:
#Multipart
#POST("/parse/files")
fun upload(
#Part file: MultipartBody.Part
): Call<ResponseBody>
And call:
var file = File("assets/escudo.png")
var requestFile = RequestBody.create(MediaType.parse("**/image"), file)
var body = MultipartBody.Part.createFormData("picture", file.name, requestFile)
var r = getService().upload(body).execute()
I created the retrofit instance as below:
fun getService(): ParserService {
val retrofit = Retrofit
.Builder()
.baseUrl("https://parseapi.back4app.com")
.addConverterFactory(GsonConverterFactory.create())
.client(createClient()).build()
return retrofit.create(ParserService::class.java)
}
fun createClient(): OkHttpClient {
return OkHttpClient.Builder().addInterceptor(createHeadInterceptor()).build()
}
fun createHeadInterceptor(): Interceptor {
return HeaderInterceptor()
}
class HeaderInterceptor : Interceptor {
override fun intercept(chain: Interceptor.Chain): Response =
chain.run {
val credentials = CredentialsUtils.readCredentials()
log.info { credentials }
proceed(
request().newBuilder()
// .addHeader("Content-Type", "application/json")
.addHeader("Content-Type", "image/png")
.addHeader("X-Parse-Application-Id", credentials.back4appAppId)
.addHeader("X-Parse-REST-API-Key", credentials.back4appRestApiKey)
.build()
)
}
}
I was able to use it to posting Json data (by uncommenting the content/type header). But when I tried to upload an image I receive this response:
Response{protocol=h2, code=400, message=, url=https://parseapi.back4app.com/parse/files}
More info:
-- EDIT
I tried a different approuch without Retrofit, it gives a 201 response code and gives me an objectId, but it doesn't upload the file:
val file2 = File("assets/escudo.png")
val serverUrl = "https://parseapi.back4app.com/classes/myfiles"
val url = URL(serverUrl)
val conn = url.openConnection() as HttpURLConnection
conn.requestMethod = "POST"
conn.doOutput = true
val postData = file2.readBytes()
conn.addRequestProperty("Content-length", postData.size.toString())
conn.setRequestProperty("Content-Type", "image/*")
conn.setRequestProperty("X-Parse-Application-Id", credentials.back4appAppId)
conn.setRequestProperty("X-Parse-REST-API-Key", credentials.back4appRestApiKey)
val outputStream = DataOutputStream(conn.outputStream)
outputStream.write(postData)
outputStream.flush()
println(conn.responseCode)
-- EDIT
Trying now using Khttp:
val file = File("assets/foto.jpg")
val file2 = File("assets/escudo.png")
val serverUrl = "https://parseapi.back4app.com/classes/myfiles"
val files = listOf(FileLike("foto.jpg", file), FileLike("escudo.png", file2))
val response = post(serverUrl, headers = getHeaders(), files = files)
println(response)
println(response.text)
}
fun getHeaders(): Map<String, String> {
return mapOf(
"Content-Type" to "image/*",
"X-Parse-Application-Id" to credentials.back4appAppId,
"X-Parse-REST-API-Key" to credentials.back4appRestApiKey
)
}
Getting this error:
<Response [400]>
{"error":"Unexpected token - in JSON at position 0"}
If you're using Back4App, the correct Server URL is:
https://parseapi.back4app.com/files/pic.jpg
How to do HTTP PUT/POSTs from inside Groovy code without having to import any libraries (if at all possible)? I know there is a simple getText() methods that Groovy adds to the java.net.URL class, that could be used without adding any dependencies. Is there a way to do Rest PUT in the same fashion?
You can do it with HttpURLConnection in a similar way as you would do it with java:
def url = new URL('http://your_rest_endpoint')
def http = url.openConnection()
http.setDoOutput(true)
http.setRequestMethod('PUT')
http.setRequestProperty('User-agent', 'groovy script')
def out = new OutputStreamWriter(http.outputStream)
out.write('data')
out.close()
http.inputStream // read server response from it
import groovyx.net.http.RESTClient
import static groovyx.net.http.ContentType.JSON
import groovy.json.JsonSlurper
import groovy.json.JsonOutput
url = "http://restapi3.apiary.io"
#Grab (group = 'org.codehaus.groovy.modules.http-builder', module = 'http-builder', version = '0.5.0')
def client = new RESTClient(url)
def jsonObj = new JsonSlurper().parseText('{ "title": "Pick-up posters from Post-Office" }')
def response = client.put(path: "/notes/id",
contentType: JSON,
body: jsonObj,
headers: [Accept: 'application/json'])
println("Status: " + response.status)
if (response.data) {
println("Content Type: " + response.contentType)
println("Headers: " + response.getAllHeaders())
println("Body:\n" + JsonOutput.prettyPrint(JsonOutput.toJson(response.data)))
}
I have a project with Scala Play - Finagle and ElasticSearch
I am using a Finagle client with the old version of the API, like this:
val client = ClientBuilder()
.codec(Http)
.hosts("localhost:10000,localhost:10001,localhost:10003")
.hostConnectionLimit(1)
.build()
Here is my code:
https://gist.github.com/hectorgool/f217d16e2c15b122d7a7
and works fine, but now i want to upgrade my code to a new APIs version, like this:
val client = Http.newService("localhost:10000,localhost:10001")
The new version of my code is here:
https://github.com/hectorgool/es-client/blob/master/app/lib/FinagleClient.scala
But, now my project does not compile, I have an error with this line(111):
111: val client = clientFactory.apply()()
I don't know how to fix it
Solved
I change this:
val clientFactory: ServiceFactory[HttpRequest, HttpResponse] = ClientBuilder()
.codec(Http())
.hosts(hosts)
.tcpConnectTimeout(1.second)
.hostConnectionLimit(1)
.buildFactory()
for this:
val clientFactory: Service[HttpRequest, HttpResponse] = Http.newService(hosts)
I sumprime this:
val client = clientFactory.apply()()
And i change this:
httpResponse.onSuccess{
response =>
Logger.debug("Received response: " + response)
client.close()
}.onFailure{ err: Throwable =>
Logger.error(err.toString)
client.close()
}
for this:
httpResponse.onSuccess{
response =>
Logger.debug("Received response: " + response)
}.onFailure{ err: Throwable =>
Logger.error(err.toString)
}
the cause of the problem is:
client.close()
because closing the connections