I've billed set of support function into helper.jar library and imported to Databricks cluster. The jar is installed on the cluster, but I'm not able to reference the functions in the library.
The jar import has been tested, cluster restarted and the jar can be referenced in InelliJ where it was developed as Azure Spark/HDInsight project.
//next line generates error value helper is not a member of org.apache.spark.sql.SparkSession
import helper
//nex line generates error: not found: value fn_conversion
display(df.withColumn("RevenueConstantUSD", fn_conversion($"Revenue"))
I'd expect the helper function would be visible after library deployment or possibly after adding the import command.
Edit: added information about IntelliJ project type
Related
I am running databricks 7.3LTS and having errors while trying to use scala bulk copy.
The error is:
object sqldb is not a member of package com.microsoft.
I have installed correct sqlconnector drivers but not sure how to fix this error.
The installed drivers are:
com.microsoft.azure:spark-mssql-connector_2.12:1.1.0.
also i have installed the JAR dependencies as below:
spark_mssql_connector_2_12_1_1_0.jar
i couldnt find any scala code example for the above configurations on the internet.
my scala code sample is as below:
%scala
import com.microsoft.azure.sqldb.spark.config.Config
as soon as i run this command i get the error
Object sqldb is not a member of package com.microsoft.azure
any help please
In the new connector you need to use com.microsoft.sqlserver.jdbc.spark.SQLServerBulkJdbcOptions class to specify bulk copy options.
I have a JAR that I created with intellij and sbt, that defines a case class and Object. I've uploaded it to my databricks workspace, and attached it to a cluster as a library.
How do I actually import code from it/reference it in my notebook? So far all I can think of is to try
import NameOfJar._
which gives
java.lang.NoClassDefFoundError: DataFrameComparison$
Do I need to have built the jar differently somehow? (Package statement or something?)
you should import import packageName._, jar name is not used in import statement. It should work the same as in usual local java/scala code.
You can check this article for details - https://docs.databricks.com/libraries.html
Btw, does your notebook fail on import itself, or later, when you're trying to use class, that exists inside jar?
I uploaded the spark time series spark-ts library to DataBricks using maven coordinate option in the Create Library. I was able to successfully create the library and attach it to my cluster. But when I tried to import the spark-ts library in DataBricks using org.apache.spark.spark-ts. But it throws an error stating that notebook:1: error: object ts is not a member of package org.apache.spark Please let me know how to handle this issue.
I want to use phantom with my scala IDE.So for this i clone the git hub repository and created a .jar file of phantom using sbt -> compile -> package.I add this .jar file to build path in my Scala IDE but still while importing
import com.websudos.phantom.connectors._
is throwing error that
object connector is not a member of com.websudos.phantom.
While using auto complete function of scala ide it is showing only the import for
import com.websudos.phantom.example
.I don't know if the jar files got created for example then why it is not created for other.
I search in internet but all other option are given as to add dependency in sbt build path but i dont want to use it.
Use sbt-assebly instead to create a fat jar.
https://github.com/sbt/sbt-assembly
I am working with Spark streaming and have written a custom streaming adapter. I want to export this adapter as a jar and use it in my scala streaming jobs. When I refer the jar inside my streaming code, I am getting this error:
import org.custom.streaming
[ERROR] object custom is not a member of package org
Note that the adapter doesn't have any main method, so I can't use generic methods available online to export the project as a runnable JAR.
I also tried exporting it as a shaded JAR but in that case I am getting:
error: error while loading <root>, error in opening zip file
[EDIT]
I am using maven for packaging
Have you considered using the package command of your maven build file ?