error: object Service is not a member of package com.twitter.finagle - Defining Bazel dependencies in Build file, Scala finagle - scala

Im trying to add the finagle-http library to my new bazel project as an external maven dependency. But getting the following error. I assume im doing something wrong in creating the build without fully understanding it. Trying to learning. Appreciate any help on this.
error: object Service is not a member of package com.twitter.finagle
error: object util is not a member of package com.twitter
error: type Request is not a member of package com.twitter.finagle.http
error: object Response is not a member of package com.twitter.finagle.http
error: Symbol 'type com.twitter.finagle.Client' is missing from the classpath. This symbol is required by 'object com.twitter.finagle.Http'.
error: not found: value Await
The same code is working using sbt. Below is the code.
import com.twitter.finagle.{Http, Service}
import com.twitter.finagle.http
import com.twitter.util.{Await, Future}
object HelloWorld extends App {
val service = new Service[http.Request, http.Response] {
def apply(req: http.Request): Future[http.Response] =
Future.value(http.Response(req.version, http.Status.Ok))
}
val server = Http.serve(":8080", service)
Await.ready(server)
}
WORKSPACE file
maven_install(
artifacts = [
"org.apache.spark:spark-core_2.11:2.4.4",
"org.apache.spark:spark-sql_2.11:2.4.1",
"org.apache.spark:spark-unsafe_2.11:2.4.1",
"org.apache.spark:spark-tags_2.11:2.4.1",
"org.apache.spark:spark-catalyst_2.11:2.4.1",
"com.twitter:finagle-http_2.12:21.8.0",
],
repositories = [
"https://repo.maven.apache.org/maven2/",
"https://repo1.maven.org/maven2/",
]
)
BUILD file
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_binary")
package(default_visibility = ["//visibility:public"])
scala_binary(
name="helloworld",
main_class="microservices.HelloWorld",
srcs=[
"Main.scala",
],
deps = ["spark],
)
java_library(
name = "spark",
exports = [
"#maven//:com_twitter_finagle_http_2_12_21_8_0",
],
)
Working SBT dependency that was working in my initial sbt project
libraryDependencies += "com.twitter" %% "finagle-http" % "21.8.0"

Figured out the issue, unlike in sbt, in bazel i had induvidualy add the related dependencies. I modified the workspace as below.
maven_install(
artifacts = [
"com.twitter:finagle-http_2.12:21.8.0",
"com.twitter:util-core_2.12:21.8.0",
"com.twitter:finagle-core_2.12:21.8.0",
"com.twitter:finagle-base-http_2.12:21.8.0",
"com.fasterxml.jackson.module:jackson-module-scala_2.12:2.11.2",
"com.fasterxml.jackson.core:jackson-databind:2.11.2",
],
repositories = [
"https://repo.maven.apache.org/maven2/",
"https://repo1.maven.org/maven2/",
]
Build file --
java_library(
name = "finagletrial",
exports = [
"#maven//:com_twitter_finagle_http_2_12_21_8_0",
"#maven//:com_twitter_util_core_2_12_21_8_0",
"#maven//:com_twitter_finagle_core_2_12_21_8_0",
"#maven//:com_twitter_finagle_base_http_2_12_21_8_0",
"#maven//:com_fasterxml_jackson_module_jackson_module_scala_2_12_2_11_2",
"#maven//:com_fasterxml_jackson_core_jackson_databind_2_11_2"
],

Related

rust-lld: error: unable to find library -llibpq

Useful info
I am building a webapp and using as back-end rocket.rs, with diesel.rs to connect to the Postgres database. I wanto to dockerize this webapp: I'm on Windows 11, and I want to cross-compile for Linux alpine, so I can just add the executable in the container.
Toolchain: nightly-2021-12-05-x86_64-pc-windows-msvc
Target: x86_64-unknown-linux-musl
Cargo.toml file:
[package]
name = "project-api"
version = "1.0.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
rocket = { version = "0.4.10", default-features = false }
diesel = { version = "1.4.8", features = ["postgres"] }
serde = { version = "1.0.131", features = ["derive"]}
dotenv = "0.15.0"
[dependencies.rocket_contrib]
version = "0.4.10"
default-features = false
features = ["json", "serve", "diesel_postgres_pool"]
.cargo/config.toml file:
[target.x86_64-unknown-linux-musl]
linker = "rust-lld"
Compile command: cargo build --target x86_64-unknown-linux-musl --release
In my PATH I already added:
.../PostgreSQL/14/bin
.../PostgreSQL/14/lib
.../PostgreSQL/14/include
The problem
When I compile the project, I get the following error:
error: linking with `rust-lld` failed: exit code: 1
...
= note: rust-lld: error: unable to find library -llibpq
This is the complete error:
Compiling project-api v1.0.0 (D:\projects\project-management-system)
error: linking with `rust-lld` failed: exit code: 1
|
= note: "rust-lld" "-flavor" "gnu" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\rcrt1.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crti.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtbeginS.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.0.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.1.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.10.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.11.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.12.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.13.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.14.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.15.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.2.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.3.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.4.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.5.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.6.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.7.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.8.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.9.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.22qos1iwh0pu1l3f.rcgu.o" "--as-needed" "-L" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps" "-L" "D:\\projects\\project-management-system\\target\\release\\deps" "-L" "D:/programs/PostgreSQL/14/lib" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib" "-Bstatic" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libdotenv-17787ac5e36ba405.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket_contrib-908e672c5a94e5a1.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libdiesel-b618b4aafb14a3d4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpq_sys-0deff7ab575185bc.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbitflags-cd7b464110d56932.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libr2d2-17991d433d6b8815.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libscheduled_thread_pool-ed3fe870ef2578f4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libparking_lot-084e1692fb5bda8c.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libparking_lot_core-57505ee8fbe2715a.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblock_api-c4af51bd9cc3c613.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libscopeguard-d10f930610969f1b.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libinstant-5b170197aa126111.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libserde_json-f02c39ef48f71da7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libryu-62ce0cec239f97c8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libitoa-8ec78a5baa91caac.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket-28b96e263956caad.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libatty-11406b7a761cb014.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbase64-d9fe97a8863a409c.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmemchr-1dd949379d427674.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtoml-6efc66767f0cacd7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libserde-4add0bb05fd83ce8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libyansi-077b82a357bfe296.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket_http-9426f414126ee4c8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhyper-5854b3142fe943a8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmime-41e7a782b7da09c1.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblog-8290fbf24ebfc650.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblog-52629d6d7adb881b.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libcfg_if-b1347d601d516de4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblanguage_tags-1f120d4eb4c64568.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtypeable-37b90a66c0e53fd4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtraitobject-e42a82912129d07a.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libnum_cpus-2c9ebf56e8506369.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhttparse-5fdd6843d2b90a53.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicase-1ff8862237f0ad7e.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liburl-cb2f1703c2051404.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libidna-2dccc330fa120c40.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_normalization-05c479f908735c18.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtinyvec-89542cb3fe5ec50d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtinyvec_macros-0d79dc173afd5963.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_bidi-25a8617db7cb03aa.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmatches-5a438e594be46ce5.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbase64-1d678e0cfd664cb9.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libsafemem-7dc2f1cd7551f3b9.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbyteorder-2481812d3e7c910d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_xid-d2b897ae1242d958.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libstate-d915de2c2db68eef.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libindexmap-3190e1dd9e2809b7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhashbrown-4794792eab7bd8e3.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libcookie-b22759f25ba38a69.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtime-6373f03850cf300d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblibc-d520e3667288b8da.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpercent_encoding-ee9e622516b87769.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libsmallvec-2817ef9eba1ed532.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpercent_encoding-dd42f0c66c80313d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpear-f2f7fd587bab036d.rlib" "--start-group" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libstd-84fa96c5a6567c2a.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libpanic_unwind-2694d6b5ee3337f3.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libminiz_oxide-58e7d510a302cb06.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libadler-3b97f36d6f391442.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libobject-7f2f74160dfc1203.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libmemchr-bc62538a6b3d16cf.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libaddr2line-531fc0db5fed2ff4.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libgimli-267be56b670e3626.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libstd_detect-5efe0f28e08ac5c2.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_demangle-5cb8086f7dea7d11.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libhashbrown-26229185d2048651.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_std_workspace_alloc-5e539cff88cca34d.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libunwind-a33879496a1412b2.rlib" "-lunwind" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcfg_if-cb8b44b450e6e086.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\liblibc-9946a456dc1d9b49.rlib" "-lc" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\liballoc-7fa17c9ac5f3bbe0.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_std_workspace_core-d23d1fdf2f9c2e82.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcore-4927c240c984c2ab.rlib" "--end-group" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcompiler_builtins-ff05cc303f3269e0.rlib" "-Bdynamic" "-llibpq" "--eh-frame-hdr" "-znoexecstack" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained" "-o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7" "--gc-sections" "-static" "-pie" "--no-dynamic-linker" "-z" "text" "-zrelro" "-znow" "-O1" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtendS.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtn.o"
= note: rust-lld: error: unable to find library -llibpq
error: could not compile `project-api` due to previous error
What I tried
I already tried adding the libpq.dll and libpq.lib files to the x86_64-unknown-linux-musl/lib folder, but I got the same error. I tried adding it to the default target too, but it didn't work. I found out it could be a problem of some build.rs file in a library... but I don't know which one. I also tried compiling with the regular nightly toolchain.

What is the preferred way in mill to point to the directory where build.sc file is located?

What is the preferred way in mill to point to the directory where build.sc file is located?
In the mill documentation os.pwd is used e.g. here, but if it is possible to start/run mill from elsewhere, then os.pwd points to incorrect location.
import mill._, mill.modules.Jvm
def sourceRoot = T.sources { os.pwd / "src" }
def resourceRoot = T.sources { os.pwd / "resources" }
The root location of each defined module is defined in def millSourcePath: os.Path, and it's the preferred way to access the module path.
Example:
import mill._, mill.scalalib._
object mymodule extends JavaModule {
def sources = T.sources(millSourcePath / "src")
}
$ mill show __.sources
No mill version specified.
You should provide a version via '.mill-version' file or --mill-version option.
Using mill version 0.9.9
Compiling /tmp/stackoverflow/build.sc
[1/1] show
[1/1] show > [1/1] mymodule.sources
[
"ref:c984eca8:/tmp/stackoverflow/mymodule/src"
]
If you need to access the path of project itself, notice, that it's a mill.define.Module itself and you can access it via the variable build. The project directory is therefore accessible via build.millSourcePath.
Example:
import mill._, mill.scalalib._
val baseDir = build.millSourcePath
object mymodule extends JavaModule {
def sources = T.sources {
super.sources() ++ Seq(PathRef(baseDir / "src"))
}
}
$ mill show __.sources
No mill version specified.
You should provide a version via '.mill-version' file or --mill-version option.
Using mill version 0.9.9
Compiling /tmp/stackoverflow/build.sc
[1/1] show
[1/1] show > [2/2] mymodule.sources
[
"ref:c984eca8:/tmp/stackoverflow/mymodule/src",
"ref:c984eca8:/tmp/stackoverflow/src"
]

Setup of Scala/Flink project using Bazel

I am trying to setup a simple flink application from scratch using Bazel. I've bootstrapped the project by running
sbt new tillrohrmann/flink-project.g8
and after that I have added some files in order for Bazel to take control of the building (i.e., migrate from sbt). This is how the WORKSPACE looks like
# WORKSPACE
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
skylib_version = "1.0.3"
http_archive(
name = "bazel_skylib",
sha256 = "1c531376ac7e5a180e0237938a2536de0c54d93f5c278634818e0efc952dd56c",
type = "tar.gz",
url = "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib-{}.tar.gz".format(skylib_version, skylib_version),
)
rules_scala_version = "5df8033f752be64fbe2cedfd1bdbad56e2033b15"
http_archive(
name = "io_bazel_rules_scala",
sha256 = "b7fa29db72408a972e6b6685d1bc17465b3108b620cb56d9b1700cf6f70f624a",
strip_prefix = "rules_scala-%s" % rules_scala_version,
type = "zip",
url = "https://github.com/bazelbuild/rules_scala/archive/%s.zip" % rules_scala_version,
)
# Stores Scala version and other configuration
# 2.12 is a default version, other versions can be use by passing them explicitly:
load("#io_bazel_rules_scala//:scala_config.bzl", "scala_config")
scala_config(scala_version = "2.12.11")
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()
load("#io_bazel_rules_scala//scala:toolchains.bzl", "scala_register_toolchains")
scala_register_toolchains()
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_binary", "scala_test")
# optional: setup ScalaTest toolchain and dependencies
load("#io_bazel_rules_scala//testing:scalatest.bzl", "scalatest_repositories", "scalatest_toolchain")
scalatest_repositories()
scalatest_toolchain()
load("//vendor:workspace.bzl", "maven_dependencies")
maven_dependencies()
load("//vendor:target_file.bzl", "build_external_workspace")
build_external_workspace(name = "vendor")
and this is the BUILD file
package(default_visibility = ["//visibility:public"])
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "job",
srcs = glob(["src/main/scala/**/*.scala"]),
deps = [
"#vendor//vendor/org/apache/flink:flink_clients",
"#vendor//vendor/org/apache/flink:flink_scala",
"#vendor//vendor/org/apache/flink:flink_streaming_scala",
]
)
I'm using bazel-deps for vendoring the dependencies (put in the vendor folder). I have this on my dependencies.yaml file:
options:
buildHeader: [
"load(\"#io_bazel_rules_scala//scala:scala_import.bzl\", \"scala_import\")",
"load(\"#io_bazel_rules_scala//scala:scala.bzl\", \"scala_library\", \"scala_binary\", \"scala_test\")",
]
languages: [ "java", "scala:2.12.11" ]
resolverType: "coursier"
thirdPartyDirectory: "vendor"
resolvers:
- id: "mavencentral"
type: "default"
url: https://repo.maven.apache.org/maven2/
strictVisibility: true
transitivity: runtime_deps
versionConflictPolicy: highest
dependencies:
org.apache.flink:
flink:
lang: scala
version: "1.11.2"
modules: [clients, scala, streaming-scala] # provided
flink-connector-kafka:
lang: java
version: "0.10.2"
flink-test-utils:
lang: java
version: "0.10.2"
For downloading the dependencies, I'm running
bazel run //:parse generate -- --repo-root ~/Projects/bazel-flink-scala --sha-file vendor/workspace.bzl --target-file vendor/target_file.bzl --deps dependencies.yaml
Which runs just fine, but then when I try to build the project
bazel build //:job
I'm getting this error
Starting local Bazel server and connecting to it...
ERROR: Traceback (most recent call last):
File "/Users/salvalcantara/Projects/me/bazel-flink-scala/WORKSPACE", line 44, column 25, in <toplevel>
build_external_workspace(name = "vendor")
File "/Users/salvalcantara/Projects/me/bazel-flink-scala/vendor/target_file.bzl", line 258, column 91, in build_external_workspace
return build_external_workspace_from_opts(name = name, target_configs = list_target_data(), separator = list_target_data_separator(), build_header = build_header())
File "/Users/salvalcantara/Projects/me/bazel-flink-scala/vendor/target_file.bzl", line 251, column 40, in list_target_data
"vendor/org/apache/flink:flink_clients": ["lang||||||scala:2.12.11","name||||||//vendor/org/apache/flink:flink_clients","visibility||||||//visibility:public","kind||||||import","deps|||L|||","jars|||L|||//external:jar/org/apache/flink/flink_clients_2_12","sources|||L|||","exports|||L|||","runtimeDeps|||L|||//vendor/commons_cli:commons_cli|||//vendor/org/slf4j:slf4j_api|||//vendor/org/apache/flink:force_shading|||//vendor/com/google/code/findbugs:jsr305|||//vendor/org/apache/flink:flink_streaming_java_2_12|||//vendor/org/apache/flink:flink_core|||//vendor/org/apache/flink:flink_java|||//vendor/org/apache/flink:flink_runtime_2_12|||//vendor/org/apache/flink:flink_optimizer_2_12","processorClasses|||L|||","generatesApi|||B|||false","licenses|||L|||","generateNeverlink|||B|||false"],
Error: dictionary expression has duplicate key: "vendor/org/apache/flink:flink_clients"
ERROR: error loading package 'external': Package 'external' contains errors
INFO: Elapsed time: 3.644s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
Why is that? Anyone can help? It would be great having detailed instructions and project templates for Flink/Scala applications using Bazel. I've put everything together in the following repo: https://github.com/salvalcantara/bazel-flink-scala, feel free to send a PR or whatever.

How to set up a local development environment for Scala Spark ETL to run in AWS Glue?

I'd like to be able to write Scala in my local IDE and then deploy it to AWS Glue as part of a build process. But I'm having trouble finding the libraries required to build the GlueApp skeleton generated by AWS.
The aws-java-sdk-glue doesn't contain the classes imported, and I can't find those libraries anywhere else. Though they must exist somewhere, but perhaps they are just a Java/Scala port of this library: aws-glue-libs
The template scala code from AWS:
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.MappingSpec
import com.amazonaws.services.glue.errors.CallSite
import com.amazonaws.services.glue.util.GlueArgParser
import com.amazonaws.services.glue.util.Job
import com.amazonaws.services.glue.util.JsonOptions
import org.apache.spark.SparkContext
import scala.collection.JavaConverters._
object GlueApp {
def main(sysArgs: Array[String]) {
val spark: SparkContext = new SparkContext()
val glueContext: GlueContext = new GlueContext(spark)
// #params: [JOB_NAME]
val args = GlueArgParser.getResolvedOptions(sysArgs, Seq("JOB_NAME").toArray)
Job.init(args("JOB_NAME"), glueContext, args.asJava)
// #type: DataSource
// #args: [database = "raw-tickers-oregon", table_name = "spark_delivery_2_1", transformation_ctx = "datasource0"]
// #return: datasource0
// #inputs: []
val datasource0 = glueContext.getCatalogSource(database = "raw-tickers-oregon", tableName = "spark_delivery_2_1", redshiftTmpDir = "", transformationContext = "datasource0").getDynamicFrame()
// #type: ApplyMapping
// #args: [mapping = [("exchangeid", "int", "exchangeid", "int"), ("data", "struct", "data", "struct")], transformation_ctx = "applymapping1"]
// #return: applymapping1
// #inputs: [frame = datasource0]
val applymapping1 = datasource0.applyMapping(mappings = Seq(("exchangeid", "int", "exchangeid", "int"), ("data", "struct", "data", "struct")), caseSensitive = false, transformationContext = "applymapping1")
// #type: DataSink
// #args: [connection_type = "s3", connection_options = {"path": "s3://spark-ticker-oregon/target", "compression": "gzip"}, format = "json", transformation_ctx = "datasink2"]
// #return: datasink2
// #inputs: [frame = applymapping1]
val datasink2 = glueContext.getSinkWithFormat(connectionType = "s3", options = JsonOptions("""{"path": "s3://spark-ticker-oregon/target", "compression": "gzip"}"""), transformationContext = "datasink2", format = "json").writeDynamicFrame(applymapping1)
Job.commit()
}
}
And the build.sbt I have started putting together for a local build:
name := "aws-glue-scala"
version := "0.1"
scalaVersion := "2.11.12"
updateOptions := updateOptions.value.withCachedResolution(true)
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.1"
The documentation for AWS Glue Scala API seems to outline similar functionality as is available in the AWS Glue Python library. So perhaps all that is required is to download and build the PySpark AWS Glue library and add it on the classpath? Perhaps possible since the Glue python library uses Py4J.
#Frederic gave a very helpful hint to get the dependency from s3://aws-glue-jes-prod-us-east-1-assets/etl/jars/glue-assembly.jar.
Unfortunately that version of glue-assembly.jar is already outdated and brings spark in versoin 2.1.
It's fine if you're using backward compatible features, but if you rely on latest spark version (and possibly latest glue features) you can get the appropriate jar from a Glue dev-endpoint under /usr/share/aws/glue/etl/jars/glue-assembly.jar.
Provided you have a dev-endpoint named my-dev-endpoint you can copy the current jar from it:
export DEV_ENDPOINT_HOST=`aws glue get-dev-endpoint --endpoint-name my-dev-endpoint --query 'DevEndpoint.PublicAddress' --output text`
scp -i dev-endpoint-private-key \
glue#$DEV_ENDPOINT_HOST:/usr/share/aws/glue/etl/jars/glue-assembly.jar .
Unfortunately, there are no libraries available for Scala glue API. Already contacted amazon support and they are aware about this problem. However, they didn't provide any ETA for delivering API jar.
As a workaround you can download the jar from S3. The S3 URI is s3://aws-glue-jes-prod-us-east-1-assets/etl/jars/glue-assembly.jar
See https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-tutorial-repl.html
now it supports, a recent release from AWS.
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html

Cargo.toml OS Dependency for Crate

I have a rust project where i include the mysql-crate dependency and i want to have it os independent.
So i tried: Cargo.toml
[package]
name = "test"
version = "0.1.0"
authors = ["daMaex"]
[dependencies]
ws = "*"
clap = "*"
env_logger = "*"
[target.'cfg(any(unix, macos))'.dependencies.mysql]
version = "*"
default-features = false
features = ["socket"]
[target.'cfg(windows)'.dependencies.mysql]
version = "*"
default-features = false
features = ["pipe"]
[features]
default = []
ssl = []
The error already happens with a minimal main: src/main.rs
fn main () {
}
But the build fails. On macos/unix it always wants to compile the pipe and get's an unresolved import:
error[E0432]: unresolved import `std::os::windows::io::RawHandle`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/named_pipe-0.2.2/src/lib.rs:38:5
|
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Could not find `windows` in `std::os`
and the same happens on Windows for the mysql crate itself:
error[E0432]: unresolved import `std::os::unix`
--> C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\mysql-7.1.2\src\io.rs:24:5
|
24 | use std::os::unix as unix;
| ^^^^^^^^^^^^^^^^^^^^^ no `unix` in `std::os`
So my question is, how do i handle the OS-Dependency in this case?
This looks like a cargo bug to me.
One related issue is Cannot declare different version of dependencies in mutually exclusive targets #3195
Edit: It is more of an unsupported feature than a bug according to the code.