I have a rust project where i include the mysql-crate dependency and i want to have it os independent.
So i tried: Cargo.toml
[package]
name = "test"
version = "0.1.0"
authors = ["daMaex"]
[dependencies]
ws = "*"
clap = "*"
env_logger = "*"
[target.'cfg(any(unix, macos))'.dependencies.mysql]
version = "*"
default-features = false
features = ["socket"]
[target.'cfg(windows)'.dependencies.mysql]
version = "*"
default-features = false
features = ["pipe"]
[features]
default = []
ssl = []
The error already happens with a minimal main: src/main.rs
fn main () {
}
But the build fails. On macos/unix it always wants to compile the pipe and get's an unresolved import:
error[E0432]: unresolved import `std::os::windows::io::RawHandle`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/named_pipe-0.2.2/src/lib.rs:38:5
|
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Could not find `windows` in `std::os`
and the same happens on Windows for the mysql crate itself:
error[E0432]: unresolved import `std::os::unix`
--> C:\Users\user\.cargo\registry\src\github.com-1ecc6299db9ec823\mysql-7.1.2\src\io.rs:24:5
|
24 | use std::os::unix as unix;
| ^^^^^^^^^^^^^^^^^^^^^ no `unix` in `std::os`
So my question is, how do i handle the OS-Dependency in this case?
This looks like a cargo bug to me.
One related issue is Cannot declare different version of dependencies in mutually exclusive targets #3195
Edit: It is more of an unsupported feature than a bug according to the code.
Related
I'm trying to cross-compile rust application using cross. Hello application compiles with no issue. The problem is when I try to use a diesel package with postgres support which depends on libpq library.
I did the installation of the libpq-dev package on my machine and I included a sys-pq package to my Cargo.toml, but it did not help.
How can I solve this?
Error:
➜ cross build --target armv7-unknown-linux-musleabi
Compiling libc v0.2.138
Compiling cfg-if v1.0.0
Compiling version_check v0.9.4
Compiling autocfg v1.1.0
Compiling memchr v2.5.0
...
= note: /usr/local/bin/../lib/gcc/arm-linux-musleabi/9.2.0/../../../../arm-linux-musleabi/bin/ld: cannot find -lpq
collect2: error: ld returned 1 exit status
➜ cat Cargo.toml
[package]
name = "webserver"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
actix-web = "4"
env_logger = "0.9"
futures-util = { version = "0.3.17", default-features = false, features = ["std"] }
json = "0.12"
log = "0.4"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
diesel = { version = "2.0.0", features = ["postgres"] }
dotenvy = "0.15"
# Cross dependencies
pq-sys = "0.4.7"
[profile.release]
strip = true
opt-level = "z"
lto = true
Useful info
I am building a webapp and using as back-end rocket.rs, with diesel.rs to connect to the Postgres database. I wanto to dockerize this webapp: I'm on Windows 11, and I want to cross-compile for Linux alpine, so I can just add the executable in the container.
Toolchain: nightly-2021-12-05-x86_64-pc-windows-msvc
Target: x86_64-unknown-linux-musl
Cargo.toml file:
[package]
name = "project-api"
version = "1.0.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
rocket = { version = "0.4.10", default-features = false }
diesel = { version = "1.4.8", features = ["postgres"] }
serde = { version = "1.0.131", features = ["derive"]}
dotenv = "0.15.0"
[dependencies.rocket_contrib]
version = "0.4.10"
default-features = false
features = ["json", "serve", "diesel_postgres_pool"]
.cargo/config.toml file:
[target.x86_64-unknown-linux-musl]
linker = "rust-lld"
Compile command: cargo build --target x86_64-unknown-linux-musl --release
In my PATH I already added:
.../PostgreSQL/14/bin
.../PostgreSQL/14/lib
.../PostgreSQL/14/include
The problem
When I compile the project, I get the following error:
error: linking with `rust-lld` failed: exit code: 1
...
= note: rust-lld: error: unable to find library -llibpq
This is the complete error:
Compiling project-api v1.0.0 (D:\projects\project-management-system)
error: linking with `rust-lld` failed: exit code: 1
|
= note: "rust-lld" "-flavor" "gnu" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\rcrt1.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crti.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtbeginS.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.0.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.1.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.10.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.11.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.12.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.13.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.14.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.15.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.2.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.3.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.4.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.5.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.6.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.7.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.8.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.9.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.22qos1iwh0pu1l3f.rcgu.o" "--as-needed" "-L" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps" "-L" "D:\\projects\\project-management-system\\target\\release\\deps" "-L" "D:/programs/PostgreSQL/14/lib" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib" "-Bstatic" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libdotenv-17787ac5e36ba405.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket_contrib-908e672c5a94e5a1.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libdiesel-b618b4aafb14a3d4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpq_sys-0deff7ab575185bc.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbitflags-cd7b464110d56932.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libr2d2-17991d433d6b8815.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libscheduled_thread_pool-ed3fe870ef2578f4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libparking_lot-084e1692fb5bda8c.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libparking_lot_core-57505ee8fbe2715a.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblock_api-c4af51bd9cc3c613.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libscopeguard-d10f930610969f1b.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libinstant-5b170197aa126111.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libserde_json-f02c39ef48f71da7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libryu-62ce0cec239f97c8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libitoa-8ec78a5baa91caac.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket-28b96e263956caad.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libatty-11406b7a761cb014.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbase64-d9fe97a8863a409c.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmemchr-1dd949379d427674.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtoml-6efc66767f0cacd7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libserde-4add0bb05fd83ce8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libyansi-077b82a357bfe296.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket_http-9426f414126ee4c8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhyper-5854b3142fe943a8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmime-41e7a782b7da09c1.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblog-8290fbf24ebfc650.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblog-52629d6d7adb881b.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libcfg_if-b1347d601d516de4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblanguage_tags-1f120d4eb4c64568.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtypeable-37b90a66c0e53fd4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtraitobject-e42a82912129d07a.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libnum_cpus-2c9ebf56e8506369.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhttparse-5fdd6843d2b90a53.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicase-1ff8862237f0ad7e.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liburl-cb2f1703c2051404.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libidna-2dccc330fa120c40.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_normalization-05c479f908735c18.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtinyvec-89542cb3fe5ec50d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtinyvec_macros-0d79dc173afd5963.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_bidi-25a8617db7cb03aa.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmatches-5a438e594be46ce5.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbase64-1d678e0cfd664cb9.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libsafemem-7dc2f1cd7551f3b9.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbyteorder-2481812d3e7c910d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_xid-d2b897ae1242d958.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libstate-d915de2c2db68eef.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libindexmap-3190e1dd9e2809b7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhashbrown-4794792eab7bd8e3.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libcookie-b22759f25ba38a69.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtime-6373f03850cf300d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblibc-d520e3667288b8da.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpercent_encoding-ee9e622516b87769.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libsmallvec-2817ef9eba1ed532.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpercent_encoding-dd42f0c66c80313d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpear-f2f7fd587bab036d.rlib" "--start-group" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libstd-84fa96c5a6567c2a.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libpanic_unwind-2694d6b5ee3337f3.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libminiz_oxide-58e7d510a302cb06.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libadler-3b97f36d6f391442.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libobject-7f2f74160dfc1203.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libmemchr-bc62538a6b3d16cf.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libaddr2line-531fc0db5fed2ff4.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libgimli-267be56b670e3626.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libstd_detect-5efe0f28e08ac5c2.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_demangle-5cb8086f7dea7d11.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libhashbrown-26229185d2048651.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_std_workspace_alloc-5e539cff88cca34d.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libunwind-a33879496a1412b2.rlib" "-lunwind" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcfg_if-cb8b44b450e6e086.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\liblibc-9946a456dc1d9b49.rlib" "-lc" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\liballoc-7fa17c9ac5f3bbe0.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_std_workspace_core-d23d1fdf2f9c2e82.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcore-4927c240c984c2ab.rlib" "--end-group" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcompiler_builtins-ff05cc303f3269e0.rlib" "-Bdynamic" "-llibpq" "--eh-frame-hdr" "-znoexecstack" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained" "-o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7" "--gc-sections" "-static" "-pie" "--no-dynamic-linker" "-z" "text" "-zrelro" "-znow" "-O1" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtendS.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtn.o"
= note: rust-lld: error: unable to find library -llibpq
error: could not compile `project-api` due to previous error
What I tried
I already tried adding the libpq.dll and libpq.lib files to the x86_64-unknown-linux-musl/lib folder, but I got the same error. I tried adding it to the default target too, but it didn't work. I found out it could be a problem of some build.rs file in a library... but I don't know which one. I also tried compiling with the regular nightly toolchain.
Im trying to add the finagle-http library to my new bazel project as an external maven dependency. But getting the following error. I assume im doing something wrong in creating the build without fully understanding it. Trying to learning. Appreciate any help on this.
error: object Service is not a member of package com.twitter.finagle
error: object util is not a member of package com.twitter
error: type Request is not a member of package com.twitter.finagle.http
error: object Response is not a member of package com.twitter.finagle.http
error: Symbol 'type com.twitter.finagle.Client' is missing from the classpath. This symbol is required by 'object com.twitter.finagle.Http'.
error: not found: value Await
The same code is working using sbt. Below is the code.
import com.twitter.finagle.{Http, Service}
import com.twitter.finagle.http
import com.twitter.util.{Await, Future}
object HelloWorld extends App {
val service = new Service[http.Request, http.Response] {
def apply(req: http.Request): Future[http.Response] =
Future.value(http.Response(req.version, http.Status.Ok))
}
val server = Http.serve(":8080", service)
Await.ready(server)
}
WORKSPACE file
maven_install(
artifacts = [
"org.apache.spark:spark-core_2.11:2.4.4",
"org.apache.spark:spark-sql_2.11:2.4.1",
"org.apache.spark:spark-unsafe_2.11:2.4.1",
"org.apache.spark:spark-tags_2.11:2.4.1",
"org.apache.spark:spark-catalyst_2.11:2.4.1",
"com.twitter:finagle-http_2.12:21.8.0",
],
repositories = [
"https://repo.maven.apache.org/maven2/",
"https://repo1.maven.org/maven2/",
]
)
BUILD file
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_binary")
package(default_visibility = ["//visibility:public"])
scala_binary(
name="helloworld",
main_class="microservices.HelloWorld",
srcs=[
"Main.scala",
],
deps = ["spark],
)
java_library(
name = "spark",
exports = [
"#maven//:com_twitter_finagle_http_2_12_21_8_0",
],
)
Working SBT dependency that was working in my initial sbt project
libraryDependencies += "com.twitter" %% "finagle-http" % "21.8.0"
Figured out the issue, unlike in sbt, in bazel i had induvidualy add the related dependencies. I modified the workspace as below.
maven_install(
artifacts = [
"com.twitter:finagle-http_2.12:21.8.0",
"com.twitter:util-core_2.12:21.8.0",
"com.twitter:finagle-core_2.12:21.8.0",
"com.twitter:finagle-base-http_2.12:21.8.0",
"com.fasterxml.jackson.module:jackson-module-scala_2.12:2.11.2",
"com.fasterxml.jackson.core:jackson-databind:2.11.2",
],
repositories = [
"https://repo.maven.apache.org/maven2/",
"https://repo1.maven.org/maven2/",
]
Build file --
java_library(
name = "finagletrial",
exports = [
"#maven//:com_twitter_finagle_http_2_12_21_8_0",
"#maven//:com_twitter_util_core_2_12_21_8_0",
"#maven//:com_twitter_finagle_core_2_12_21_8_0",
"#maven//:com_twitter_finagle_base_http_2_12_21_8_0",
"#maven//:com_fasterxml_jackson_module_jackson_module_scala_2_12_2_11_2",
"#maven//:com_fasterxml_jackson_core_jackson_databind_2_11_2"
],
I am trying to setup a simple flink application from scratch using Bazel. I've bootstrapped the project by running
sbt new tillrohrmann/flink-project.g8
and after that I have added some files in order for Bazel to take control of the building (i.e., migrate from sbt). This is how the WORKSPACE looks like
# WORKSPACE
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
skylib_version = "1.0.3"
http_archive(
name = "bazel_skylib",
sha256 = "1c531376ac7e5a180e0237938a2536de0c54d93f5c278634818e0efc952dd56c",
type = "tar.gz",
url = "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib-{}.tar.gz".format(skylib_version, skylib_version),
)
rules_scala_version = "5df8033f752be64fbe2cedfd1bdbad56e2033b15"
http_archive(
name = "io_bazel_rules_scala",
sha256 = "b7fa29db72408a972e6b6685d1bc17465b3108b620cb56d9b1700cf6f70f624a",
strip_prefix = "rules_scala-%s" % rules_scala_version,
type = "zip",
url = "https://github.com/bazelbuild/rules_scala/archive/%s.zip" % rules_scala_version,
)
# Stores Scala version and other configuration
# 2.12 is a default version, other versions can be use by passing them explicitly:
load("#io_bazel_rules_scala//:scala_config.bzl", "scala_config")
scala_config(scala_version = "2.12.11")
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()
load("#io_bazel_rules_scala//scala:toolchains.bzl", "scala_register_toolchains")
scala_register_toolchains()
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_binary", "scala_test")
# optional: setup ScalaTest toolchain and dependencies
load("#io_bazel_rules_scala//testing:scalatest.bzl", "scalatest_repositories", "scalatest_toolchain")
scalatest_repositories()
scalatest_toolchain()
load("//vendor:workspace.bzl", "maven_dependencies")
maven_dependencies()
load("//vendor:target_file.bzl", "build_external_workspace")
build_external_workspace(name = "vendor")
and this is the BUILD file
package(default_visibility = ["//visibility:public"])
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "job",
srcs = glob(["src/main/scala/**/*.scala"]),
deps = [
"#vendor//vendor/org/apache/flink:flink_clients",
"#vendor//vendor/org/apache/flink:flink_scala",
"#vendor//vendor/org/apache/flink:flink_streaming_scala",
]
)
I'm using bazel-deps for vendoring the dependencies (put in the vendor folder). I have this on my dependencies.yaml file:
options:
buildHeader: [
"load(\"#io_bazel_rules_scala//scala:scala_import.bzl\", \"scala_import\")",
"load(\"#io_bazel_rules_scala//scala:scala.bzl\", \"scala_library\", \"scala_binary\", \"scala_test\")",
]
languages: [ "java", "scala:2.12.11" ]
resolverType: "coursier"
thirdPartyDirectory: "vendor"
resolvers:
- id: "mavencentral"
type: "default"
url: https://repo.maven.apache.org/maven2/
strictVisibility: true
transitivity: runtime_deps
versionConflictPolicy: highest
dependencies:
org.apache.flink:
flink:
lang: scala
version: "1.11.2"
modules: [clients, scala, streaming-scala] # provided
flink-connector-kafka:
lang: java
version: "0.10.2"
flink-test-utils:
lang: java
version: "0.10.2"
For downloading the dependencies, I'm running
bazel run //:parse generate -- --repo-root ~/Projects/bazel-flink-scala --sha-file vendor/workspace.bzl --target-file vendor/target_file.bzl --deps dependencies.yaml
Which runs just fine, but then when I try to build the project
bazel build //:job
I'm getting this error
Starting local Bazel server and connecting to it...
ERROR: Traceback (most recent call last):
File "/Users/salvalcantara/Projects/me/bazel-flink-scala/WORKSPACE", line 44, column 25, in <toplevel>
build_external_workspace(name = "vendor")
File "/Users/salvalcantara/Projects/me/bazel-flink-scala/vendor/target_file.bzl", line 258, column 91, in build_external_workspace
return build_external_workspace_from_opts(name = name, target_configs = list_target_data(), separator = list_target_data_separator(), build_header = build_header())
File "/Users/salvalcantara/Projects/me/bazel-flink-scala/vendor/target_file.bzl", line 251, column 40, in list_target_data
"vendor/org/apache/flink:flink_clients": ["lang||||||scala:2.12.11","name||||||//vendor/org/apache/flink:flink_clients","visibility||||||//visibility:public","kind||||||import","deps|||L|||","jars|||L|||//external:jar/org/apache/flink/flink_clients_2_12","sources|||L|||","exports|||L|||","runtimeDeps|||L|||//vendor/commons_cli:commons_cli|||//vendor/org/slf4j:slf4j_api|||//vendor/org/apache/flink:force_shading|||//vendor/com/google/code/findbugs:jsr305|||//vendor/org/apache/flink:flink_streaming_java_2_12|||//vendor/org/apache/flink:flink_core|||//vendor/org/apache/flink:flink_java|||//vendor/org/apache/flink:flink_runtime_2_12|||//vendor/org/apache/flink:flink_optimizer_2_12","processorClasses|||L|||","generatesApi|||B|||false","licenses|||L|||","generateNeverlink|||B|||false"],
Error: dictionary expression has duplicate key: "vendor/org/apache/flink:flink_clients"
ERROR: error loading package 'external': Package 'external' contains errors
INFO: Elapsed time: 3.644s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
Why is that? Anyone can help? It would be great having detailed instructions and project templates for Flink/Scala applications using Bazel. I've put everything together in the following repo: https://github.com/salvalcantara/bazel-flink-scala, feel free to send a PR or whatever.
I just discovered RollUP and I am stuck with an issue where Babel does not run for imports from node_modules.
Here is an example:
My Javascript Code
import { _map } from "#varunsridharan/js-vars"
const myarray = _map.call([1,2,3,4],(x)=> x * 2);
console.log(myarray);
Rollup Config
import { babel } from '#rollup/plugin-babel';
import { nodeResolve } from '#rollup/plugin-node-resolve';
import { uglify } from 'rollup-plugin-uglify';
import filesize from 'rollup-plugin-filesize';
import visualizer from 'rollup-plugin-visualizer';
export default {
input: './src/index.js',
output: {
file: './dist/myfile.min.js',
format: 'iife',
plugins: [
uglify( { mangle: true } ),
]
},
plugins: [
nodeResolve(),
babel(),
filesize(),
visualizer()
]
};
When I run rollup -c in CLI I get this output:
babelHelpers: 'bundled' option was used by default. It is recommended to configure this option explicitly, read more here: https://github.com/rollup/plugins/tree/master/packages/babel#babelhelpers
198 | * Array Related Vars.
199 | */
> 200 | const Arr = Array;
| ^ Unexpected token: keyword «const»
201 | const _Arrayprop = Arr.prototype;
202 | const _filter = _Arrayprop.filter;
203 | const _push = _Arrayprop.push;
[!] (plugin uglify) Error: Unexpected token: keyword «const»
SyntaxError: Unexpected token: keyword «const»
at JS_Parse_Error.get (eval at <anonymous> (E:\localhost\www\javascript\dizzle\node_modules\uglify-js\tools\node.js:18:1), <anonymous>:69:23)
at reportError (E:\localhost\www\javascript\dizzle\node_modules\jest-worker\build\workers\processChild.js:107:11)
at reportClientError (E:\localhost\www\javascript\dizzle\node_modules\jest-worker\build\workers\processChild.js:87:10)
at execFunction (E:\localhost\www\javascript\dizzle\node_modules\jest-worker\build\workers\processChild.js:157:5)
at execHelper (E:\localhost\www\javascript\dizzle\node_modules\jest-worker\build\workers\processChild.js:139:5)
at execMethod (E:\localhost\www\javascript\dizzle\node_modules\jest-worker\build\workers\processChild.js:143:5)
at process.<anonymous> (E:\localhost\www\javascript\dizzle\node_modules\jest-worker\build\workers\processChild.js:64:7)
at process.emit (events.js:315:20)
at emit (internal/child_process.js:876:12)
at processTicksAndRejections (internal/process/task_queues.js:85:21)
Based on the output i was able to understand that babel did not run for the imported modules. so i checking the options provided for rollup babel plugin # (https://github.com/rollup/plugins/tree/master/packages/babel) and found that it has include AND exclude options and i tried with the below config
babel( {
include: [ "node_modules/#varunsridharan/*/**", "./src/**" ],
exclude: "node_modules/**",
} ),
Still, nothing happened so I tried without ./src/** in babel include config and found that babel is not running in my main javascript file which imports the node_modules's file
Node Module Project Link: https://github.com/varunsridharan/js-vars