Pyspark write fails on overwrite mode in bigquery database - pyspark

Caused by: java.lang.IllegalArgumentException: com.google.cloud.bigquery.connector.common.BigQueryConnectorException$InvalidSchemaException: Destination table's schema is not compatible with dataframe's schema
df.write \
.format('bigquery') \
.option('table', (project + '.db.tbl')) \
.mode("overwrite") \
.save()

For some reason the schema validation checks became much stricter on June 1st 2022. I suspect there was an update to the dataproc image. We're using 2.0.29-debian10.

This is due to the latest version of BQ spark connector (0.25.0) : https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases
Fix your version to the previous version (0.24.2) and you will not have this issue.

I had the same situation since 1st jully 2022, it seems like you should not use the latest versions as recommended in google documentation :
For non-production use, you can also point to the latest jars, as follows:
Dataproc image version 1.5 and above: --jars=gs://spark-lib/bigquery/spark->bigquery-latest_2.12.jar
I was working with the latest version and as soon as i changed the spark-biquery version to : gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.23.2.jar -> It worked just fine.
Edit: For more information about the upgrade (schema equality) see here.

I had the same issue, and when I upgraded to an even newer version, the problem went away.
The failing version:
gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.25.0.jar
The working version:
gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.27.0.jar

Related

OpenAPI npm cli is not generating spring code using v3 oas

I'm generating spring server code using the below command
openapi-generator generate -g spring -o eventspec -i pets.yaml
Pets.yaml is Petstore 3.0.0 yaml file.
Code is getting generated properly, however the springfox dependency is getting adding and API code is annotated with io.swagger.annotations not with io.swagger.v3.oas.annotations
Let me know if i'm missing some thing.
Thanks
I don't think anyone has the cycle to update the template to use io.swagger.v3.oas.annotations. We welcome contributions from the community for such enhancement.
Ref: https://github.com/OpenAPITools/openapi-generator/issues/4245

adapter Ecto.Adapters.Postgres was not compiled

I am not able to create my Phoenix project. Would love some advice on how to fix it.
Setup details:
Ubuntu 16.04.4 LTS
Erlang/OTP 21 [erts-10.1] [source] [64-bit]
[smp:1:1] [ds:1:1:10] [async-threads:1] [hipe]
Elixir 1.7.3 (compiled
with Erlang/OTP 20)
Mix 1.7.3 (compiled with Erlang/OTP 20)
Ecto v3.0.0
I am following the Phoenix Up and Running to make an app.
mix phx.new hello
cd hello
mix ecto.create
last command gives me:
== Compilation error in file lib/hello/repo.ex ==
** (ArgumentError) adapter Ecto.Adapters.Postgres was not compiled, ensure it is correct and it is included as a project dependency
lib/ecto/repo/supervisor.ex:71: Ecto.Repo.Supervisor.compile_config/2
lib/hello/repo.ex:2: (module)
(stdlib) erl_eval.erl:680: :erl_eval.do_apply/6
(elixir) lib/kernel/parallel_compiler.ex:206: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6
I have postgres installed. I have postgres super user.
Starting with Ecto 3.0, Ecto.Adapters.Postgres is not shipped with Ecto by default, therefore you have to add ecto_sql to the Mixfile dependencies:
###########
# mix.exs #
###########
defp deps do
[
# (...)
{:ecto_sql, "~> 3.0-rc.1"},
{:postgrex, ">= 0.0.0"}
]
end
# Feeling skittish about dependencies,
# I usually do this instead of simply
# doing `mix deps.get`:
$ mix deps.clean --all
$ mix do deps.get, compile
(The Ecto github repo v3.0.0 tree recommends {:ecto_sql, "~> 3.0"}, but the latest release is the 3.0.0-rc.1) therefore it won't work as of now. Interestingly there is no 3.0.0-rc.1 tag in the repo, but the documentation already refers to that and it also works with mix.)
...or, as Yufrend recommends in his answer, if you are starting a new Phoenix project, use < 1.4.0 packages.
See José Valim's “A sneak peek at Ecto 3.0” series where the first post explains the breaking changes in Ecto 3.0:
Split Ecto into ecto and ecto_sql
Ecto 3.0 will be broken in two repositories: ecto and ecto_sql.
Since Ecto 2.0, an increased number of developers and teams have been
using Ecto for data mapping and validation, without a need for a
database. However, adding Ecto to your application would still bring a
lot of the SQL baggage, such as adapters, sandboxes and migrations,
which many considered to be a mixed message.
In Ecto 3.0, we will move all of the SQL adapters to a separate
repository and Ecto will focus on the four building blocks: schemas,
changesets, queries and repos. You can see the discussion in the
issues tracker.
If you are using Ecto with a SQL database, migrating to Ecto 3.0 will
be very straight-forward. Instead of:
{:ecto, "~> 2.2"}
You should list:
{:ecto_sql, "~> 3.0"}
And if you are using Ecto only for data manipulation but with no
database access, then it is just a matter of bumping its version.
That’s it!
UPDATE
For some reason, I also needed to add {:plug_cowboy, "~> 1.0"} to the Mixfile dependencies when updating a Phoenix 1.3 project and it all started working.
Do you have phoenix_ecto 3.5.0 in your dependencies? Downgrading to 3.4.0 worked for me as a temporary fix until I figure out the underlying issue.
To force a downgrade:
Run mix deps.clean --all
Delete your mix.lock file
Update your mix.exs file limiting the phoenix_ecto version. Find the appropriate line and replace with:
{:phoenix_ecto, ">= 3.2.0 and < 3.5.0"},
Run mix deps.get
Alternatively, if you are just starting with Phoenix, you could use version 1.4 to learn, which will be released soon and does not have this issue.
First remove your current local Phoenix archive:
mix archive.uninstall phx_new
Then, to install the latest development version, follow the instructions in https://github.com/phoenixframework/phoenix/blob/master/installer/README.md
Installing the new phoenix version worked for me.
Uninstall old version:
mix archive.uninstall phx_new
Install new version:
mix archive.install hex phx_new 1.4.0-rc.2
New Projects
For creating new projects with Ecto 3.0, it's highly recommended you upgrade to the new phoenix 1.4.x installer:
$ mix archive.uninstall phx_new
$ mix archive.install hex phx_new 1.4.0-rc.2
Existing Projects
To upgrade your existing Phoenix 1.3.x projects to 1.4, read the Official Upgrade Guide and the accompanying announcement.
TLDR is that Ecto has been broken into sub-packages, and you need to specify them explicitly:
Remove your explicit :ecto dependency and update your :phoenix_ecto and :ecto_sql dependencies with the following versions:
{:ecto_sql, "~> 3.0-rc"},
{:phoenix_ecto, "~> 4.0"},

Meteor package (yogiben:admin) refuses to install, seems to be picking up wrong version from github?

I'm using aldeed:autoform#5. I also want to use yogiben's autoform-file, but I had trouble getting that to work with autoform#5 so I've used a fork by abdj:autoform-file.
Now I want to use yogiben:admin. According to the discussion on github and the latest master (https://github.com/yogiben/meteor-admin/blob/master/package.js), yogiben:admin#1.1.0 uses aldeed:autoform#4.2.2 || 5.0.0.
So why do I get this error?
C:\Webdev\koolaid>meteor add yogiben:admin#1.1.0
=> Errors while adding packages:
While selecting package versions:
error: Conflict: Constraint aldeed:autoform#4.2.2 is not satisfied by aldeed:autoform 5.1.2.
Constraints on package "aldeed:autoform":
* aldeed:autoform#5.0.2 <- abdj:autoform-file 0.2.0
* aldeed:autoform#4.2.2 <- yogiben:admin 1.1.0
Looks to me like Meteor is somehow picking up the wrong version from github? BTW I'm using Meteor WindowsPREVIEW#0.3.0.
How do I get all these packages to play nice together?

EventStore build.cmd Errors

Just getting started with JOlivers event store library and having issues with some of the Nuget commands in the build.cmd. here's the log, perhaps you're already aware of the issue but i thought i'd put it up here.
=== COMPILING ===
Compiling / Target: v4.0 / Config: Release
S:\SourceControl\Test Projects\EventStore\bin\nuget\NuGet.targets(6,9): error : Input string was not in a correct format. [S:\SourceControl\Test Projects\EventStore\src\proj\EventStore.Persistence.RavenPersistence\EventStore.Persistence.RavenPersistence.csproj]
S:\SourceControl\Test Projects\EventStore\bin\nuget\NuGet.targets(6,9): error MSB3073: The command ""S:\SourceControl\Test Projects\EventStore\src\..\bin\nuget\nuget.exe" install "S:\SourceControl\Test Projects\EventStore\src\proj\EventStore.Persistence.RavenPersistence\packages.config" -source "" -o "S:\SourceControl\Test Projects\EventStore\src\..\bin"" exited with code 1. [S:\SourceControl\Test Projects\EventStore\src\proj\EventStore.Persistence.RavenPersistence\EventStore.Persistence.RavenPersistence.csproj]
The master branch of EventStore appears to be broken at the moment. The 3.0 branch is in better shape but is also slightly broken. To fix the 3.0 branch so that it builds, you need to upgrade the 1.5 version of nuget.exe to 1.6. The easiest way to do that is to execute \bin\nuget\NuGet.exe update -self in the repository. I already submitted a pull request to JOliver with that change.
(I also submitted a pull request with a few minor fixes for the example project.)

Intellij IDEA showing error on legit Scala code

I am new to Scala and IntelliJ IDEA. I am learning Lift from Simply Lift examples. I have setup my IDE as per instructions provided here. I am using Maven based approach.The Scala plugin is versioned (0.4.1084 -- latest as of June 30, 2011). IntelliJ IDEA version is 10.5.
The archetype is
-DarchetypeGroupId=net.liftweb \
-DarchetypeArtifactId=lift-archetype-blank_2.8.1 \
-DarchetypeVersion=2.3 \
-DarchetypeRepository=http://scala-tools.org/repo-releases \
-DremoteRepositories=http://scala-tools.org/repo-releases \
-DgroupId=com.company \
-DartifactId=lift_test \
-Dversion=1.0
When I started coding the Snippet and Sitemap example. I see IDE show error (and nasty wavy line below the code) at the following line
Menu.i("Info") / "info" submenus (
Menu.i("About") / "about" >> Hidden >> LocGroup("bottom"),
Menu.i("Contact") / "contact",
Menu.i("Feedback") / "feedback" >> LocGroup("bottom")
),
The error says,
Expression of type (Menu.Menuable , Menu.Menuable with Menu.withSlash,
Menu.Menuable) doesn't conform to expected type ConvertableToMenu
However, when I run a Maven > Install or Compile 'Boot.scala from IDEA they seem to run without error.
I am having hard time on getting Scala/Lift working perfectly on either of the two IDEs, IDEA and Eclipse.
Please let me know what am I missing?
I have added a bug report, you may see it here http://youtrack.jetbrains.net/issue/SCL-3429?projectKey=SCL
The plugin is still under development. Good code red and bad code green are both issues that arise in every day usage. If you want to help with the development of the plugin, please file a bug report with a short example demonstrating your issue.
http://youtrack.jetbrains.net/issues/SCL