I am trying to get the description of a bq table using the following:
val bigquery = BigQueryOptions.getDefaultInstance().getService()
val table = bigquery.getTable(tableId)
tableDescription = table.getDescription()
what I am thinking of what if not description is provided to the table I am trying to get the description from? It's written in the documenation that getDescription returns string, but what if there is no descprition provided? will it return null? and if yes, how do I avoid this using an option for example or any other way?
The documentation indicates that the method has the signature:
public String getDescription()
It's reasonable to assume that it might return null since we're dealing with a Java library. You can handle this safely by wrapping the expression as an Option:
tableDescription: Option[String] = Option(table.getDescription())
Related
I am working on building a flink application which reads data from kafka
topics, apply some transformations and writes to the Iceberg table.
I read the data from kafka topic (which is in json) and use circe to decode
that to scala case class with scala Option values in it.
All the transformations on the datastream works fine.
Case Class Looks like below
Event(app_name: Option[String], service_name: Option[String], ......)
But when I try to convert the stream to a table to write to iceberg table
due to the case classes the columns are converted to Raw type as shown
below.
table.printSchema()
service_name RAW('scala.Option', '...'),
conversion_id RAW('scala.Option', '...'),
......
And the table write fails as below.
Query schema: [app_name: RAW('scala.Option', '...'), .........
Sink schema: [app_name: STRING, .......
Does flink table api support scala case classes with option values?
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#special-types
I found out that it is supported in datastream at this documentation.
Is there a way to do this in Table API.
Thanks in advance for the help..
I had the exact same issue of Option being parsed as RAW and found yet another workaround that might interest you:
TL;DR:
instead of using .get which returns an Option and diligently declaring the return type to be Types.OPTION(Types.INSTANT) which doesn't work, I instead use .getOrElse("key", null) and declare the type as conventional. Table API then recognizes the column type, creates a nullable column and interprets the null correctly. I can then filter those rows with IS NOT NULL.
Detailed example:
For me it starts with a custom map function in which I unpack data where certain fields could be missing:
class CustomExtractor extends RichMapFunction[MyInitialRecordClass, Row] {
def map(in: MyInitialRecordClass): Row = {
Row.ofKind(
RowKind.INSERT,
in._id
in.name
in.times.getOrElse("time", null) // This here did the trick instead of using .get and returning an option
)
}
}
And then I use it like this explicitly declaring a return type.
val stream: DataStream[Row] = data
.map(new CustomExtractor())
.returns(
Types.ROW_NAMED(
Array("id", "name", "time"),
Types.STRING,
Types.STRING,
Types.INSTANT
)
val table = tableEnv.fromDataStream(stream)
table.printSchema()
// (
// `id` STRING,
// `name` STRING,
// `time` TIMESTAMP_LTZ(9),
// )
tableEnv.createTemporaryView("MyTable", table)
tableEnv
.executeSql("""
|SELECT
| id, name, time
|FROM MyTable""".stripMargin)
.print()
// +----+------------------+-------------------+---------------------+
// | op | id | name | time |
// +----+------------------+-------------------+---------------------+
// | +I | foo | bar | <Null> |
// | +I | spam | ham | 2022-10-12 11:32:06 |
This was at least for me exactly what I wanted. I'm very much new to Flink and would be curious if the Pros here think this is workable or horribly hacky instead.
Using scala 2.12.15 and flink 1.15.2
The type system of the Table API is more restrictive than the one of the DataStream API. Unsupported classes are immediately treated as black-boxed type RAW. This allows objects to still pass the API but it might not be supported by every connector.
From the exception, it looks like you declared the sink table with app_name: STRING, so I guess you are fine with a string representation. If this is the case, I would recommend to implement a user-defined function that performs the conversion to string.
I would like a generic function that converts the result of a SQL query to JSON. I would like to build a JSON string manually (or use an external library). For that to happen, I need to be able to enumerate the columns in a row dynamically.
let rows = client
.query("select * from ExampleTable;")
.await?;
// This is how you read a string if you know the first column is a string type.
let thisValue: &str = rows[0].get(0);
Dynamic types are possible with Rust, but not with the tokio-postgres library API.
The row.get function of tokio-postgres is designed to require generic inference according to the source code
Without the right API, how can I enumerate rows and columns?
You need to enumerate the rows and columns, doing so you can get the column reference while enumerating, and from that get the postgresql-type. With the type information it's possible to have conditional logic to choose different sub-functions to both: i) get the strongly typed variable; and, ii) convert to a JSON value.
for (rowIndex, row) in rows.iter().enumerate() {
for (colIndex, column) in row.columns().iter().enumerate() {
let colType: string = col.type_().to_string();
if colType == "int4" { //i32
let value: i32 = row.get(colIndex);
return value.to_string();
}
else if colType == "text" {
let value: &str = row.get(colIndex);
return value; //TODO: escape characters
}
//TODO: more type support
else {
//TODO: raise error
}
}
}
Bonus tips for tokio-postgres code maintainers
Ideally, tokio-postgres would include a direct API that returns a dyn any type. The internals of row.rs already use the database column type information to confirm that the supplied generic type is valid. Ideally a new API uses would use the internal column information quite directly with improved FromSQL API, but a simpler middle-ground exists:-
It would be possible for an extra function layer in row.rs that uses the same column type conditional logic used in this answer to then leverage the existing get function. If a user such as myself needs to handle this kind of conditional logic, I also need to maintain this code when new types are handled by tokio-postgresql, therefore, this kind of logic should be included inside the library where such functionality can be better maintained.
I'm having some issues with a Wicket (8.0.0-M4) NumberTextField in Kotlin (1.1.0).
My stripped-down form looks like this:
class Test : AbstractWebPage() {
val housenumberModel: Model<Int> = Model<Int>()
val housenumber = NumberTextField<Int>("housenumberModel", housenumberModel)
val form: Form<Unit> = object : Form<Unit>("adressForm") {}
override fun onInitialize() {
super.onInitialize()
form.add(housenumber.setRequired(false))
form.add(object : SubmitLink("submit") {
override fun onSubmit() {
super.onSubmit()
println(housenumberModel.`object`) // this is line 28
}
})
add(form)
}
}
After submitting the form I get the following stacktrace:
java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Number
at com.mycompany.test.pages.Test$onInitialize$1.onSubmit(Test.kt:28)
at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1312)
at org.apache.wicket.markup.html.form.Form.process(Form.java:979)
at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:802)
at org.apache.wicket.markup.html.form.Form.onRequest(Form.java:715)
at org.apache.wicket.core.request.handler.ListenerRequestHandler.internalInvoke(ListenerRequestHandler.java:301)
at org.apache.wicket.core.request.handler.ListenerRequestHandler.invoke(ListenerRequestHandler.java:250)
at org.apache.wicket.core.request.handler.ListenerRequestHandler.invokeListener(ListenerRequestHandler.java:210)
at org.apache.wicket.core.request.handler.ListenerRequestHandler.respond(ListenerRequestHandler.java:203)
at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:912)
at org.apache.wicket.request.RequestHandlerExecutor.execute(RequestHandlerExecutor.java:65)
at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:283)
at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:253)
at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:221)
at org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:262)
at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:204)
at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:286)
[...]
If I use
val housenumberModel: Model<Int> = Model.of(0)
instead of
val housenumberModel: Model<Int> = Model<Int>()
everything works fine. But since my NumberTextField is optional I don't want to have it pre-initialized with 0.
Me and my colleagues were trying to change the type signature of the Model in every way we could imagine but came to no solution. A co-worker suggested to write a custom Wicket converter since Kotlins Int is represendeted as a primitive type (From the docs: "On the JVM, non-nullable values of this type are represented as values of the primitive type int.") Even though I don't know yet if this would work it seems like an overkill for me.
Another hack I could think of: writing some JavaScript to delete the zero from the input field. Also not really something I would want to do.
Question: Is there a simple solution to my problem?
(And as a bonus-question: has already anyone written a larger Wicket application in Kotlin and could tell me if this combination is ready for prime time to develop a critical project with this stack or is my problem just the tip of the iceberg?)
[edit]
Solution as pointed out by svenmeier:
Using
val housenumber = NumberTextField<Int>("housenumberModel", housenumberModel, Int::class.java)
works.
Or as an alternative:
val housenumbervalue: Int? = null
val housenumberModel: IModel<Int> = PropertyModel<Int>(this, "housenumbervalue")
val housenumber = NumberTextField<Int>("housenumberModel", housenumberModel)
Because of type erasure your NumberTextField cannot detect the generic type parameter of your model. Since your model object is null, it cannot be used to derive the type either.
In this case Wicket assumes a String model object type :/.
Either provide the type to the NumberTextField explicitly, or use a model that keeps its generic information, e.g. a PropertyModel.
There is a way to tell wicket about the type you want, it is by adding the type in the constructor. More here.
In Java it looks like this:
new NumberTextField<Integer>("housenumberModel", housenumberModel, Integer.class);
This is the query I am trying to run in PostgreSQL:
SELECT * FROM message WHERE id IN (
SELECT unnest(message_ids) "mid"
FROM session_messages WHERE session_id = '?' ORDER BY "mid" ASC
);
However, I am not able do something:
create.selectFrom(Tables.MESSAGE).where(Tables.MESSAGE.ID.in(
create.select(DSL.unnest(..))
Because DSL.unnest is a Table<?>, which makes sense since it is trying to take a List-like object (mostly a literal) and convert it to table.
I have a feeling that I need to find a way to wrap the function around my field name, but I have no clue as to how to proceed.
NOTE. The field message_ids is of type bigint[].
EDIT
So, this is how I am doing it now, and it works exactly as expected, but I am not sure if this is the best way to do it:
Field<Long> unnestMessageIdField = DSL.field(
"unnest(" + SESSION_MESSAGES.MESSAGE_IDS.getName() + ")",
Long.class)
.as("mid");
Field<Long> messageIdField = DSL.field("mid", Long.class);
MESSAGE.ID.in(
ctx.select(messageIdField).from(
ctx.select(unnestMessageIdField)
.from(Tables.CHAT_SESSION_MESSAGES)
.where(Tables.CHAT_SESSION_MESSAGES.SESSION_ID.eq(sessionId))
)
.where(condition)
)
EDIT2
After going through the code on https://github.com/jOOQ/jOOQ/blob/master/jOOQ/src/main/java/org/jooq/impl/DSL.java I guess the right way to do this would be:
DSL.function("unnest", SQLDataTypes.BIGINT.getArrayType(), SESSION_MESSAGES.MESSAGE_IDS)
EDIT3
Since as always lukas is here for my jOOQ woes, I am going to capitalize on this :)
Trying to generalize this function, in a signature of sort
public <T> Field<T> unnest(Field<T[]> arrayField) {
return DSL.function("unnest", <??>, arrayField);
}
I don't know how I can fetch the datatype. There seems to be a way to get DataType<T[]> from DataType<T> using DataType::getArrayDataType(), but the reverse is not possible. There is this class I found ArrayDataType, but it seems to be package-private, so I cannot use it (and even if I could, it does not expose the field elementType).
Old PostgreSQL versions had this funky idea that it is OK to produce a table from within the SELECT clause, and expand it into the "outer" table, as if it were declared in the FROM clause. That is a very obscure PostgreSQL legacy, and this example is a good chance to get rid of it, and use LATERAL instead. Your query is equivalent to this one:
SELECT *
FROM message
WHERE id IN (
SELECT "mid"
FROM session_messages
CROSS JOIN LATERAL unnest(message_ids) AS t("mid")
WHERE session_id = '?'
);
This can be translated to jOOQ much more easily as:
DSL.using(configuration)
.select()
.from(MESSAGE)
.where(MESSAGE.ID).in(
select(field(name("mid"), MESSAGE.ID.getDataType()))
.from(SESSION_MESSAGES)
.crossJoin(lateral(unnest(SESSION_MESSAGES.MESSAGE_IDS)).as("t", "mid"))
.where(SESSION_MESSAGES.SESSION_ID.eq("'?'"))
)
The Edit3 in the question is quite close to a decent solution for this problem.
We can create a custom generic unnest method for jOOQ which accepts Field and use it in jOOQ query normally.
Helper method:
public static <T> Field<T> unnest(Field<T[]> field) {
var type = (Class<T>) field.getType().getComponentType();
return DSL.function("unnest", type, field);
}
Usage:
public void query(SessionId sessionId) {
var field = unnest(SESSION_MESSAGES.MESSAGE_IDS, UUID.class);
dsl.select().from(MESSAGE).where(
MESSAGE.ID.in(
dsl.select(field).from(SESSION_MESSAGES)
.where(SESSION_MESSAGES.SESSION_ID.eq(sessionId.id))
.orderBy(field)
)
);
}
I am writing a translator which converts DSL to multiple programming language (It seems like Apache Thrift).
For example,
// an example DSL
LOG_TYPE: COMMERCE
COMMON_FIELD : session_id
KEY: buy
FIELD: item_id, transaction_id
KEY: add_to_cart
FIELD: item_id
// will be converted to Java
class Commerce {
private String session_id
private String key;
private String item_id;
private String transaction_id
// auto-created setter, getter, helper methods
...
}
It also should be translated into objective-c and javascript.
To implement it, I have to replace string
// 1. create or load code fragments
String variableDeclarationInJava = "private String {$field};";
String variableDeclarationInJavascript = "...";
String variableDeclarationInObjC = "...";
// 2. replace it
variableDeclarationInJava.replace(pattern, fieldName)
...
Replacing code fragment in String is not type safe and frustrating since it does not any information even if there are errors.
So, my question is It is possible to parse String at compile time? like Scala sqltyped library
If it is possible, I would like to know how can I achieve it.
Thanks.
As far, as I understand, it could be. Please take a look at string interpolation. You implement a custom interpolator, (like it was done for quasi quotations or in Slick).
A nice example of the thing you may want to do is here