I have a mongo database that contains a list of DBRef objects. I am converting this collection into a data frame in spark.
This gives the below error. Does anyone know how to resolve this error ??
EDIT:: Conversion to dataframe gives following error -> org.codehaus.janino.InternalCompilerException: Two non-abstract methods "public int scala.collection.TraversableOnce.size()" have the same parameter types, declaring type and return type
The error got resolved after upgrading the version of "org.codehaus.janino" to 3.0.9 (previously, I was using 3.0.8).
Related
Im hoping to use jooq dsl to do batch inserts to postgres. I know it's possible but Im having issues getting the data formatted properly.
dslContext.loadInto(table).loadJSON(json-data).fields(...).execute();
is where Im starting from. The tricky part seems to be getting Map<String, String> into a jsonb column.
I have the data formatted according to this description and jooq seems to be ok with it.. until the map/json-in-json shows up.
Another json-array column still needs to be dealt with too.
Questions:
is this a reasonable approach?
if not - what would you recommend instead?
Error(s) Im seeing:
ERROR: column "changes_to" is of type jsonb but expression is of type character varying
Hint: You will need to rewrite or cast the expression.
Edit:
try (DSLContext context = DSL.using(pgClient.getDataSource(), SQLDialect.POSTGRES_10)) {
context.loadInto(table(RECORD_TABLE))
.loadJSON(jsonData)
.fields(field(name(RECORD_ID_COLUMN)),
field(name(OTHER_ID_COLUMN)),
field(name(CHANGES_TO_COLUMN)),
field(name(TYPE_COLUMN)),
IDS_FIELD)
.execute();
} catch (IOException e) {
throw new RuntimeException(e);
}
with json data:
{"fields":[{"name":"rec_id","type":"VARCHAR"},{"name":"other_id","type":"VARCHAR"},{"name":"changes_to","type":"jsonb"},{"name":"en_type","type":"VARCHAR"},{"name":"ids","type":"BIGINT[]"}],"records":[["recid","crmid","{\"key0\":\"val0\"}","ent type",[10,11,12]],["recid2","crmid2","{\"key0\":\"val0\"}","ent type2",[10,11,12]]]}
The problem(s) being how to format the 'changes_to' and 'ids' columns.
There's a certain price to pay if you're not using jOOQ's code generator (and you should!) jOOQ doesn't know what of data type your columns are if you create a field(name("...")), so it won't be able to bind your values correctly. Granted, the Loader API could read the JSON header information, but it currently doesn't.
Instead, why not just either:
Provide explicit type information to your column references, like field(name(CHANGES_TO_COLUMN), SQLDataType.JSONB)
Much better: use the code generator, in case of which you already have all the type information associated with your Field expression.
I'm using a locally hosted postgres DB to test queries to a postgres DB in production. The production database has an info field of type jsonb; and I'm trying to mimic this schema locally when using gorm's AutoMigrate. The model I've defined is below:
import "github.com/jinzhu/gorm/dialects/postgres"
type Event struct {
...
Info postgres.Jsonb
...
}
But when I query JSON attributes, e.g. stmt.Where("info->>'attr' = value"), I get the following error:
...
Message:"operator does not exist: bytea ->> unknown", Detail:"", Hint:"No operator matches the given name and argument type(s). You might need to add explicit type casts.",
...
This query works however in the production environment. It seems that the Info field is being stored as bytea instead of jsonb. I'm aware that I can do stmt.Where("encode(info, "escape")::jsonb->>'attr' = value"), but I'd prefer to mimic the production environment more closely (if possible) than change the query to support these unit tests.
I've tried using type tags in the model (e.g. gorm:"type=jsonb") as well as defining my own JSON type implmementing the valuer, scanner, and GormDataTypeInterface as suggested here. None of these approaches have automigrated the type as jsonb.
Is there any way to ensure AutoMigrate creates a table with type jsonb? Thanks!
I was facing the same problem, type JsonB is automigrated to bytea. I solved it by adding the tag gorm:"type:jsonb". It's also mentioned in your question, but you're using gorm:"type=jsonb", which is not correct.
I have a program where I need to access Mongo Sharred Key from DB collection in Go lang. I retrieve the collection output as
map[string]interface{}
and type cast Mongo Sharred Key. However sometime its definition is int64 and sometime it is only int. So my Go program is getting panic sating
"invalid interface conversion - interface is defined as int but its type is int64".
And when I change its definition to int64 again I am getting same type error for other records. Please help.
I'm using OrientDB and trying to create new property after I inserted my data (millions of rows).
I'm trying to create property on V in order to create an index and I'm getting the following error:
The database contains some schema-less data in the property
'V.ACCOUNT_NO' that is not compatible with the type STRING. Fix those
records and change the schema again [ONetworkProtocolHttpDb]
Now part of the fields type is INTEGER but it seems to me that it's very easy to convert the type to STRING.
how can I do it to the entire data?
I tried your case by creating this simple structure in schema-less mode:
These records are a mix of INTEGER and STRING types:
Now you can convert the not string records type by using this query:
UPDATE V SET ACCOUNT_NO = ACCOUNT_NO.asString() WHERE ACCOUNT_NO.type() <> 'STRING'
Output:
About the exception, I got correctly the same error when I try to create a new property V.ACCOUNT_NO of type STRING in schema-full mode and this is correct because the property already exists in the database and contains mixed types of records, although in schema-less mode.
Once all the records were converted, you'll able to create the new property.
Hope it helps
I am using the Grails MongoDb plugin and trying to get the Date conversion.
In my config I have defined different dateFormats
grails.databinding.dateFormats = [
'yyyy-MM-dd',
'yyyy-MM-dd HH:mm:ss.S',
"yyyy-MM-dd'T'hh:mm:ss'Z'"
]
However when I read a record from mongodb into my Customer domain class, I still get this error: What am I doing wrong?
org.springframework.beans.TypeMismatchException: Failed to convert
property value of type 'java.lang.String' to required type
'java.util.Date' for property 'lastUpdateDate'; nested exception is
org.springframework.core.convert.ConversionFailedException: Failed to
convert from type java.lang.String to type
#org.grails.databinding.BindingFormat java.util.Date for value
'1994-01-29T00:00:00Z'; nested exception is
java.lang.IllegalArgumentException: java.lang.NumberFormatException:
For input string: "1994-01-29T00:00:00Z"
It looks like you have manually inserted the value of lastUpdateDate in your MongoDB database which got stored in String format instead of MongoDB's ISODate() format. Delete or modify that record in database and this will work fine.