How to represent PostgreSQL interval in Go?
My struct looks like this:
type Product struct {
Id int
Name string
Type int
Price float64
Execution_time ????
}
The execution_time field on my database is interval.
The best answer I've come across is to use bigint in your schema, and implement Value & Scan on a wrapper type for time.Duration.
// Duration lets us convert between a bigint in Postgres and time.Duration
// in Go
type Duration time.Duration
// Value converts Duration to a primitive value ready to written to a database.
func (d Duration) Value() (driver.Value, error) {
return driver.Value(int64(d)), nil
}
// Scan reads a Duration value from database driver type.
func (d *Duration) Scan(raw interface{}) error {
switch v := raw.(type) {
case int64:
*d = Duration(v)
case nil:
*d = Duration(0)
default:
return fmt.Errorf("cannot sql.Scan() strfmt.Duration from: %#v", v)
}
return nil
}
Unfortunately, you'll sacrifice the ability to do interval arithmetic inside queries - unless some clever fellow wants to post the type conversion for bigint => interval.
If you're OK with conforming to the time.Duration limits and you need only seconds accuracy you could:
Create the table with a SECOND precision
...
someInterval INTERVAL SECOND(0),
...
Convert INTERVAL into seconds:
SELECT EXTRACT(EPOCH FROM someInterval) FROM someTable;
Use time.Duration::Seconds to insert data to prepared statements
One solution is to wrap the time.Duration type in a wrapper type, and on it provide implementations of sql.Scanner and driver.Valuer.
// PgDuration wraps a time.Duration to provide implementations of
// sql.Scanner and driver.Valuer for reading/writing from/to a DB.
type PgDuration time.Duration
Postgres appears to be quite flexible with the format provided when inserting into an INTERVAL column. The default format returned by calling String() on a duration is accepted, so for the implementation of driver.Value, simply call it:
// Value converts the PgDuration into a string.
func (d PgDuration) Value() (driver.Value, error) {
return time.Duration(d).String(), nil
}
When retrieving an INTERVAL value from Postgres, it returns it in a format that is not so easily parsed by Go (ex. "2 days 05:00:30.000250"), so we need to do some manual parsing in our implementation of sql.Scanner. In my case, I only care about supporting hours, minutes, and seconds, so I implemented it as follows:
// Scan converts the received string in the format hh:mm:ss into a PgDuration.
func (d *PgDuration) Scan(value interface{}) error {
switch v := value.(type) {
case string:
// Convert format of hh:mm:ss into format parseable by time.ParseDuration()
v = strings.Replace(v, ":", "h", 1)
v = strings.Replace(v, ":", "m", 1)
v += "s"
dur, err := time.ParseDuration(v)
if err != nil {
return err
}
*d = PgDuration(dur)
return nil
default:
return fmt.Errorf("cannot sql.Scan() PgDuration from: %#v", v)
}
}
If you need to support other duration units, you can start with a similar approach, and handle the additional units appropriately.
Also, if you happen to be using the GORM library to automatically migrate your table, you will also want to provide an implementation of GORM's migrator.GormDataTypeInterface:
// GormDataType tells GORM to use the INTERVAL data type for a PgDuration column.
func (PgDuration) GormDataType() string {
return "INTERVAL"
}
Related
As the title shows, I have a struct defined an uint64 field, but error returned when I set its value as math.MaxUint64.
This is my code:
type MyDoc struct {
Number uint64 `bson:"_id"`
Timestamp int64 `bson:"time"`
}
// I just want to know whether uint64 overflows or not.
func main() {
mydoc := &MyDoc{
Number: math.MaxUint64,
Timestamp: time.Now().UnixNano(),
}
v, err := bson.Marshal(mydoc)
if err != nil {
panic(err)
}
fmt.Println(v)
}
After executed, error is following:
panic: 18446744073709551615 overflows int64 [recovered]
panic: 18446744073709551615 overflows int64
Obviously, uint64 types of data are processed as int64 which is not I expect.
So, how to store an uint64 data but not overflows in MongoDB?? I can not use string type instead, because I need to compare the size of number so that sorts documents.
I am using MongoDB official Go Driver.
Thanks in advance!
You could use a custom encoder and decoder.
With the code you provided, you could do something like that:
// define a custom type so you can register custom encoder/decoder for it
type MyNumber uint64
type MyDoc struct {
Number MyNumber `bson:"_id"`
Timestamp int64 `bson:"time"`
}
// create a custom registry builder
rb := bsoncodec.NewRegistryBuilder()
// register default codecs and encoders/decoders
var primitiveCodecs bson.PrimitiveCodecs
bsoncodec.DefaultValueEncoders{}.RegisterDefaultEncoders(rb)
bsoncodec.DefaultValueDecoders{}.RegisterDefaultDecoders(rb)
primitiveCodecs.RegisterPrimitiveCodecs(rb)
// register custom encoder/decoder
myNumberType := reflect.TypeOf(MyNumber(0))
rb.RegisterTypeEncoder(
myNumberType,
bsoncodec.ValueEncoderFunc(func(_ bsoncodec.EncodeContext, vw bsonrw.ValueWriter, val reflect.Value) error {
if !val.IsValid() || val.Type() != myNumberType {
return bsoncodec.ValueEncoderError{
Name: "MyNumberEncodeValue",
Types: []reflect.Type{myNumberType},
Received: val,
}
}
// IMPORTANT STEP: cast uint64 to int64 so it can be stored in mongo
vw.WriteInt64(int64(val.Uint()))
return nil
}),
)
rb.RegisterTypeDecoder(
myNumberType,
bsoncodec.ValueDecoderFunc(func(_ bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
// IMPORTANT STEP: read sore value in mongo as int64
read, err := vr.ReadInt64()
if err != nil {
return err
}
// IMPORTANT STEP: cast back to uint64
val.SetUint(uint64(read))
return nil
}),
)
// build the registry
reg := rb.Build()
To use this custom registry, you can pass it as option when creating your collection
collection := client.Database("testing").Collection("myCollection", &options.CollectionOptions{
Registry: reg,
})
This might be a weird question because I don't fully understand how transient and the new derived properties work in Core Data.
So imagine I have a RegularClass entity, which stores any class that repeats over time. A class can repeat for example every third day or every one week. This is how it looks in the data model:
(a RegularClass belongs to a Schedule entity, which in turn belongs to a Course entity)
Now, if our class repeats every third day, we store the number 3 in the frequency property, and a string "days" in unit property, which is then converted to an enum in Swift. A Schedule, which every RegularClass belongs to, has a startDate property.
To check if a class happens at a given date, I came up with nothing better than calculating the difference in specified unit between the startDate and the given date, then taking a remainder between the difference and frequency, and if it's 0, than it's the date in which a class can occur.
var differenceComponent: Calendar.Component {
switch unitType {
case .weeks:
return .weekOfYear
case .days:
return .day
}
}
func getDifferenceFromDateComponents(_ dateComponents: DateComponents) -> Int? {
switch unitType {
case .weeks:
return dateComponents.weekOfYear
case .days:
return dateComponents.day
}
}
func dateIsInActiveState(_ date: Date) -> Bool {
if let startDate = schedule?.startDate {
let comps = Calendar.current.dateComponents([differenceComponent], from: startDate, to: date)
if let difference = getDifferenceFromDateComponents(comps) {
let remainder = Int64(difference) % frequency // that is the key!
return remainder == 0
}
}
return false
}
func containsDate(_ date: Date) -> Bool {
if dateIsInActiveState(date) {
if unitType == .days {
return true
}
let weekday = Calendar.current.component(.weekday, from: date)
return (weekdays?.allObjects as? [Weekday])?.contains(where: { $0.number == weekday }) ?? false
}
return false
}
Now, the thing is that this code works perfectly for courses that I've already got from a fetch request. But is there a way to pass a date parameter in a NSPredicate to calculate this while request happens? Or do I have to fetch all the courses and then filter them out manually?
To solve this issue you could store your data as scalar types and then do simple arithmetic in your predicate. Rather than dates, use integers with a days-from-day-zero figure (or whatever minimum unit of time is necessary for these calculations). Store your repeat cycle as number-of-days.
Then you can use the calculation ((searchDate - startDate) mod repeatCycle) == 0 in your predicate to find matching classes.
As you have suggested, it might be sensible to denormalise your data for different search cases.
I have a struct referencing a *big.Int. When storing this struct naively into MongoDB (using the official driver) the field turns to be nil when fetching the struct back. What is the proper/best way to store a big.Int into MongoDB?
type MyStruct struct {
Number *big.Int
}
nb := MyStruct{Number: big.NewInt(42)}
r, _ := db.Collection("test").InsertOne(context.TODO(), nb)
result := &MyStruct{}
db.Collection("test").FindOne(context.TODO(), bson.D{{"_id", r.InsertedID}}).Decode(result)
fmt.Println(result) // <== Number will be 0 here
My best idea so far would be to create a wrapper around big.Int that implements MarshalBSON and UnmarshalBSON (which I am not even sure how to do properly to be honest). But that'd be quite inconvenient.
Here's a possible implementation I came up with that stores the big.Int as plain text into MongoDb. It is also possible to easily store as byte array by using methods Bytes and SetBytes of big.Int instead of MarshalText/UnmarshalText.
package common
import (
"fmt"
"math/big"
"go.mongodb.org/mongo-driver/bson"
)
type BigInt struct {
i *big.Int
}
func NewBigInt(bigint *big.Int) *BigInt {
return &BigInt{i: bigint}
}
func (bi *BigInt) Int() *big.Int {
return bi.i
}
func (bi *BigInt) MarshalBSON() ([]byte, error) {
txt, err := bi.i.MarshalText()
if err != nil {
return nil, err
}
a, err := bson.Marshal(map[string]string{"i": string(txt)})
return a, err
}
func (bi *BigInt) UnmarshalBSON(data []byte) error {
var d bson.D
err := bson.Unmarshal(data, &d)
if err != nil {
return err
}
if v, ok := d.Map()["i"]; ok {
bi.i = big.NewInt(0)
return bi.i.UnmarshalText([]byte(v.(string)))
}
return fmt.Errorf("key 'i' missing")
}
the field turns to be nil when fetching the struct back
The reason why it returns 0, is because there is no bson mapping available for big.Int. If you check the document inserted into MongoDB collection you should see something similar to below:
{
"_id": ObjectId("..."),
"number": {}
}
Where there is no value stored in number field.
What is the proper/best way to store a big.Int into MongoDB?
Let's first understand what BigInt is. Big integer data type is intended for use when integer values might exceed the range that is supported by the int data type. The range is -2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807) with storage size of 8 bytes. Generally this is used in SQL.
In Go, you can use a more precise integer types. Available built-in types int8, int16, int32, and int64 (and their unsigned counterparts) are best suited for data. The counterparts for big integer in Go is int64. With range -9,223,372,036,854,775,808 through 9,223,372,036,854,775,807, and storage size of 8 bytes.
Using mongo-go-driver,you can just use int64 which will be converted to bson RawValue.Int64. For example:
type MyStruct struct {
Number int64
}
collection := db.Collection("tests")
nb := MyStruct{Number: int64(42)}
r, _ := collection.InsertOne(context.TODO(), nb)
var result MyStruct
collection.FindOne(context.TODO(), bson.D{{"_id", r.InsertedID}}).Decode(&result)
I have a table with timestamp TIMESTAMP, data TEXT columns. I have a failing test because I can't get a timestamp value out of postgresql without time zone annotation. Here's an abridged version of what I've done in my Go application:
type Datapoint struct {
Timestamp string
Data sql.NullString
}
var testData = Datapoint{Timestamp:'2018-12-31 00:00:00', Data:'test'}
db.Exec("CREATE TABLE mytable (id SERIAL, timestamp TIMESTAMP, data TEXT);")
db.Exec("INSERT INTO mytable(timestamp, data) VALUES ($1, $2);", testData.Timestamp, testData.Data)
datapoints, err = db.Exec("SELECT timestamp::TIMESTAMP WITHOUT TIME ZONE, data FROM mytable;")
This trouble is that this query (after about 20 lines of error checking and row.Scan; golang's a bit verbose like that...) gives me:
expected 2018-12-31 00:00:00, received 2018-12-31T00:00:00Z
I requested without timezone (and the query succeeds in psql), so why am I getting the extra T and Z in the string?
Scan into a value of time.Time instead of string, then you can format the time as desired.
package main
import (
"database/sql"
"fmt"
"log"
"time"
)
type Datapoint struct {
Timestamp time.Time
Data sql.NullString
}
func main() {
var db *sql.DB
var dp Datapoint
err := db.QueryRow("SELECT timestamp, data FROM mytable").Scan(
&dp.Timestamp, &dp.Data,
)
switch {
case err == sql.ErrNoRows:
log.Fatal("No rows")
case err != nil:
log.Fatal(err)
default:
fmt.Println(dp.Timestamp.Format("2006-01-02 15:04:05"))
}
}
What you are receiving is an ISO 8601 representation of time.
T is the time designator that precedes the time components of the representation.
Z is used to represent that it is in UTC time, with Z representing zero offset.
In a way you are getting something without a timezone but it can be confusing, especially as you haven't localised your time at any point. I would suggest you consider using ISO times, or you could convert your time to a string like this
s := fmt.Sprintf("%d-%02d-%02d %02d:%02d:%02d\n",
t.Year(), t.Month(), t.Day(),
t.Hour(), t.Minute(), t.Second())
Situation:
I'm using a postgres database and have the following struct:
type Building struct {
ID int `json:"id,omitempty"`
Name string `gorm:"size:255" json:"name,omitempty"`
Lon string `gorm:"size:64" json:"lon,omitempty"`
Lat string `gorm:"size:64" json:"lat,omitempty"`
StartTime time.Time `gorm:"type:time" json:"start_time,omitempty"`
EndTime time.Time `gorm:"type:time" json:"end_time,omitempty"`
}
Problem:
However, when I try to insert this struct into the database, the following error occurs:
parsing time ""10:00:00"" as ""2006-01-02T15:04:05Z07:00"": cannot
parse "0:00"" as "2006""}.
Probably, it doesn't recognize the StartTime and EndTime fields as Time type and uses Timestamp instead. How can I specify that these fields are of the type Time?
Additional information
The following code snippet shows my Building creation:
if err = db.Create(&building).Error; err != nil {
return database.InsertResult{}, err
}
The SQL code of the Building table is as follows:
DROP TABLE IF EXISTS building CASCADE;
CREATE TABLE building(
id SERIAL,
name VARCHAR(255) NOT NULL ,
lon VARCHAR(31) NOT NULL ,
lat VARCHAR(31) NOT NULL ,
start_time TIME NOT NULL ,
end_time TIME NOT NULL ,
PRIMARY KEY (id)
);
While gorm does not support the TIME type directly, you can always create your own type that implements the sql.Scanner and driver.Valuer interfaces to be able to put in and take out time values from the database.
Here's an example implementation which reuses/aliases time.Time, but doesn't use the day, month, year data:
const MyTimeFormat = "15:04:05"
type MyTime time.Time
func NewMyTime(hour, min, sec int) MyTime {
t := time.Date(0, time.January, 1, hour, min, sec, 0, time.UTC)
return MyTime(t)
}
func (t *MyTime) Scan(value interface{}) error {
switch v := value.(type) {
case []byte:
return t.UnmarshalText(string(v))
case string:
return t.UnmarshalText(v)
case time.Time:
*t = MyTime(v)
case nil:
*t = MyTime{}
default:
return fmt.Errorf("cannot sql.Scan() MyTime from: %#v", v)
}
return nil
}
func (t MyTime) Value() (driver.Value, error) {
return driver.Value(time.Time(t).Format(MyTimeFormat)), nil
}
func (t *MyTime) UnmarshalText(value string) error {
dd, err := time.Parse(MyTimeFormat, value)
if err != nil {
return err
}
*t = MyTime(dd)
return nil
}
func (MyTime) GormDataType() string {
return "TIME"
}
You can use it like:
type Building struct {
ID int `json:"id,omitempty"`
Name string `gorm:"size:255" json:"name,omitempty"`
Lon string `gorm:"size:64" json:"lon,omitempty"`
Lat string `gorm:"size:64" json:"lat,omitempty"`
StartTime MyTime `json:"start_time,omitempty"`
EndTime MyTime `json:"end_time,omitempty"`
}
b := Building{
Name: "test",
StartTime: NewMyTime(10, 23, 59),
}
For proper JSON support you'll need to add implementations for json.Marshaler/json.Unmarshaler, which is left as an exercise for the reader ๐
As mentioned in "How to save time in the database in Go when using GORM and Postgresql?"
Currently, there's no support in GORM for any Date/Time types except timestamp with time zone.
So you might need to parse a time as a date:
time.Parse("2006-01-02 3:04PM", "1970-01-01 9:00PM")
I am have come across the same error. It seems like there is a mismatch between type of the column in the database and the Gorm Model
Probably the type of the column in the database is text which you might have set earlier and then changed the column type in gorm model.