I'm using Ent with Pgx.
The column created in Postgres is:
used_at timestamp with time zone NOT NULL,
The value in Postgres is saved without timezone (in UTC):
2022-06-30 22:49:03.970913+00
Using this query:
show timezone
I get:
Etc/UTC
But from Ent (using pgx stdlib) I get the value:
2022-07-01T00:49:03.970913+02:00
Using pgdriver/pq I get the UTC value from DB.
How can I setup pgx to get UTC value?
I tried using this connection string with this code too:
import (
"database/sql"
_ "github.com/jackc/pgx/v4/stdlib"
)
conn, err := sql.Open("pgx", "postgres://postgres:postgres#localhost/project?sslmode=disable&timezone=UTC")
//handle err
The problem is still here.
I need a way to get back from DB the UTC values (that are laready stored in the DB).
timezone is not a valid parameter key word.
You can however use the options key word to specify command-line options to send to the server at connection start. Just keep in mind that you need to percent encode the values therein.
Example of how to set the TimeZone:
package main
import (
"context"
"fmt"
"github.com/jackc/pgx/v4"
)
func main() {
ctx := context.Background()
c1, err := pgx.Connect(ctx, "postgres:///?sslmode=disable")
if err != nil {
panic(err)
}
defer c1.Close(ctx)
// sends "-c TimeZone=UTC" to server at connection start
c2, err := pgx.Connect(ctx, "postgres:///?sslmode=disable&options=-c%20TimeZone%3DUTC")
if err != nil {
panic(err)
}
defer c2.Close(ctx)
var tz1, tz2 string
if err := c1.QueryRow(ctx, "SHOW timezone").Scan(&tz1); err != nil {
panic(err)
}
if err := c2.QueryRow(ctx, "SHOW timezone").Scan(&tz2); err != nil {
panic(err)
}
fmt.Println(tz1)
fmt.Println(tz2)
}
Europe/Prague
UTC
However this only enforces the connection's timezone which does not seem to have an effect on how pgx parses the timestamps themselves once read from the database. In fact it seems it relies, directly or indirectly, on the host machine's local timezone. To confirm that you can update the global time.Local variable to UTC and observe the difference.
// ...
var t1, t2 time.Time
if err := c1.QueryRow(ctx, "select now()::timestamptz").Scan(&t1); err != nil {
panic(err)
}
if err := c2.QueryRow(ctx, "select now()::timestamptz").Scan(&t2); err != nil {
panic(err)
}
fmt.Println(t1)
fmt.Println(t2)
// explicitly set Local to UTC
time.Local = time.UTC
if err := c1.QueryRow(ctx, "select now()::timestamptz").Scan(&t1); err != nil {
panic(err)
}
if err := c2.QueryRow(ctx, "select now()::timestamptz").Scan(&t2); err != nil {
panic(err)
}
fmt.Println(t1)
fmt.Println(t2)
}
Europe/Prague
UTC
2022-06-27 17:18:13.189557 +0200 CEST
2022-06-27 17:18:13.190047 +0200 CEST
2022-06-27 15:18:13.190401 +0000 UTC
2022-06-27 15:18:13.190443 +0000 UTC
For obvious reasons I'd avoid doing the above. If pgx doesn't provide a way to configure the default location it uses to parse the timestamps then the next best option, that I can think of, would be to use a custom time.Time type.
// ...
var t1, t2 time.Time
if err := c1.QueryRow(ctx, "select now()::timestamptz").Scan(&t1); err != nil {
panic(err)
}
if err := c2.QueryRow(ctx, "select now()::timestamptz").Scan(&t2); err != nil {
panic(err)
}
fmt.Println(t1)
fmt.Println(t2)
var tt1, tt2 myTime
if err := c1.QueryRow(ctx, "select now()::timestamptz").Scan(&tt1); err != nil {
panic(err)
}
if err := c2.QueryRow(ctx, "select now()::timestamptz").Scan(&tt2); err != nil {
panic(err)
}
fmt.Println(tt1)
fmt.Println(tt2)
}
type myTime struct {
time.Time
}
func (tt *myTime) Scan(src interface{}) error {
if t, ok := src.(time.Time); ok {
tt.Time = t.In(time.UTC)
return nil
}
return fmt.Errorf("myTime: unsupported type %T", src)
}
Europe/Prague
UTC
2022-06-27 17:26:45.94049 +0200 CEST
2022-06-27 17:26:45.940959 +0200 CEST
2022-06-27 15:26:45.941321 +0000 UTC
2022-06-27 15:26:45.941371 +0000 UTC
Related
I have mongo capped collection and a simple API, written on Go. I built and run it. When I try to sent Get request or simply go localhost:8000/logger in browser - my process closes. Debug shows this happens, while executing "find" in collection. It produces error "client is disconnected". Collection has 1 document, and debug shows it is connected with my helper.
Go version 1.13
My code:
func main() {
r := mux.NewRouter()
r.HandleFunc("/logger", getDocs).Methods("GET")
r.HandleFunc("/logger", createDoc).Methods("POST")
log.Fatal(http.ListenAndServe("localhost:8000", r))
}
func getDocs(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var docs []models.Logger
//Connection mongoDB with helper class
collection := helper.ConnectDB()
cur, err := collection.Find(context.TODO(), bson.M{})
if err != nil {
helper.GetError(err, w)
return
}
defer cur.Close(context.TODO())
for cur.Next(context.TODO()) {
var doc models.Logger
err := cur.Decode(&doc)
if err != nil {
log.Fatal(err)
}
docs = append(docs, doc)
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
json.NewEncoder(w).Encode(docs)
}
func ConnectDB() *mongo.Collection {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to MongoDB!")
logCollection := client.Database("local").Collection("loggerCollection")
return logCollection
}
According to the documentation, the call to mongo.NewClient doesn't ensure that you can connect the Mongo server. You should first call mongo.Client.Ping() to verify if you can connect to the database or not.
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
if err := client.Ping(context.TODO(), readpref.Primary()); err != nil {
// Can't connect to Mongo server
log.Fatal(err)
}
There could be several reasons behind failing to connect, the most obvious one is incorrect setup of ports. Is your mongodb server up and listening on port 27017? Is there any change you're running mongodb with Docker and it's not forwarding to the correct port?
I faced similar issue , read #Jay answer it definitely helped , as I checked my MongoDB was running using "MongoDB Compass" , then I changed the location of my insert statement , previously I was calling before the call of "context.WithTimeout". Below is working code.
package main
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type Book struct {
Name string `json:"name,omitempty"`
PublisherID string `json:"publisherid,omitempty"`
Cost string `json:"cost,omitempty"`
StartTime string `json:"starttime,omitempty"`
EndTime string `json:"endtime,omitempty"`
}
func main() {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
testCollection := client.Database("BooksCollection").Collection("BooksRead")
inserRes, err := testCollection.InsertOne(context.TODO(), Book{Name: "Harry Potter", PublisherID: "IBN123", Cost: "1232", StartTime: "2013-10-01T01:11:18.965Z", EndTime: "2013-10-01T01:11:18.965Z"})
log.Println("InsertResponse : ", inserRes)
log.Println("Error : ", err)
}
I can see document inserted in console as well as in "MongoDB Comapass."
In heiper function "ConnectDB" after "NewClient" I must use "client.Connect(context.TODO())"
before any other use of client
I am working on a go project where I need to serve files stored in mongodb. The files are stored in a GridFs. I use gopkg.in/mgo.v2 as package to connect and query the db.
I can retrieve the file from the db, that is not hard.
f, err := s.files.OpenId(id)
But how can I serve that file with http?
I work with the JulienSchmidt router to handle all the other restfull requests.
The solutions I find always use static files, not files from a db.
Thanks in advance
Tip: Recommended to use github.com/globalsign/mgo instead of gopkg.in/mgo.v2 (the latter is not maintained anymore).
The mgo.GridFile type implements io.Reader, so you could use io.Copy() to copy its content into the http.ResponseWriter.
But since mgo.GridFile also implements io.Seeker, you may take advantage of http.ServeContent(). Quoting its doc:
The main benefit of ServeContent over io.Copy is that it handles Range requests properly, sets the MIME type, and handles If-Match, If-Unmodified-Since, If-None-Match, If-Modified-Since, and If-Range requests.
Example handler serving a file:
func serveFromDB(w http.ResponseWriter, r *http.Request) {
var gridfs *mgo.GridFS // Obtain GridFS via Database.GridFS(prefix)
name := "somefile.pdf"
f, err := gridfs.Open(name)
if err != nil {
log.Printf("Failed to open %s: %v", name, err)
http.Error(w, "something went wrong", http.StatusInternalServerError)
return
}
defer f.Close()
http.ServeContent(w, r, name, time.Now(), f) // Use proper last mod time
}
its old but i got another solution with goMongo driver by importing
"go.mongodb.org/mongo-driver/mongo/gridfs"
var bucket *gridfs.Bucket //creates a bucket
dbConnection, err := db.GetDBCollection() //connect db with your your
if err != nil {
log.Fatal(err)
}
bucket, err = gridfs.NewBucket(dbConnection)
if err != nil {
log.Fatal(err)
}
name := "br100_update.txt"
downloadStream, err := bucket.OpenDownloadStreamByName(name)
if err != nil {
log.Printf("Failed to open %s: %v", name, err)
http.Error(w, "something went wrong", http.StatusInternalServerError)
return
}
defer func() {
if err := downloadStream.Close(); err != nil {
log.Fatal(err)
}
}()
// Use SetReadDeadline to force a timeout if the download does not succeed in
// 2 seconds.
if err = downloadStream.SetReadDeadline(time.Now().Add(2 * time.Second)); err
!= nil {
log.Fatal(err)
}
// this code below use to read the file
fileBuffer := bytes.NewBuffer(nil)
if _, err := io.Copy(fileBuffer, downloadStream); err != nil {
log.Fatal(err)
}
I am trying to connect to two remote mongodb servers using ssh port-forwarding in golang which are used by my graphql server for querying. The intermediary host for the two tunnels is same. So let's say the intermediary host is 123.45.678.678 and the two remote mongodb servers are 1.23.45.67 and 1.23.45.78, I create the tunnels like this,
conn, err := ssh.Dial("tcp", 123.45.678.678, config)
if err != nil {
panic(err)
}
remote1, err := conn.Dial("tcp", "1.23.45.67:27017")
if err != nil {
panic(err)
}
remote2, err := conn.Dial("tcp", "1.23.45.78:27017")
if err != nil {
panic(err)
}
local1, err := net.Listen("tcp", "localhost:27018")
if err != nil {
panic(err)
}
local2, err := net.Listen("tcp", "localhost:27019")
if err != nil {
panic(err)
}
Now i forward traffic from local1 to remote1 and local2 to remote2 like this
go func() {
for {
l, err := local1.Accept()
if err != nil {
panic(err)
}
go func() {
_, err := io.Copy(l, remote1)
if err != nil {
panic(err)
}
}()
go func() {
_, err := io.Copy(remote1, l)
if err != nil {
panic(err)
}
}()
}
}()
go func() {
for {
l, err := local2.Accept()
if err != nil {
panic(err)
}
go func() {
_, err := io.Copy(l, remote2)
if err != nil {
panic(err)
}
}()
go func() {
_, err := io.Copy(remote2, l)
if err != nil {
panic(err)
}
}()
}
}()
And I create two mongo sessions using mgo.Dial and export these two sessions to the graphql whenever this function is called. For some queries ( not all queries, only some complex queries ) which need both the sessions, i see the write: broken pipe error
panic: readfrom tcp 127.0.0.1:27019->127.0.0.1:53128: write tcp 127.0.0.1:27019->127.0.0.1:53128: write: broken pipe
When I debugged this, i figured out that this error occurs when the io.copy happens between l and remote2 in the code snippet above which i guess is due to the disconnection of remote2 tunnel.
The tcpdump showed that the intermediary host is sending the finish flag to the remote2 server after sometime which inturn is leading to the termination of the connection. I am wondering how I can resolve this.
I am doing attempting to build a basic API using Go which returns the results of a SQL query using the PostgreSQL library.
At the moment I can make the program return the values, but I can't get it to return a failed message to the user i.e. some JSON with an error message.
I have an error function as follows :
func handleError(w http.ResponseWriter, err error) {
if err != nil {
log.Print(err.Error() + "\r\n") // Logging
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
However the http.Error method doesn't appear to ever return anything. The error thrown is a table that doesn't exist in the database (which gets logged to a text file: i.e. 2016/01/11 23:28:19 pq: relation "building_roof" does not exist
My programmes query code looks like this:
table := pq.QuoteIdentifier(table)
identifier := pq.QuoteIdentifier("ID")
rows, err := db.Query( fmt.Sprintf("SELECT %s, ST_AsText(geom) FROM %s WHERE %s = $1", identifier, table, identifier), feature)
handleError(w, err)
Causing an error just gives a Chrome error:
No data received
ERR_EMPTY_RESPONSE
EDIT Full Code:
package main
import (
"fmt"
"encoding/json"
"os"
"log"
"net/http"
"database/sql"
"strings"
"time"
"github.com/lib/pq"
)
func handler(w http.ResponseWriter, r *http.Request) {
f, err := os.OpenFile("pgdump_errorlog.txt", os.O_RDWR | os.O_CREATE | os.O_APPEND, 0666)
log.Print("Couldn't open file")
defer f.Close()
log.SetOutput(f)
// Timing
start := time.Now()
// Postgres Credentials
const (
DB_USER = "postgres"
DB_PASSWORD = "OMITTED" // Removed details !
DB_PORT = "OMITTED"
DB_NAME = "OMITTED"
)
// Postgres Connect
dbinfo := fmt.Sprintf("user=%s password=%s dbname=%s port=%s sslmode=disable",
DB_USER, DB_PASSWORD, DB_NAME, DB_PORT)
db, err := sql.Open("postgres", dbinfo)
handleError(w, err)
defer db.Close()
table := r.FormValue("table")
feature := r.FormValue("id")
if table != "" {
//Postgres Query
var (
id int
geom string
)
table := pq.QuoteIdentifier(table)
identifier := pq.QuoteIdentifier("ID")
rows, qerr := db.Query( fmt.Sprintf("SELECT %s, ST_AsText(geom) FROM %s WHERE %s = $1", identifier, table, identifier), feature)
handleError(w, err)
defer rows.Close()
for rows.Next() {
err := rows.Scan(&id, &geom)
handleError(w, err)
}
err = rows.Err()
handleError(w, err)
// Maniplate Strings
returngeom := strings.Replace(geom, "1.#QNAN", "", -1)
i := strings.Index(returngeom, "(")
wkt := strings.TrimSpace(returngeom[:i])
returngeom = returngeom[i:]
type WTKJSON struct {
WTKType string
Geometry string
Elapsed time.Duration
}
returnjson := WTKJSON{Geometry: returngeom, WTKType: wkt , Elapsed: time.Since(start)/1000000.0}
json.NewEncoder(w).Encode(returnjson)
}
}
func handleError(w http.ResponseWriter, err error) {
if err != nil {
log.Print(err.Error() + "\r\n") // Logging
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
The following appeared to allow me to return JSON Errors:
func handleError(w http.ResponseWriter, err string) {
type APIError struct {
Error string
}
re, _ := json.Marshal(APIError{Error: err})
io.WriteString(w, string(re))
}
Used like so:
rows, err := db.Query( fmt.Sprintf("SELECT %s, ST_AsText(geom) FROM %s WHERE %s = $1", identifier, table, identifier), feature)
if err != nil {
handleError(w, err.Error())
return
}
Not suggesting this is the best method, but worked in my case.
When you are using the http.Error function the error message should a string. For simple testing I would suggest take this line
http.Error(w, err.Error(), http.StatusInternalServerError)
and change it to something like
http.Error(w,"there was an error", http.StatusInternalServerError)
and see if that response comes through. If it does its likely that you are trying to pass something that isn't a string in http.Error()
This question already has answers here:
How to use global var across files in a package?
(3 answers)
Closed 3 years ago.
I keep getting this error when I run my Go code which makes queries to my local postgres database.
Error:
panic serving [::1]:56708: runtime error: invalid memory address or nil pointer dereference
goroutine 23 [running]:
net/http.funcĀ·011()
/usr/local/go/src/pkg/net/http/server.go:1100 +0xb7
runtime.panic(0x2ef0a0, 0x4d8ee4)
/usr/local/go/src/pkg/runtime/panic.c:248 +0x18d
database/sql.(*DB).conn(0x0, 0x277a1, 0x0, 0x0)
/usr/local/go/src/pkg/database/sql/sql.go:625 +0x751
database/sql.(*DB).Ping(0x0, 0x0, 0x0)
/usr/local/go/src/pkg/database/sql/sql.go:452 +0x39
main.firstHandler(0x58e9a8, 0xc208052320, 0xc2080284e0)
/Users/Tommy/Documents/gocode/server/server.go:122 +0x35
net/http.HandlerFunc.ServeHTTP(0x3c6be8, 0x58e9a8, 0xc208052320, 0xc2080284e0)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/gorilla/mux.(*Router).ServeHTTP(0xc2080186e0, 0x58e9a8, 0xc208052320, 0xc2080284e0)
/Users/Audrey/gocode/src/github.com/gorilla/mux/mux.go:98 +0x292
net/http.(*ServeMux).ServeHTTP(0xc208022660, 0x58e9a8, 0xc208052320, 0xc2080284e0)
/usr/local/go/src/pkg/net/http/server.go:1511 +0x1a3
net/http.serverHandler.ServeHTTP(0xc208004660, 0x58e9a8, 0xc208052320, 0xc2080284e0)
/usr/local/go/src/pkg/net/http/server.go:1673 +0x19f
net/http.(*conn).serve(0xc208050500)
/usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e
created by net/http.(*Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1721 +0x313
Go:
func firstHandler(w http.ResponseWriter, r *http.Request) {
err := db.Ping()
if err != nil {
log.Fatal(err)
}
rows, err := db.Query("SELECT id, created_at, updated_at FROM script WHERE updated_at = $1", 3)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
var created_at, updated_at, id int
for rows.Next() {
err := rows.Scan(&id, &created_at, &updated_at)
if err != nil {
log.Fatal(err)
}
fmt.Fprintf("%s %s %s", id, created_at, updated_at)
}
}
var r = mux.NewRouter()
var db *sql.DB
func main() {
db, err := sql.Open("postgres", "user=Tommy host=localhost dbname=dbgo sslmode=verify-full")
if err != nil {
log.Fatal(err)
}
defer db.Close()
r.HandleFunc("/ping", firstHandler)
http.Handle("/", r)
http.ListenAndServe(":8080", nil)
}
Help. What am I doing wrong? I referred to this also: https://gophercasts.io/lessons/4-postgres-basics.
Actually, you declare the connection with:
var db *sql.DB
but you open the connection with:
db, err := sql.Open("postgres", "user=Tommy host=localhost dbname=dbgo sslmode=verify-full")
Note the := (it combines a variable declaration with an assignment). This will actually shadow the global db variable by a local one. The connection is opened but assigned to the local variable. So the value of the global db variable is nil.
When the firstHandler function is called, its value is still nil, which triggers the panic.
Replace the := by a = (and declare the err object before).