pq: date/time field value out of range: "22/02/2022" - postgresql

I have this query:
l.loyaltycard_number, l.recipt_no,
l.totaltrans_amount, l.amount_paid,
l.reward_points, l.redeemed_points,
cashier FROM loyalty l
JOIN warehouses w
ON l.machine_ip = w.machine_ip
WHERE l.machine_name = $1
AND redeemed_points != $2
AND trasaction_time BETWEEN $3 AND $4
ORDER BY trasaction_time DESC;
I have HTML datepickers for the transaction_time that is in the format dd/mm/yyyy.
anytime I select a date range that the first number is greater than 12, (22/02/2022).
I get the above error.
I suspected the formatting was the problem.
I found in the docs how to set the postgresql date style to DMY. After doing that, I get the same error.
However, when I run the same query in Postgres cli like so:
SELECT w.machine_name, l.trasaction_time,
l.loyaltycard_number, l.recipt_no,
l.totaltrans_amount, l.amount_paid,
l.reward_points, l.redeemed_points,
cashier FROM loyalty l
JOIN warehouses w
ON l.machine_ip = w.machine_ip
WHERE l.machine_name = 'HERMSERVER'
AND redeemed_points != 0
AND trasaction_time BETWEEN '14/11/21' AND '22/02/22'
ORDER BY trasaction_time DESC;
I get the expected result. I don't know what I am doing wrong.
I want to know how I can make the database treat the date from the datepicker as dd/mm/yyyy instead of mm/dd/yyyy. I am using google cloudsql Postgres
This is the code for the handler that gets the data from the datepicker
err := r.ParseForm()
if err != nil {
app.clientError(w, http.StatusBadRequest)
}
startDate := r.PostForm.Get("startDate")
endDate := r.PostForm.Get("endDate")
outlet := r.PostForm.Get("outlet")
reportType := r.PostForm.Get("repoType")
if reportType == "0" {
rReport, err := app.models.Reports.GetRedeemedReport(startDate, endDate, outlet, reportType)
if err != nil {
app.serverError(w, err)
return
}
app.render(w, r, "tranxreport.page.tmpl", &templateData{
Reports: rReport,
})
} else if reportType == "1" {
rReport, err := app.models.Reports.GetAllReport(startDate, endDate, outlet)
if err != nil {
app.serverError(w, err)
return
}
app.render(w, r, "tranxreport.page.tmpl", &templateData{
Reports: rReport,
})
} else {
app.render(w, r, "tranxreport.page.tmpl", &templateData{})
}

As per the comments while it should be possible to change DateStyle there are a few issues with this:
The SET datestyle command changes the style for the current session. As the SQL package uses connection pooling this is of limited use.
You may be able to use "the DateStyle parameter in the postgresql.conf configuration file, or the PGDATESTYLE environment variable on the server" but this may not be available where Postgres is offered as a managed service. Note that making this change also means your software will fail if the parameter is not set (and this is easily done when moving to a new server).
A relatively simple solution is to edit your query to use TO_DATE e.g.:
BETWEEN TO_DATE($3,'DD/MM/YYYY') AND TO_DATE($4,'DD/MM/YYYY')
However while this will work it makes your database code dependent upon the format of the data sent into your API. This means that the introduction of a new date picker, for example, could break your code in a way that is easily missed (testing at the start of the month works either way).
A better solution may be to use a standard format for the date in your API (e.g. ISO 8601) and/or pass the dates to your database functions as a time.Time. However this does require care due to time zones, daylight saving etc.

Related

Golang slow scan() for multiple rows

I am running a query in Golang where I select multiple rows from my Postgresql Database.
I am using the following imports for my query
"database/sql"
"github.com/lib/pq"
I have narrowed down to my loop for scanning the results into my struct.
// Returns about 400 rows
rows, err = db.Query('SELECT * FROM infrastructure')
if err != nil {
return nil, err
}
var arrOfInfra []model.Infrastructure
for rows.Next() {
obj, ptrs := model.InfrastructureInit()
rows.Scan(ptrs...)
arrOfInfra = append(arrOfInfra, *obj)
}
rows.Close()
The above code takes about 8 seconds to run, and while the query is fast, the loop in rows.Next() takes the entire 8 seconds over to complete.
Any ideas? Am I doing something wrong, or is there a better way?
My configuration for my database
// host, port, dbname, user, password masked for obvious reasons
db, err := sql.Open("postgres", "host=... port=... dbname=... user=... password=... sslmode=require")
if err != nil {
panic(err)
}
// I have tried using the default, or setting to high number (100), but it doesn't seem to help with my situation
db.SetMaxIdleConns(1)
db.SetMaxOpenConns(1)
UPDATE 1:
I placed print statements in the for loop. Below is my updated snippet
for rows.Next() {
obj, ptrs := model.InfrastructureInit()
rows.Scan(ptrs...)
arrOfInfra = append(arrOfInfra, *obj)
fmt.Println("Len: " + fmt.Sprint(len(arrOfInfra)))
fmt.Println(obj)
}
I noticed that in this loop, it will actually pause half-way, and continue after a short break. It looks like this:
Len: 221
Len: 222
Len: 223
Len: 224
<a short pause about 1 second, then prints Len: 225 and continues>
Len: 226
Len: 227
...
..
.
and it will happen again later on at another row count, and again after a few hundred records.
UPDATE 2:
Below is a snippet of my InfrastructureInit() method
func InfrastructureInit() (*Infrastructure, []interface{}) {
irf := new(Infrastructure)
var ptrs []interface{}
ptrs = append(ptrs,
&irf.Base.ID,
&irf.Base.CreatedAt,
&irf.Base.UpdatedAt,
&irf.ListingID,
&irf.AddressID,
&irf.Type,
&irf.Name,
&irf.Description,
&irf.Details,
&irf.TravellingFor,
)
return irf, ptrs
}
I am not exactly sure what is causing this slowness, but I currently placed a quick patch on my server to using a redis database and precache my infrastructures, saving it as a string. It seems to be okay for now, but I now have to maintain both redis and my postgres.
I am still puzzled over this weird behavior, but I'm not exactly how rows.Next() work - does it make a query to the database everytime I call rows.Next()?
How do you think about just do like this?
defer rows.Close()
var arrOfInfra []*Infrastructure
for rows.Next() {
irf := &Infrastructure{}
err = rows.Scan(
&irf.Base.ID,
&irf.Base.CreatedAt,
&irf.Base.UpdatedAt,
&irf.ListingID,
&irf.AddressID,
&irf.Type,
&irf.Name,
&irf.Description,
&irf.Details,
&irf.TravellingFor,
)
if err == nil {
arrOfInfra = append(arrOfInfra, irf)
}
}
Hope this help.
I went some weird path myself while consolidating my understanding of how rows.Next() work and what might be impacting performance so thought about sharing this here for posterity (despite the question asked a long time ago).
Related to:
I am still puzzled over this weird behavior, but I'm not exactly how
rows.Next() work - does it make a query to the database everytime I
call rows.Next()?
It doesn't make a 'query' but it reads (transfers) data from the db through a driver on each iteration which means it can be impacted by e.g. bad network performance. Especially true if, for example, your db is not local to where you are running your Go code.
One approach to confirm whether network performance is an issue would be to run your go app on the same machine where your db is (if possible).
Assuming columns that are scanned in the above code are not of extremely large size or having custom conversions - reading ~400 rows should take in the order of 100ms at most (in a local setup).
For example - I had a case where I needed to read about 100k rows with about 300B per row and that was taking ~4s (local setup).

How do I allow my model to accept two different date formats?

I'm using Ruby on Rails 5. I have the following code in my model ...
class User < ActiveRecord::Base
...
attr_accessor :dob_string
def dob_string
#dob_string || (self.dob ? self.dob.strftime('%m/%d/%Y') : "")
end
def dob_string=(dob_s)
date = dob_s && !dob_s.empty? ? Date.strptime(dob_s, '%m/%d/%Y') : nil
self.dob = date
rescue ArgumentError
errors.add(:dob, 'The birth date is not in the correct format (MM/DD/YYYY)')
#dob_string = dob_s
end
Despite my instructions, sometimes users enter the date in the form MM-DD-YYYY. So what I want to do is allow my model to accept both formats, MM/DD/YYYY or MM-DD-YYYY, but not mixed, that is MM/DD-YYYY shoud still be illegal. How do I modify my model so that it can accept either format?
I think this might do what you need:
class User < ActiveRecord::Base
def dob_string
#dob_string || (dob ? dob.strftime("%m/%d/%Y") : "")
end
def dob_string=(dob_s)
unless convert_preferred_date(dob_s) || convert_acceptable_date(dob_s)
errors.add(:dob, 'The birth date is not in the correct format (MM/DD/YYYY)')
#dob_string = dob_s
end
end
private
def convert_preferred_date(dob_s)
convert_date(dob_s, "%m/%d/%Y")
end
def convert_acceptable_date(dob_s)
convert_date(dob_s, "%m-%d-%Y")
end
def convert_date(dob_s, format)
self.dob = (dob_s && !dob_s.empty? ? Date.strptime(dob_s, format) : nil)
rescue ArgumentError
nil
end
end
It's a little inelegant, but I can't think of a way to do only two date formats. If you're prepared to accept any human-readable data format, there's a much simpler way to do this.
Please test this thoroughly in your own app. I was just messing around with this in the Rails console, and I might not have tested every case. Hopefully this helps.

Utilities.formatDate() in Google Apps Script outputs previous date (e.g. input 25.05.2015 -> output 24.05.2015)

I have a problem with Google Docs' Utilities.formatDate() function.
I have a spreadsheet that contains all of the orders we place in the lab. When an order is delivered our lab manager enters the delivery date in the relevant cell in such a spreadsheet, in the following format: dd.MM.yyyy.
I created a script that, provided certain conditions, will email whoever placed that order alerting them that the order has been delivered on that particular date. Here is the code:
function DeliveryAlerts() {
try {
var email_dict = {"Y":"Y#Z.com"}
var spreadsheet = SpreadsheetApp.openById("ABC");
SpreadsheetApp.setActiveSpreadsheet(spreadsheet);
var sheet = spreadsheet.getSheetByName("Orders");
var values = sheet.getRange("A2:Q251").getValues();
var bgcolours = sheet.getRange("A2:Q251").getBackgrounds();
for(var i=0;i<=249;i++)
{
var j = i + 2;
if (values[i][16]=="Yes" && values[i][11]!="" && bgcolours[i][16]!="#b8b8b8")
{
var email_address = email_dict[values[i][13]];
var cur_date = Utilities.formatDate(values[i][11], "GMT+1", "EEE dd.MM.yyyy");
var message = "Hello there,\n\nYour order of " + values[i][4] + " has been delivered on "+ cur_date +".\n\nBest wishes";
var subject = "Delivery alert";
MailApp.sendEmail(email_address, subject, message,{replyTo:"abc#abc.com", name:"ABC"});
sheet.getRange("Q"+j).setBackground("#b8b8b8");
}
}
} catch (err) {
MailApp.sendEmail("abc#abc.com", "Delivery Alerts Script in Order Master List", err);
}
}
I use
Utilities.formatDate(values[i][11], "GMT+1", "EEE dd.MM.yyyy") to reformat the date from, say, 25.05.2015 (that is, the value in the cell) to Mon 25.05.2015. However, what I get instead is Sun 24.05.2015.
Does anybody know what is going on?
Thank you in advance.
Nicola
Check the time zone setting in the script editor. Under the FILE menu, choose PROJECT PROPERTIES in the script editor. It's possible to have a different time zone setting in Apps Script, than is in the spreadsheet. This is a common issue that arises. Apps Script allows a separate time zone setting from the spreadsheet. Also, even if the time is only off by one minute, if the time setting of the date is all zeros, it's common to get the problem that you are having. When a user enters a date, it's possible that no time setting is made. So the time is set to all zeros. The date is correct, but the time is all zeros. Even if the date was typed in at 3 in the afternoon, for example, and the date is correct, the time setting can be midnight of that day. So, even if you subtracted one second from that date, it would now be the day before.

breezejs: date is not set to the right time

I've noticed that if a date property comes back from the server with the value "2013-07-11T17:11:04.700", then breeze changes the value to Thu Jul 11 19:11:04 UTC+0200 2013.
Notice the time is now 2 hours ahead !
I had already come across this issue when saving entities, so I had to explicitly convert my date properties using momentjs :
date.hours(date.hours() - moment().zone() / 60);
But now it seems the problem occurs also when doing read operations.
What's the best way to make sure breeze does not alter values of my date properties ?
Breeze does not manipulate the datetimes going to and from the server in any way EXCEPT to add a UTZ timezone specifier to any dates returned from the server that do not already have one. This is only done because different browsers interpret dates without a timezone specifier differently and we want consistency between browsers.
The source of your issues is likely to be that when you save your data with dates to the database, that the dateTime datatype you are using does NOT contain a timezone offset. This means that when the data is retrieved you are likely "losing" the offset and the Breeze default mentioned above kicks in. This can be corrected by using a database date time datatype with an timezone offset ( datetime2 or datetimeoffset in SQLServer).
Note that your browser DOES format dates according to it's current timezone.
Another approach is that you can replace Breeze's DataType.parseDateFromServer to NOT infer any time zone info if it is not provided:
breeze.DataType.parseDateFromServer = function (source) {
return new Date(Date.parse(source));
};
However, this can run into the problem that different browsers interpret DateTime strings without a time zone offset differently... So you may still get strange results depending on the browser. If that happens you will need to add some browser detection code to the snippet above.
Another alternative is to do the following using the moment.js library.
breeze.DataType.parseDateFromServer = function (source) {
var date = moment(source);
return date.toDate();
};
Not sure how helpful this is, but hopefully Breeze's behavior is clearer.
By default, Breeze does not provide any way to do this, but you can keep the below code in your model JS file to overcome this issue:
breeze.DataType.parseDateFromServer = function (source) {
if (typeof source === 'string') {
//Check for local offset time or UTC time from server
if (source.slice(-1) !== "Z") {
var oldSource = source;
try {
source = source.substring(0, source.lastIndexOf("-") - 1)
source = new Date(source);
var tzDifference = source.getTimezoneOffset();
//convert the offset to milliseconds, add to targetTime, and make a new Date
var offsetTime = new Date(source.getTime() + tzDifference * 60 * 1000);
return offsetTime;
}
catch (err) {
source = new Date(source);
return source;
}
}
else {
source = new Date(source);
var tzDifference = source.getTimezoneOffset();
//convert the offset to milliseconds, add to targetTime, and make a new Date
var offsetTime = new Date(source.getTime() + tzDifference * 60 * 1000);
return offsetTime;
}
}
}

Validating two dates in acrobat javascripts

well the code I posted below all works perfectly except for a small detail. When I input today date in the field dateEntered, the later rejects it, it validates if the date entered is before todays date, validate if the date falls on a weekends, but it also show an error message when it is todays date. Actually the user should be able to enter Today or after date.
Anyone can tell me where am wrong, already tried every possible ways but still not working even the ( ==) or (===) or (<=) ..nothing
if (event.value!="")
{
var e = util.scand("ddd, dd.mmm.yy", event.value);
var a = (e.getTime()) < (new Date().getTime());
if (a) {
app.alert("The Date cannot be before Today's Date", 1);
event.rc = null;
}
if (e.getDay()==6 || e.getDay()==0) {
app.alert("Cannot take permission on a Weekend!", 2);
event.rc=null;
}
}
I found the solution to my problem, I had to set the hour to 0. Thank to the one who updated this on stackoverflow and sorry forget to retain your name.
if (event.value!="")
{
var e = util.scand("ddd, dd.mmm.yy", event.value);
var b=new Date();
b.setHours(0,0,0,0);
if (e<b) {
app.alert("ERROR: Date cannot be before"+" "+ new Date(b), 5);
event.rc = null;
}
if (e.getDay()==6 || e.getDay()==0) {
app.alert("ALERT: The date you entered ("+event.value+") falls on a WEEKEND!", 3);
event.rc=null;
}
}
This codes also contains a condition of removing one weekend from the dates since the number of leaves allowed to take ranges from 1 to 7 thus only one weekend is remove.