Filter a polars dataframe by date in rust - date

Im trying to filter a dataframe by date.
But filter it with expressions like this would be really cumbersome for a date like "2019-11-01 10:15:00".
My goal is to do something like the python version:
use polars::export::chrono::NaiveDateTime;
use polars::prelude::*;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let df = LazyCsvReader::new(path)
.with_parse_dates(true)
.has_header(true)
.finish()?
.collect()?;
let dt = NaiveDateTime::parse_from_str("2019-11-01 10:15:00", "%Y-%m-%d %H:%M:%S")?;
//This will not compile!
let filtered = df.filter(col("time") < dt);
}
However I'm having a really hard time to filter the dateframe in-place or just creating a boolean mask.

After more time than I dare to admit I finally solved it by using the eager API, there is probably a better solution in the Lazy-API but this works for now!
use polars::export::chrono::NaiveDateTime;
use polars::prelude::*;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let df = LazyCsvReader::new(path)
.with_parse_dates(true)
.has_header(true)
.finish()?
.collect()?;
// Set date to filter by
let dt = NaiveDateTime::parse_from_str("2019-11-01 10:15:00", "%Y-%m-%d %H:%M:%S")?;
// Create boolean mask
let mask = df["time"]
.datetime()?
.as_datetime_iter()
.map(|x| x.unwrap() < dt)
.collect();
// New filtered df
let filtered_df = df.filter(&mask)?;
}
To get a date value from the "time" column and parse it to as a NaiveDateTime:
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Lets take the last date from a series of datetime[µs]
let date: Vec<Option<NaiveDateTime>> = df["time"]
.tail(Some(1))
.datetime()?
.as_datetime_iter()
.collect();
// Create new NaiveDateTime, can be used as filter/condition in map-function
let dt2 = NaiveDateTime::parse_from_str(&date[0].unwrap().to_string(), "%Y-%m-%d %H:%M:%S")?;
}

Related

Retrieve Postgres TSTZRANGE to PgRange<Datetime<Utc>> with Rust sqlx

I have the following SQL table scheme
CREATE TABLE reservation (
id INTEGER PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
timespan TSTZRANGE
);
INSERT INTO reservation(timespan) VALUES
(TSTZRANGE(now() + INTERVAL '0 hour', now() + INTERVAL '1 hour')),
(TSTZRANGE(now() + INTERVAL '2 hour', now() + INTERVAL '3 hour'));
and using rust-sqlx, I would like to retrieve those rows directly into a struct.
/*
[dependencies]
chrono = "0.4"
tokio = { version = "1", features = ["full"] }
sqlx = { version = "0.5", features = [ "runtime-tokio-native-tls" , "postgres" ] }
*/
use chrono::prelude::*;
use sqlx::FromRow;
use sqlx::postgres::{types::PgRange, PgPoolOptions};
#[derive(Debug, FromRow)]
struct Reservation {
id: i32,
timespan: PgRange<DateTime<Utc>>,
}
#[tokio::main]
async fn main() -> Result<(), sqlx::Error> {
let db_url: String = "postgresql://postgres:postgres#localhost/mydb".to_owned();
let pool = PgPoolOptions::new()
.max_connections(2)
.connect(&db_url)
.await?;
let select_query = sqlx::query_as::<_, Reservation>("SELECT id, timespan FROM reservation");
let reservations: Vec<Reservation> = select_query.fetch_all(&pool).await?;
dbg!("{:?}", reservations);
Ok(())
}
I am getting the following 2 following errors
error[E0277]: the trait bound `PgRange<chrono::DateTime<chrono::Utc>>: Type<_>` is not satisfied
...
...
error[E0277]: the trait bound `chrono::DateTime<chrono::Utc>: Type<Postgres>` is not satisfied
How do I go about manually implementing these trait bounds? or is there a simpler way to achieve the same thing?
Newbie mistake! I just had to add chrono flag for sqlx in Cargo.toml
sqlx = { version = "0.5", features = [ "runtime-tokio-native-tls" , "postgres" , "chrono" ] }

Neatly parsing a date in "MMddyy" format along with other formats in dart

I guess it is not possible to parse a date in "MMddyy" format in dart.
void main() {
String strcandidate = "031623";
String format = "MMddyy";
var originalFormat = DateFormat(format).parse(strcandidate);
}
Output:
Uncaught Error: FormatException: Trying to read dd from 031623 at position 6
The following works fine when parsing a date in "MM-dd-yy" format.
void main() {
String strcandidate = "03-16-23";
String format = "MM-dd-yy";
var originalFormat = DateFormat(format).parse(strcandidate);
}
In the problem, the input date string can be in any format e.g ['yyyy-MM-dd', 'MMM'-yyyy, 'MM/dd/yy']. I am parsing the input string for these formats in a loop as follows.
dateFormatsList = ['yyyy-MM-dd', 'MMM'-yyyy, 'MM/dd/yy'];
for (String format in dateFormatsList ) {
try {
originalFormat = DateFormat(format).parse(strcandidate);
dateFound = true;
} catch (e) {}
}
Adding 'MMddyy' to dateFormatsList is not going to work.
But regular expression be used to parse this format.
However if all formats are parsed using parse method and one additional format is parsed using regular expression, then the code is not that neat, and cluttered.
To write as much neat and efficient code as possible, if you want, you can share your insights about any possibility for making it efficient and clean while incorporating 'MMddyy'format. Tysm!
See How do I convert a date/time string to a DateTime object in Dart? for how to parse various date/time strings to DateTime objects.
If you need to mix approaches, you can provide a unified interface. Instead of using a List<String> for your list of formats, you can use a List<DateTime Function(String)>:
import 'package:intl/intl.dart';
/// Parses a [DateTime] from [dateTimeString] using a [RegExp].
///
/// [re] must have named groups with names `year`, `month`, and `day`.
DateTime parseDateFromRegExp(RegExp re, String dateTimeString) {
var match = re.firstMatch(dateTimeString);
if (match == null) {
throw FormatException('Failed to parse: $dateTimeString');
}
var year = match.namedGroup('year');
var month = match.namedGroup('month');
var day = match.namedGroup('day');
if (year == null || month == null || day == null) {
throw ArgumentError('Regular expression is malformed');
}
// In case we're parsing a two-digit year format, instead of
// parsing the strings ourselves, reparse it with [DateFormat] so that it can
// apply its -80/+20 rule.
//
// [DateFormat.parse] doesn't work without separators, which is why we
// can't use directly on the original string. See:
// https://github.com/dart-lang/intl/issues/210
return DateFormat('yy-MM-dd').parse('$year-$month-$day');
}
typedef DateParser = DateTime Function(String);
DateParser dateParserFromRegExp(String rePattern) =>
(string) => parseDateFromRegExp(RegExp(rePattern), string);
var parserList = [
DateFormat('yyyy-MM-dd').parse,
DateFormat('MMM-yyyy').parse,
DateFormat('MM/dd/yy').parse,
dateParserFromRegExp(
r'^(?<month>\d{2})(?<day>\d{2})(?<year>\d{4})$',
)
];
void main() {
var strcandidate = '12311776';
DateTime? originalFormat;
for (var tryParse in parserList) {
try {
originalFormat = tryParse(strcandidate);
break;
} on Exception {
// Try the next format.
}
}
print(originalFormat);
}
I think it's a bit hacky but what about use a regular expression (RegExp) to parse the date divider and then replace it with just ""?
void main() {
String strcandidate = "031623";
String strYear = strcandidate.substring(4);
//Taken 20 as the year like 2023 as year is in 2 digits
String _newDateTime = '20' + strYear + strcandidate.substring(0, 4);
var _originalFormat = DateTime.parse(_newDateTime);
print(_originalFormat);
}
add the intl to yaml then write this code:
import 'package:intl/intl.dart';
void main() {
var strcandidate = DateTime(2023, 3, 16);
String format = "MMddyy";
var originalFormat = DateFormat(format).format(strcandidate);
print(originalFormat);
}

How can I avoid string conversions when ingesting timestamps to postgres in Rust?

I'm using the rust-postgres crate to ingest data. This is a working example adding rows successfully:
let name: &str = "hello from rust";
let val: i32 = 123;
let now: DateTime<Utc> = Utc::now();
let timestamp = now.format("%Y-%m-%dT%H:%M:%S%.6f").to_string();
client.execute(
"INSERT INTO trades VALUES(to_timestamp($1, 'yyyy-MM-ddTHH:mm:ss.SSSUUU'),$2,$3)",
&[&timestamp, &name, &val],
)?;
This doesn't look so nice as I have to do this forward and back string conversion, I would like to be able to write something like
let name: &str = "hello from rust";
let val: i32 = 123;
let now: DateTime<Utc> = Utc::now();
client.execute(
"INSERT INTO trades VALUES($1,$2,$3)",
&[&now, &name, &val],
)?;
What's the most performant way of ingesting timestamps in this way?
Edit:
Here's the returned error from the second example above
Error: Error { kind: ToSql(0), cause: Some(WrongType { postgres: Timestamp, rust: "chrono::datetime::DateTime<chrono::offset::utc::Utc>" }) }
And my cargo.toml looks like this (which has the chrono feature enabled for the rust postgres crate):
[dependencies]
chrono = "0.4.19"
postgres={version="0.19.0", features=["with-serde_json-1", "with-bit-vec-0_6", "with-chrono-0_4"]}
I think the problem is a mismatch between your postgres schema and your Rust type: the error seems to say that your postgres type is timestamp, while your rust type is DateTime<Utc>.
If you check the conversion table, DateTime<Utc> converts to a TIMESTAMP WITH TIME ZONE. The only types which convert to TIMESTAMP are NaiveDateTime and PrimitiveDateTime.
As per Masklinn's response, I needed to pass a NaiveDateTime type for this to work, the full example with naive_local looks like:
use postgres::{Client, NoTls, Error};
use chrono::{Utc};
use std::time::SystemTime;
fn main() -> Result<(), Error> {
let mut client = Client::connect("postgresql://admin:quest#localhost:8812/qdb", NoTls)?;
// Basic query
client.batch_execute("CREATE TABLE IF NOT EXISTS trades (ts TIMESTAMP, date DATE, name STRING, value INT) timestamp(ts);")?;
// Parameterized query
let name: &str = "rust example";
let val: i32 = 123;
let utc = Utc::now();
let sys_time = SystemTime::now();
client.execute(
"INSERT INTO trades VALUES($1,$2,$3,$4)",
&[&utc.naive_local(), &sys_time, &name, &val],
)?;
// Prepared statement
let mut txn = client.transaction()?;
let statement = txn.prepare("insert into trades values ($1,$2,$3,$4)")?;
for value in 0..10 {
let utc = Utc::now();
let sys_time = SystemTime::now();
txn.execute(&statement, &[&utc.naive_local(), &sys_time, &name, &value])?;
}
txn.commit()?;
println!("import finished");
Ok(())
}

How can I read a timestamp with timezone (timestamptz) value from PostgreSQL in Rust?

What's the right Rust data type to use for a timestamptz when using postgres version 0.17.0 with Rust 1.40.0?
I read the docs for Timestamp but have no idea what this means or how to implement it.
The readme for 0.17.0-alpha.1 has a table which says that timezone corresponds to Rust types time::Timespec or chrono::DateTime<Utc> but neither works for me.
When I try to use the stipulated features in my Cargo.toml using:
[dependencies]
postgres = {version="0.17.0-alpha.1", features=["with-chrono", "with-time"]}
I get this error:
the package `mypackage` depends on `postgres`, with features: `with-time, with-chrono` but `postgres` does not have these features.
Here's some functional code and corresponding dependencies. I want to be able to read and print the timezone per row (commented out)
main.rs
use postgres::{Client, Error, NoTls};
extern crate chrono;
use chrono::{DateTime, Local, NaiveDateTime, TimeZone, Utc};
extern crate time;
use time::Timespec;
pub fn main() -> Result<(), Error> {
let mut client = Client::connect("host=localhost user=postgres", NoTls)?;
client.simple_query(
"
CREATE TABLE mytable (
name text NOT NULL,
timestamp timestamptz NOT NULL)",
)?;
client.execute("INSERT INTO mytable VALUES ('bob', now());", &[])?;
for row in client.query("SELECT * FROM mytable", &[])? {
let name: &str = row.get(0);
// let timestamp: chrono::DateTime<Utc> = row.get(1); //doesnt work
// let timestamp: Timespec = row.get(1); //doesnt work
println!("name: {}", name);
// println!("timestamp: {}", timestamp);
}
Ok(())
}
Uncommenting
let timestamp: Timespec = row.get(1); //doesnt work
error[E0277]: the trait bound `time::Timespec: postgres_types::FromSql<'_>` is not satisfied
--> src/main.rs:30:39 | 30 |
let timestamp: Timespec = row.get(1); //doesnt work
^^^ the trait `postgres_types::FromSql<'_>` is not implemented for `time::Timespec`
Uncommenting
let timestamp: chrono::DateTime<Utc> = row.get(1); //doesnt work
error[E0277]: the trait bound `chrono::DateTime<chrono::Utc>: postgres_types::FromSql<'_>` is not satisfied
--> src/main.rs:29:52 29 |
let timestamp: chrono::DateTime<Utc> = row.get(1); //doesnt work
^^^ the trait `postgres_types::FromSql<'_>` is not implemented for `chrono::DateTime<chrono::Utc>`
Cargo.toml
[dependencies]
postgres = "0.17.0"
chrono = "0.4.10"
time = "0.1.14"
This link says to use time = "0.1.14". latest version also fails https://crates.io/crates/postgres/0.17.0-alpha.1
Once you know what features are available, it's reasonably direct to see that you need to use the with-chrono-0_4 feature.
use chrono::{DateTime, Utc}; // 0.4.10
use postgres::{Client, Error, NoTls}; // 0.17.0, features = ["with-chrono-0_4"]
pub fn main() -> Result<(), Error> {
let mut client = Client::connect("host=localhost user=stack-overflow", NoTls)?;
client.simple_query(
r#"
CREATE TABLE mytable (
name text NOT NULL,
timestamp timestamptz NOT NULL
)"#,
)?;
client.execute("INSERT INTO mytable VALUES ('bob', now());", &[])?;
for row in client.query("SELECT * FROM mytable", &[])? {
let name: &str = row.get(0);
let timestamp: DateTime<Utc> = row.get(1);
dbg!(name, timestamp);
}
Ok(())
}
[src/main.rs:20] name = "bob"
[src/main.rs:20] timestamp = 2020-01-16T01:21:58.755804Z
Thanks to https://github.com/sfackler/rust-postgres/issues/211, this works using version 0.15.0 of the postgres crate, but I'd like a solution using version 0.17.0.
main.rs
extern crate postgres;
use postgres::{Connection, TlsMode};
extern crate chrono;
use chrono::{DateTime, Local, NaiveDateTime, TimeZone, Utc};
fn main() {
let conn = Connection::connect("postgresql://postgres#localhost:5432", TlsMode::None).unwrap();
conn.execute(
"CREATE TABLE person (
name VARCHAR NOT NULL,
timestamp timestamptz
)",
&[],).unwrap();
conn.execute("INSERT INTO person VALUES ('bob', now());", &[]).unwrap();
for row in &conn.query("SELECT * FROM person", &[]).unwrap() {
let name: String = row.get(0);
let timestamp: chrono::DateTime<Utc> = row.get(1);
println!("name: {}", name);
println!("timestamp: {}", timestamp);
}
}
Output:
name: bob
timestamp: 2020-01-15 23:56:05.411304 UTC
Cargo.toml
[dependencies]
postgres = { version = "0.15", features = ["with-chrono"] }
chrono = "0.4.10"
time = "0.1.14"

Convert a string to base64

I need a simple thing: encode a string in base64. I found an example:
extern crate serialize;
use serialize::base64::{mod, ToBase64};
use serialize::hex::FromHex;
fn main() {
let input = "49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d";
let result = input.from_hex().unwrap().as_slice().to_base64(base64::STANDARD);
println!("{}", result);
}
Which seams to work but I don't understand why input contains only characters in HEX. Moreover, this Python code produces a different result:
base64.b64encode(input) # =>
'NDkyNzZkMjA2YjY5NmM2YzY5NmU2NzIwNzk2Zjc1NzIyMDYyNzI2MTY5NmUyMDZjNjk2YjY1MjA2MTIwNzA2ZjY5NzM2ZjZlNmY3NTczMjA2ZDc1NzM2ODcyNmY2ZjZk'
So I decided to do the following:
//....
let input = "some string 123";
let result2 = input.unwrap().as_slice().to_base64(base64::STANDARD);
let result3 = input.as_slice().to_base64(base64::STANDARD);
And it didn't compile due to the errors:
error: type `&str` does not implement any method in scope named `unwrap`
test1.rs:9 let result2 = input.unwrap().as_slice().to_base64(base64::STANDARD);
^~~~~~~~
test1.rs:9:34: 9:44 error: multiple applicable methods in scope [E0034]
So how do I encode a simple string in base64?
If you don't have hex input, try this:
let result = input.as_bytes().to_base64(base64::STANDARD);
to_base64 is only defined for a slice of bytes so you have to first call as_bytes on the string:
extern crate serialize;
use serialize::base64::{mod, ToBase64};
fn main() {
let input = "49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d";
let result = input.as_bytes().to_base64(base64::STANDARD);
println!("{}", result);
}
Input has the type &static str:
let input = "some string 123";
There is no unwrap defined for &'static str:
let result2 = input.unwrap().as_slice().to_base64(base64::STANDARD);
You already have a slice (&str) but you need &[u8]:
let result3 = input.as_slice().to_base64(base64::STANDARD);