twitter4j - setSince and setUntil don't work - twitter4j

I'm having problem to filter tweets by specifics dates, using setUntil come no tweets, using setSince come recently tweets.. The code is following and after that the output result..
public void readTweetFromKeyword(String keywordString) throws TwitterException
{
twitter4j.Query query =new twitter4j.Query("#clt20");
QueryResult result;
query.setSince("2014-12-12");
int cont = 0;
result = twitter.search(query);
for (Status status : result.getTweets() )
{
System.out.print("original "+status.getId());
System.out.println("\t\tdata "+status.getCreatedAt());
if(!status.getText().substring(0, 2).equals("RT")){
System.out.println(status.getText());
cont++;
}
}
System.out.println(result.getTweets().size());
System.out.println("cont = "+cont);
return;
}
CONSOLE:
original 619433499116896256 data Fri Jul 10 06:10:29 GMT-03:00 2015
If the #BCCI is looking for an alternative to #Clt20, then how about a
league of teams consisting of only Indian players ?
original 619408117495939072 data Fri Jul 10 04:29:37 GMT-03:00 2015
#TesT, #ODI, #T20I, #IPL, #CLT20 Live record, score, history shedule ke lia, Follow #PTV_SpOrtsOne snt to 40404.
original 619330143258050560 data Thu Jul 09 23:19:47 GMT-03:00 2015
Need 66 From 6 Balls. Kinda Impossible #clt20
original 619301555532120065 data Thu Jul 09 21:26:11 GMT-03:00 2015
Kamran Akmals feet are stuck #soshit #CLT20
original 619095093962608640 data Thu Jul 09 07:45:47 GMT-03:00 2015
original 619095079983017984 data Thu Jul 09 07:45:43 GMT-03:00 2015
original 619095051524665344 data Thu Jul 09 07:45:37 GMT-03:00 2015
original 619095028304973825 data Thu Jul 09 07:45:31 GMT-03:00 2015
original 619094989943902209 data Thu Jul 09 07:45:22 GMT-03:00 2015
original 619094910516400129 data Thu Jul 09 07:45:03 GMT-03:00 2015
original 619094893441363969 data Thu Jul 09 07:44:59 GMT-03:00 2015
original 619035151578722304 data Thu Jul 09 03:47:35 GMT-03:00 2015
#abhisek_taneja Games r played in Himachal Pradesh every year if u go
through the schedule of #IPL & #CLT20 properly
original 618914815730290688 data Wed Jul 08 19:49:25 GMT-03:00 2015
original 618908444939186177 data Wed Jul 08 19:24:06 GMT-03:00 2015
original 618862474687705088 data Wed Jul 08 16:21:26 GMT-03:00 2015
We as #T20 follower , #clt20 should be oganized #CLT20
15 cont = 6
Thanks a lot!!

If you set an until date kepp in mind this from the documentation
Returns tweets generated before the given date. Date should be
formatted as YYYY-MM-DD. Keep in mind that the search index may not go
back as far as the date you specify here.
And this too
Before getting involved, it’s important to know that the Search API is
focused on relevance and not completeness. This means that some Tweets
and users may be missing from search results. If you want to match for
completeness you should consider using a Streaming API instead.
So, if you set an until date too old you could get zero tweets, in the other hand if you set a since date too old you could get only tweets from the few past days as you get in the console.

Related

How to apply distinct in one column, while getting few other columns from table

How to apply distinct in one column, while getting few other columns from table.
For example, something like this,
Select distinct date(date0), name from note
So the goal here is to get all the results, for different days(distinct date(date0))
Sample data:
Date
Name
01st Jan 2021
Bohemian
01st Jan 2021
Bohemian
01st Jan 2021
Bohemian
02nd Jan 2021
Jack
03rd Jan 2021
John
Expected Results:
Date
Name
01st Jan, 2021
Bohemian
02nd Jan 2021
Jack
03rd Jan 2021
John
You seem to be asking for this:
Select distinct date, name from note
Best regards,Bjarni

AWS Glue Dynamic Frame Pushdown Predicate List

When using pushdown predicate with AWS Glue Dynamic frame, how does it iterate through a list?
For example, the following list was created to be used as a pushdown predicate:
day=list(p_day.select('day').toPandas()['day'])
month=list(p_month.select('month').na.drop().toPandas()['month'])
year=list(p_year.select('year').toPandas()['year'])
predicate = "day in (%s) and month in (%s) and year in (%s)"%(",".join(map(lambda s: "'"+str(s)+"'",dat))
,",".join(map(lambda s: "'"+str(s)+"'",month))
,",".join(map(lambda s: "'"+str(s)+"'",year)))
Let's say it returns this:
"day in ('07','15') and month in ('11','09','08') and year in ('2021')"
How would the push down predicate read this combination/list?
Is it:
day
month
year
07
11
2021
15
11
2021
07
09
2021
15
09
2021
07
08
2021
15
08
2021
-OR-
day
month
year
07
11
2021
15
11
2021
15
08
2021
15
09
2021
I have a feeling that this list is read like the first table rather than the latter... But, it's the latter that I would like to pass through as a pushdown predicate. Does creating a list essentially cause a permutation? It's as if the true day, month, and year combination is lost in the list which should be 11/7/2021, 11/15/2021, 08/15/2021, and 09/15/2021.
This has nothing to do with Glue itself, since the Partition Predicate is just basic Spark SQL. You will receive the first list and not the second. You would have to restructure the boolean expression to receive the second list.

mongodb searching for date strings in a range

I have a DB with entries in Universal Time such as this:
{"problemDate": "Mon May 11 2020 13:09:14 GMT+0000 (Coordinated Universal Time)"},
{"problemDate": "Mon May 11 2020 13:09:14 GMT+0000 (Coordinated Universal Time)"},
{"problemDate": "Mon Dec 07 2020 14:08:26 GMT+0000 (Coordinated Universal Time)"},
{"problemDate": "Tue May 12 2020 00:18:21 GMT+0000 (Coordinated Universal Time)"}
Although, I am trying to query for a range of dates, my db has around 2k entries with dates ranging from May to December. If I query for dates greater than December 7th, my return still shows dates such as Mon May 11. I believe this is because my dates are just strings. I was wondering if there is a work around to this. Any help would be appreciated.
This was my query:db.problem.find({"problemDate":{ $gte:("Mon Dec 07 2020 14:08:24 GMT+0000 (Coordinated Universal Time)") }});
You could try an update on the collection to convert the dates from string to IsoDate.
The following script it's designed to run in a mongo shell, and it's assumed that you have selected the database (use databasename) and your collection name is 'datestest':
db.getCollection('datestest').find({}).forEach( function(item){
if (typeof(item.problemDate) == "string"){
print(item.problemDate);
item.problemDate = new Date(item.problemDate);
db.getCollection('datestest').save(item);
}
}
);
You can first save the "bad dates" to a different collection to check the update it's all right, just replace the save item with a diffrerent collection name:
db.getCollection('datestest_check').save(item);

MongoDB/mongoose some dates as EDT and others as EST

When getting some records, I'm seeing that some dates are showing as GMT-0400 (EDT) and others are GMT-0500 (EST).
Dates are being added in mongoose simply by using Date.now in the schema.
Any ideas what could cause the offsets to be different?
Edit: Here's an example:
Stored as: ISODate("2015-10-30T15:36:47.287Z") Returned as: Fri Oct 30 2015 11:36:47 GMT-0400 (EDT) using find() with Mongoose
Stored as: ISODate("2015-11-07T14:44:47.956Z") Returned as: Sat Nov 07 2015 09:44:47 GMT-0500 (EST) using find() with Mongoose
It's just the default string representation of the underlying UTC timestamp, showing the timestamp in the local time zone that was in effect at the time.
In 2015, daylight saving time ended at 2:00 AM on Nov 1, so that's why the first one is shown in EDT and the second one is shown in EST.
Any queries you do are always performed using UTC time.

How to fast enumerate and compare objects of many NSArrays in Objective-C?

I have many different NSArray's stored in .dat files, in the Documents folder of my iPhone application, like so:
john.dat
mary.dat
bob.dat etc...
The number of .dat files is unknown and it will increase or decrease according to a number of factors related to the user's manipulation of the app.
What are the contents of these NSArray's stored in the.dat files? They contain NSString's. It is important to say that the countfor each NSArray my vary, like so:
john.dat Contains this array of strings:
"Mon, 11 Jun 2012 04:52:06 GMT",
"Tue, 12 Jun 2012 04:51:59 GMT",
"Wed, 13 Jun 2012 04:52:30 GMT",
"Thu, 07 Jun 2012 04:52:00 GMT",
mary.dat Contains this array of strings:
"Mon, 11 Jun 2012 04:52:06 GMT",
"Tue, 12 Jun 2012 04:51:59 GMT",
bob.dat Contains this array of strings:
"Mon, 11 Jun 2012 04:52:06 GMT",
"Tue, 12 Jun 2012 04:51:59 GMT",
"Wed, 13 Jun 2012 04:52:30 GMT",
"Thu, 07 Jun 2012 04:52:00 GMT",
"Fri, 08 Jun 2012 04:51:59 GMT",
"Sun, 10 Jun 2012 04:50:55 GMT"
etc..
I have to periodically verify if each NSString of these NSArray's has been modified, to notify the user of certain events. After doing a XML parsing from the Internet I'm already able to have in memory the new NSArray's corresponding to its respective .dat file previously stored in the Documents folder.
Since I have many .dat files, my question is:
How do I compare EFFICIENTLY AND AT ONCE the new NSArray's (NSArray *john, *mary, *bob, etc...) that I have in memory (as a result of this XML parsing), with the old ones stored in its corresponding john.dat, mary.dat, bob.dat, etc... files in the Documents folder and notify the user if any of the NSString's ("Mon, 11 Jun 2012 04:52:06 GMT","Tue, 12 Jun 2012 04:51:59 GMT", etc...) has been modified?
Thanks for your help!
If you're happy to change the format of your .dat file you could compute a hash of each array's contents and store that at the beginning of the file. When you get new data you can compute its hash and compare that with the hash stored in your file.
Your choice of hashing function will determine the computational efficiency, but the I/O would be minimized.
Well, if you content with storing your arrays as .dat files (with NSKeyedArchiver I'm guessing), then Martin's answer is probably best and the tried and true method of comparison
The approach I might take is to store the NSStrings in NSDictionarys instead of NSArrays. Then you could set an observer on its values, so when an string is pulled from the XML file you compare, replace if necessary and the observer will be called so you can notify the user (see NSNotificationCenter and KVO if not already familiar).
It may also be beneficial (usually is) to store your strings in CoreData as opposed to .dat files, then you could configure an NSFetchedResultsController to keep an eye on your data and do the above when new data is fetched.