I have been trying to fetch historical prices of Bitcoin on FTX using their REST API. I created a small Python script that does that for me. The scripts works but it seems like the API is bugging.
This is the request that FTX has in its docs to fetch historical prices :
GET /markets/{market_name}/candles?resolution={resolution}&start_time={start_time}&end_time={end_time}
So I tried it with resolution = 60 (in seconds)
market_name = BTC-PERP start_time = 1559881511 (which is a timestamp) and end_time = 1559881711 [ actually these numbers are in an example given by FTX on their API page]
And the result of that request was : {"success":true,"result":[]} which clearly means that something is wrong.
I tried again without end_time and it worked perfectly, and if I put only end_time and no start_time it does the same bug.
Can't really figure it out
Is there data still available via API from that far back (2019)? I believe it works without end_date because it queries in reverse-chronological order, so it is just showing you the most recent candles.
The end_date parameter seems to work with more recent chunks of time:
https://ftx.com/api/markets/BTC-PERP/candles?resolution=60&start_time=1654097434&end_time=1656689434
e.g. June to July this year, above.
Related
I'm writing a web app using Firestore that needs to be able to show "Today's Popular Posts" and I'm having trouble getting the queries right when considering users in different timezones.
The dates are stored in the DB as UTC 0, then adjusted to the current user's UTC offset in the client via Moment.js. This works correctly.
When adding a new post I use firebase.firestore.FieldValue.serverTimestamp() to store the current server timestamp in a field called timestamp, like so:
const collectionRef = db.collection('posts');
collectionRef.add({
name: "Test Post",
content: "Blah blah blah",
timestamp: firebase.firestore.FieldValue.serverTimestamp(),
likeCount: 0
});
Then on the server I have a Cloud Function that runs on create and adds another field to the document called datestamp which is the the UTC 0 timestamp, but adjusted so that the time is the beginning of the day. The function looks like this:
exports.updatePostDate = functions.firestore
.document('posts/{postID}')
.onCreate((event) => {
const db = admin.firestore();
const postRef = db.doc('post/'+event.params.postID);
const postData = event.data.data();
const startOfDay = moment(postData.timestamp).startOf('day').toDate();
return postRef.update({
datestamp: startOfDay
});
});
Storing a timestamp where the time is always the beginning of the day enables me to write a query like this for finding all posts and ordering by popularity on a given day:
const startOfDayUTC = moment.utc().startOf('day').toDate();
const postQuery = db.collection('posts')
.orderBy('likeCount', 'desc')
.orderBy('timestamp', 'desc')
.where('datestamp', '==', startOfDayUTC)
.limit(25);
The problem is, depending on the user's UTC offset, this can display posts with two different dates when parsing the post's timestamp field. So even though the query is correctly fetching all the posts where the datestamp is say, 2018-01-30T00:00:00Z, the timestamp's date might not be the same once parsed. Here's an example of two posts:
Post 2:
likeCount: 1
timestamp (UTC 0): 2018-01-30T06:41:58Z
timestamp (parsed to UTC-8): 2018-01-29T22:41:58-08:00
datestamp (UTC 0): 2018-01-30T00:00:00Z
Post 1:
likeCount: 0
timestamp (UTC 0): 2018-01-30T10:44:35Z
timestamp (parsed to UTC-8): 2018-01-30T02:44:35-08:00
datestamp (UTC 0): 2018-01-30T00:00:00Z
So you can see, while the posts have the same datestamp, after adjusting the timestamp to the local UTC, the timestamp fields can end up being on two different days.
If anyone has a solution to this I would be very grateful.
I think it is better to avoid functions in this case as you can perform compound queries now. You can simply use
query.where(date > lastMidnight).where(data < now).get().then(...)
so to speak to limit data which only belongs to one day and try to keep all your time variables in UTC 0 and just find the start point and the current time both client side and convert them to UTC0.
//get local time from midnight to now (local)
const now = new Date();
const lastMidnight = now.setHours(0,0,0,0);
//then convert those to UTC0 to pass on in your query to firestore
const lastMidNightUTC = new Date(lastMidnight + now.getTimezoneOffset() * 60000).toString();
const nowInUTC = new Date(now + now.getTimezoneOffset() * 60000).toString();
and you can get your data (remember you need to make an index or just run the query once and firebase SDK will generate a link to create the index in dev tools -> console , for you)
query.where(date > lastMidNightUTC).where(data < now).get().then(...)
I came up with a solution that I'm really not happy with... But it works!
The problem is fundamentally one post can be on more than one date, depending on the user's location. And since for this case we also want to order by a field other than timestamp we can't use a range query to select posts on a given date, because your first .orderBy must be on the field you're using a range query on (see Firestore docs).
My solution is to map localized datestamps to their corresponding UTC offset. The object contains every UTC offset as a key, and the post's datestamp in that offset's time.
An example post looks like this:
posts/{somepostid}
{
name: "Test Post",
content: "Blah blah blah",
timestamp: Mon Jan 29 2018 21:37:21 GMT-0800 (PST),
likeCount: 0,
utcDatemap: {
0: "2018-01-30,
...
-480: "2018-01-29",
...
}
}
The field utcDatemap is the the one we use in our queries now:
const now = moment();
const datestamp = now.format("YYYY-MM-DD");
const utcOffset = now.utcOffset();
const utcDatemapField = 'utcDatemap.'+utcOffset;
const postQuery = db.collection('posts')
.orderBy('likeCount', 'desc')
.orderBy('timestamp', 'desc')
.where(utcDatemapField, '==', datestamp)
.limit(25);
Now posts can show up on two different days, depending on where the user is querying from. And we can still convert the regular old timestamp to the user's local time on the client.
This is definitely not an ideal solution. For the above query I needed to create composite indexes for every single key in utcDatemap. I'm not sure what the rules of thumb are with composite indexes, but I'm thinking having 39 indexes for something simple like this is probably not great.
Additionally I checked it out using the roughSizeOfObject function from thomas-peter's answer on this post and the utcDatemap object, with all it's string datestamps clocked in at roughly 780 bytes, and it's not like 0.78kb is a lot, but you do need to be mindful of how much data you're transferring with a service like Firestore (0.78kb is a lot for a date).
I'm learning/reading up on Firestore and have Google'd to see how it deals with times, so discount my answer appropriately.
It looks as though Firestore converts times to UTC and stores them as its own Timestamp datatype. If so, then it's critical to know that this is a destructive conversion.
Though UTC is useful for comparing instants in time, it means that the wall-clock time as observed by the app user is lost forever. Some countries like the UK are in one of two timezones during the year, Daylight Savings Time and British Summer Time.
You can convert back to the user's observed time, but the problem is that the rules change over the years. You'd have to keep a record of all the different rule changes for all the timezones and countries of the world.
What was the time at the time?
The question is, what time did the user think an event happened ...at the time. This can have legal ramifications. You may need to go back through historic data to prove a person acted at a certain time as they observed it.
The solution is to capture the user's observed offset in an additional field. That way, you can always use this to convert.
Regarding the OPs problem, this seems somewhat philosophical for a web app. Does "today" mean the event, such as a post, must have happened within the user's Monday? Or just posts on today's date? Or posts within the last 24h?
An important thing to remember is that dates are the same all around the world, even when they begin and end at different instants.
What's Elvis got to do with all this?
Christmas Day is 25th everywhere. If I say something happened on Christmas Day and I'm in the USA, and then someone in Australia wants to see all the world's posts made on Christmas Day, then they need to query for posts where observedDate == 25th Dec.
Think about it. Such posts were all made on Christmas Day, even though it might have been Boxing Day for me in England at the instant that they posted.
Elvis died on 16th August. In the UK our radio stations don't all wait until it's the 16th in the timezone of the place of his death to start playing his records.
Another interesting one is whether something happened in Q1 or Q2 of a company's reporting year. Is a sale recognised as on the date at the point-of-sale in the store in New York, or in the database in LA?
The observed date is interesting.
My point is, think deeply about this and store both a normalised instant like UTC, but also the user's observed date, time and offset. Then you have all the information you'll need for the future.
Finally, consider adding observed year or week numbers, or day ordinals, H1/H2 for financial data, since it can be super useful for composing rapid queries, depending on your use-cases.
I'm quite new YQL and i've found the query to retrieve a single quote from a stock
select * from yahoo.finance.quote symbol = "YHOO"
and another query to get this same information but on date range
select * from yahoo.finance.historicaldata symbol = "YHOO" and startDate = "2016-09-01" and endDate = "2016-09-22"
What i could not figure out was: how could we retrieve quotes from a full day of trading?
I'm currently using the Yahoo finance app and notice they provide a good graphic about the price variation, so i presume there is a way to achieve it.
I also tried to read yql tables repository but on both table that i am using there is no (at least explicit) clue of how to pass hour range.
You can retrieve the complete quotes of a day by querying the Yahoo Finance API endpoint directly (not via YQL) and receiving a list in JSON format.
The end point is http://chartapi.finance.yahoo.com/instrument/1.0/$symbol/chartdata;type=$type;range=$range/json/, where:
$symbol is the stock ticker symbol, e.g. AAPL for Apple or BAS.DE for BASF traded at Xetra
$type is the type of the query, you can query for quote, sma, close, volume
$range is the desired latest days with 1d, 5d, 10d, 15d
An example query would be
http://chartapi.finance.yahoo.com/instrument/1.0/aapl/chartdata;type=quote;range=1d/json/
which gives you all quotes from AAPL from the last day.
As far as I know, you can only query for the quotes up to the last 15 days. I have not yet found a way to query for some other day further in the past.
Just my self-centric hint: check out my PHP package YahooFinanceQuery on Github, which uses an implementation of the above query and handles the returning JSON to filter the results.
As an update/extension to my previous answer I found a new API endpoint to download daily quotes.
Yahoo changed their API endpoints in early 2017.
The new endpoint is:
https://query1.finance.yahoo.com/v8/finance/chart/{$symbol}?range={$range}&interval={$interval}, where:
$symbol is the stock ticker symbol, e.g. AAPL for Apple
$range is the desired range of the query, allowed parameters are [1d, 5d, 1mo, 3mo, 6mo, 1y, 2y, 5y, 10y, ytd, max]
$interval is the desired interval of the quote, e.g. every 5 minutes, allowed parameters are [1m, 2m, 5m, 15m, 30m, 60m, 90m, 1h, 1d, 5d, 1wk, 1mo, 3mo]
An example would be: https://query1.finance.yahoo.com/v8/finance/chart/AAPL?range=10d&interval=1m where you receive OHLCV quotes for the AAPL stock from the last 10 trading days with a 1 minute interval. All in a nicely JSON format.
Not all $range parameters will return results with the specified $interval, but will return the nearest possible combination. For example, the "max" range will return all quotes with a "1mo" interval.
New software that we have installed needs to have a specific id to be used as a refrence token to know where it left off in the sql database. Presently our other software that enters in data that i am refrencing is not giving no time 2013-05-20 00:00:00. I would like to combine that date time stamp with my pipesize and period.
The data looks like this:
trandate = 2013-05-20 00:00:00 Pipesize = 30 Period = A
I need to have the data converted to look like 2013052000000030A if it is possible or something close.
I used the chart here to determine the correct format: w3schools.com/sql/func_convert.asp and the resulting sqlfiddle here. sqlfiddle.com/#!3/d9654/3
I'm trying to do the following Facebook Query Language query:
https://api.facebook.com/method/fql.query?query=SELECT name FROM user WHERE uid IN (SELECT actor_id FROM stream WHERE app_id = 131580206909839 AND xid = 'daily_thred_production' AND created_time > 2011-03-06 AND created_time < 2011-03-08)
The problem is that the dates aren't being recognized and I can't find any documentation on how to format FQL dates from the Facebook developer section. Any thoughts?
EDIT
I'm doing all of this from the URL with no programming language. I'm just trying to pull one-off statistics for some co-workers.
Epoch time seems to work, thanks! Only problem is that it's only displaying new users that contributed to the stream for the first time. Unfortunately I'm trying to find everyone in the stream, I'll have to look at the stream table more carefully.
Thanks Brian.
They're epoch time (Number of seconds since 00:00:00 Jan 1, 1970 UTC)
You need to convert your dates to epoch time in whatever language you're using.
EDIT: If you need an example, let me know what programming lang you're using.
I am trying to create some Issue Filters in JIRA based on CreateDate.
The only date/time function I can find is Now() and searches relative to that, i.e. "-1d", "-4d" etc.
The only problem with this is that Now() is time specific so there is no way of getting a particular day's created issues.
i.e. Created < Now() AND Created >= "-1d"
when run at 2pm today will show all issues created from 2pm yesterday to 2pm today
when run at 9am tomorrow will show all issues created from 9am today to 9am tomorrow
What I want is to be able to search for all issues created from 00:00 to 23:59 on any day. Is this possible?
Check out startOfDay([offset]). That gets what you are looking for without the pesky time constraints and its built in as of 4.3.x. It also has variants like endOfDay, startOfWeek, startOfMonth, etc.
I run it like this -
created > startOfDay(-0d)
It gives me all issues created today. When you change -0d to -1d, it will give you all issues created yesterday and today.
We're using Jira 6.2 and I use this query:
updatedDate > startOfDay(-1d) AND updatedDate < endOfDay(-1)
to return all of the issues that were updated from the previous day. You can combine with whichever queries you want to return the appropriate issues for the previous day.
Just for the sake of keeping the information up-to-date, with at least JIRA 7.3.0 (maybe older as well) you can explicitly specify the date in multiple formats:
'yyyy/MM/dd HH:mm';
'yyyy-MM-dd HH:mm';
'yyyy/MM/dd';
'yyyy-MM-dd';
period format, e.g. '-5d', '4w 2d'.
Example:
updatedDate > '2018/06/09 0:00' and updatedDate < '2018/06/10 15:00'
In case you want to search for all the issues updated after 9am previous day until today at 9AM, please try: updated >= startOfDay(-15h) and updated <= startOfDay(9h). (explanation: 9AM - 24h/day = -15h)
You can also use updated >= startOfDay(-900m) . where 900m = 15h*60m
Reference: https://confluence.atlassian.com/display/JIRA/Advanced+Searching
A friend who is a JIRA wiz showed me that you can actually pass the filter (escaped) as a jqlQuery parameter to JIRA via URL:
http://hostname/secure/IssueNavigator!executeAdvanced.jspa?clear=true&runQuery=true&jqlQuery=created%3E='2010-05-31%2000:00'%20AND%20created%3C='2010-06-06%2023:59'%20ORDER%20BY%20created%20ASC
I created an ASP.Net page which generates the URLs based on an offset week or month.
Everybody's happy!
You would expect that this is easily possible but that seems not be the case. The only way I see at the moment is to create a user defined JQL function. I never tried this but here is a plug-in:
http://confluence.atlassian.com/display/DEVNET/Plugin+Tutorial+-+Adding+a+JQL+Function+to+JIRA
You might use one of our plugins: the JQL enhancement functions - check out
https://plugins.atlassian.com/plugin/details/22514
There is no interval on day, but we might add it in a next iteration, if you think it is usefull.
Francis.