I was playing around with a few magnetic cards (parking lot tickets, time limited access etc) with my MSR605 reader.
When the format is ISO, it is quite straigforward to extract clear data. When it is not, I don't understand how I can decode the data on the card. I can't neither understand how I can see what kind of encoding has been used.
For example, I have one ticket printed out on the 17th of December 2016 # 18:01:22. the raw data on the track 3 of the card is:
D58F6A9C21856A2AD6DE722A2AD6DE72021084210C2A1587DA591B6A849975AA70861525ABEB06A283841B78
I have another ticket printed out on the 17th of December 2016 # 18:03:21. the raw data on the card is:
D58F6A9C21856A2AD5C1ACEA2AD5C1ACC21084210C2A1587DA779C8B0742ADAA70861525ABE779DC5B782778
You can find redundancies... But it is impossible for me to find where the complete date/time is!
More generally, how do you know what kind of method is used to encoded data on a magnetic stripe?
Can you help me please to find the date/time on this magstripe and explain me how you did it?
Thanks!
Related
Discussing data time-formats, someone mentioned to me how he stores datetime (in a human-readable format) using floats as yyyymmdd.hhmmss, so 2019-09-18, 11:29:30am would become 20190918.112930
I'm trying to find out if this guy has invented his own format or if it is used (and described) elsewhere too - and if so, how is it even called...?
It’s probably homespun
I have seen a lot of date and time formats, and I have not seen this one before. My go is that his guy or his organization invented it themselves.
Edit: Thank you for confirming in the comment. Since comments are not always permanent on Stack Overflow, I quote here, you said:
Finally got confirmation from the source: it's homespun indeed.
As an aside I don’t like it. A float is stored in a binary format internally, and only after formatting it into decimal does it become human readable. Using a float for a “human readable” date and time was not what formatting of floating-point numbers was meant for, it’s a hack.
Use ISO 8601
For a human-readable format I recommend ISO 8601. Here 2019-09-18, 11:29:30am becomes 2019-09-18T11:29:30. Or even better and still within ISO 8601, convert to UTC and append a Z to denote UTC. So if your original time was in Europe/Berlin time zone, it would become 2019-09-18T09:29:30Z. As you can see, ISO 8601 is even more human readable than you friend’s format, and it is sortable as strings (as long as the years don’t go beyond 9999).
While he may have come up with it himself, it is also a formatting option in zipinfo.
The manual doesn't explicitly name it, but describes it as a sortable decimal format and decimal format.
Not sure if we are talking about SQL date format. If so, this date format is present in SQL Statements.
Not sure about the name, it's called in different ways: non-standard, ISO, Other format and so on.
Is present also in PHP.
According to Wikipedia, this would be similar to the ISO 8601, which permits, all of the following for date and time combined:
2019-09-18T09:18:26+00:00
2019-09-18T09:18:26Z
20190918T091826Z
except that the T to separate the time from the date is replaced by . and the time-zone information is dropped.
That specific format has limited popularity either in the yyyymmdd.HHMMSS or the C's strftime()-compatible %Y%m%d.%H%M%S form.
EDIT
As far as using float for date and time the way you suggests, it depends on the precision and machine representation.
If the system is following IEEE 754 basic standard (which is what most modern C compiler stick to), you would need at least float64.
However, it is not common to do so.
This might be in part because it may be difficult to correctly predict the accuracy of the time information, and it is not as bit-efficient as the Unix time.
Given that the only positive feature it has is that it can rely on standard %f from sprintf(), I would only see it advantageous when strftime() is not available or a performance bottleneck.
I have a node.js application that stores many dates in a database. They are stored in the ISO format, such as '2016-11-02T16:30:12-04:00'.
Some fields which are dates are just dates, other are date/times. An example of a date/time would be "last modified" for a record, where a person's birthday is just a date.
The question is about best practices for storage and query patterns on these things. Because a date always has a time, you must choose how to store for example a birthday. Following the 5 laws of API dates and times this is of course done in UTC.
There are edge cases though where proper API behavior seems unclear. Suppose someone submits a birthdate to the API of '2016-11-02T16:30:12-04:00'. This is bad news, because a search like /users?birthdate=2016-11-02 will fail, as that date will get converted to '2016-11-02T00:00:00Z' and fail to match in the DB. What then should correct behavior be?
When someone POSTs a user, convert date fields into dates at midnight UTC, and then have the convention that querying birthdates should assume the same?
Convert date queries for certain fields into implicit ranges, i.e. searching for 2016-11-02 is really looking for 2016-11-02T00:00:00Z <= x <= 2016-11-02:23:59:59Z?
Match only on the exact moment, and rely on the client to know that a birthday of '2016-11-02T16:30:12-04:00' really means 4:30PM EST, and does not mean just on November 2nd?
What's the established pattern / best practice here for distinguishing between dates and datetimes?
I have been studying REST best practices and standards a lot for a while and I can't recall reading anything about that, but for the usage of ISO standard. From your description it seems to be something that really depends on the application and its use-cases.
I would go for your option #2: if a GET request comes with a date but no time, consider it a query for the whole day, and do the "conversion" in your GET response server code. Maybe you'd want to support both a "date" and a separate "time" query string parameters if the precise time might matter occasionally. This can also help you to keep clients "unaware" of the database storage format you choose, and may even allow you to support localized date formats.
The problem here is the usage of UTC, which implies that there's a time associated with it. There's not, a birthdate is considered (in iCalendar) a 'floating date' and does not have a specific time associated with it.
If your birthdate is November 3rd, and you move to Australia, your birthdate does not actually change to November 2nd, because your birthdate does not have a time, does not have a timezone and is the same where ever you are in the world.
The solution is simple. If you allow users to submit a date/time for birthday searches, then you should just 'cut off' the time and timezone. Assume that you're only going to be using the date portion and just search your database based on that.
Ideally you don't allow users to submit a time at all though. I think this just creates confusion. Just force api clients to submit a date only.
Those '5 laws' are an extreme over-simplification and don't apply to many situations.
Im currently developing an app that can parse dates from an email - i.e extract the time and dates from an email (similar to gmail).
Currently I do this in php but this is a tad clunky.
Whats the best language to do this in and are there any existing open source solutions?
I think PHP is as capable as any other language. Can we see the code you're using so we can suggest improvements? I'd use a regular expression... you just need a good one that supports a variety of formats.
What I do in my email client is extract all the tokens delimited by whitespace and then iterate over them using heuristics to decide how to classify each token. For instance if the token has a ':' character in it then I treat it as a time, to be parsed as ##:##:##. If it has '.' or '-' treat it as a day/month/year combo, and you have to decide which end is which... could be any number of combinations. If the token starts with a letter (i.e. isalpha(*string)) then you do a month name lookup. If it's a number it could be the day or year... decide based on length and whether you have an existing day or year already etc. If the token starts with '-' or '+' then it's a timezone, parse accordingly.
Seems to work in the field quite well, my email client has been around for 10 years or so. My code is C++, but you can write the same in PHP easily, it's not particularly language specific.
if you mean the date it was sent (or received), you are retrieving them from the mail headers (for example 'Date:' header) and they have a standard date format, see the RFC 2822
Anyway, if you use javamail (it's open source now) you can get the sent date with
Date sentDate = mail.getSentDate();
I need to get the index of a exchange like NASDAQ rather than the price of a specific stock in that exchange. I suppose that Finance::Quote will come to the rescue , but after a quick go-through of the document, I find it the way one can use the module for query is like:
%info = $q->fetch("australia","CML")
which means both the exchange and the stock should be specified in the query. then the question is: does the index itself can be treated as a stock and has a symbol name which can be used in the query?
Of course, if you have other way can meet my needs rather than using Finance::Quote, please feel free to write down your solution.
The problem with your question is that you are assuming that there is just one index for a particular exchange. Whilst there may well be a particular index that is dominant (eg. for stocks primarily traded on the London Stock Exchange, the FTSE 100 might be considered the main index; similarly for the NYSE it would be the Dow Jones Industrial Average) other exchanges may have a less clear leader in their collection of associated indicies (eg. for the Australian Stock Exchange, the S&P/ASX 200 and the All Ordinaries index are both frequently quoted side-by-side in the evening broadcast news).
Symbology of stocks, indicies, option chains, futures, etc is quite a complicated field in financial IT. Many of the symbology standards are backed by a data vendor (eg. Reuters, Bloomberg) and use of their standards requires a commercial license. On the other hand there are other efforts aiming to make symbology more open (Bloomberg themselves are behind one of these efforts).
I'm not familiar with the data sources of the Finance::Quote package you reference, but if you are serious about accessing market data (ie. prepared to pay for it) but don't need the cost/complexity/speed of a solution from Reuters, Bloomberg, etc, you could do alot worse than check out what Xignite offers in the way of market data accessible via web services.
the symbol for the nasdaq composite is "^IXIC". For nyse composite it's "^NYA".
each quote provider might have a different syntax though.
Has anyone found a simple, but effective way to extract date references from text? I've done a fair amount of searching for temporal extraction tools, but there isn't a lot out there. There are a few white papers, but it seems to fall into a subset of the whole semantic web thingy but not given much attention.
I'm just looking for something that is 80% effective. There is no need to capture things like "the month after Jan 2009", but basic common dates entities would be nice.
I'm open to all suggestions, even fancy regex expressions.
Fire away!
(and thanks - Henry)
If the target temporal expressions in your data are only in limited format, use regular expression and iterative approach to refine your system
Otherwise, use Stanford NLP toolkit, SUTime, which might be an over-kill but definitely meet your demands
One way I have done this is to just look for anything that is 4 numbers and convert it to a number. If the number falls within the range of years you are interested in, you probably have a year you can use. If you are interested in any matching months and days you could check adjacent words to see if they are a month name or a number between 1 and 31. I am confident this would satisfy your 80% requirement.
Regex for years: [0-9]{4} - you will need to convert to a number and see if it's within the range of years you consider valid.
Regex for months: jan|january|feb|february ... etc for each month
Regex for days of the month: [0-9]{1,2} - you would need to convert to a number and see if it is 1-31
I'm drawing a blank on how to find what to feed it, but this library will parse a wide range of dates and could be used as the "is this a real date" function. (Full disclosure, I'm the author of that lib)