I have a list of observations where each data point is a pair of a time expression (e.g. night, morning) and an hour in a 12-hr clock (i.e. 1, 2, ..., 12): Y = {<e_i, h_i>}_i={1,...,N}. I would like to estimate the distribution of hours in a 24-hr clock given a time expression (or equivalently, classify each data point to AM or PM).
I have a feeling EM would be useful here given the hidden AM/PM variable, but I'm struggling to define the parameters. In all other examples I've used EM for, something is assumed about the distribution that generated the observations (e.g. that it is a normal distribution, or document classification based on bag-of-words). But I'm not sure how to define it here.
I'd appreciate any help!
I ended up solving it as an ILP problem:
I defined a binary variable for each combination of 12 hr and time expression (true if it is PM, false if AM), and start time and end time variables for each expression. My constraints were the order of time expressions, e.g. morning ends before noon starts, etc. I maximized the number of observations that fit within the start and end time for each expression.
I would like to use the day of the year in a machine learning model. As the day of the year is not continuous (day 365 of 2019 is followed by day 1 in 2020), I think of performing cyclic (sine or cosine) transformation, following this link.
However, in each year, there are no unique values of the new transformed variable; for example, two values for 0.5 in the same year, see figures below.
I need to be able to use the day of the year in model training and also in prediction. For a value of 0.5 in the sine transformation, it can be on either 31.01.2019 or 31.05.2019, then using 0.5 value can be confusing for the model.
Is it possible to make the model to differentiate between the two values of 0.5 within the same year?
I am modelling the distribution of a species using Maxent software. The species data is continuous every day in 20 years. I need the model to capture the signal of the day or the season, without using either of them explicitly as categorical variable.
Thanks
EDIT1
Based on furcifer's comment below. However, I find the Incremental modelling approach not useful for my application. It solves the issue of consistent difference between subsequent days; e.g. 30.12.2018, 31.12.2018, and 01.01.2019. But it does not differ than counting the number of days from a certain reference day (weight = 1). Having much higher values on the same date for 2019 than 2014 does not make ecological sense. I hope that interannual changes to be captured from the daily environmental conditions used (explanatory variables). The reason for my need to use day in the model is to capture the seasonal trend of the distribution of a migratory species, without the explicit use of month or season as a categorical variable. To predict suitable habitats for today, I need to make this prediction not only depends on the environmental conditions of today but also on the day of the year.
This is a common problem, but I'm not sure if there is a perfect solution. One thing I would note is that there are two things that you might want to model with your date variable:
Seasonal effects
Season-independent trends and autocorrelation
For seasonal effects, the cyclic transformation is sometimes used for linear models, but I don't see the sense for ML models - with enough data, you would expect a nice connection at the edges, so what's the problem? I think the posts you link to are a distraction, or at least they do not properly explain why and when a cyclic transformation is useful. I would just use dYear to model the seasonal effect.
However, the discontinuity might be a problem for modelling trends / autocorrelation / variation in the time series that is not seasonal, or common between years. For that reason, I would add an absolute date to the model, so use
y = dYear + dAbsolute + otherPredictors
A well-tuned ML model should be able to do the rest, with the usual caveats, and if you have enough data.
This may not the right choice depending on your needs, there are two choices that comes to my mind.
Incremental modeling
In this case, the dates are modeled in a linear fashion, so say 12 Dec, 2018 < 12, Dec, 2019.
For this you just need some form of transformation function that converts dates to numeric values.
As there are many dates that need to be converted to numeric representation, the first thing to make sure is that the output list also has the same order as Lukas mentioned. The easiest way to do this is by adding weight to each unit (weight_year > weight_month > weight_day).
def date2num(date_time):
d, m, y = date_time.split('-')
num = int(d)*10 + int(m)*100 + int(y)*1000 # these weights can be anything as long as
# they are ordered
return num
Now, it's important to normalize the numeric values.
import numpy as np
date_features = []
for d in list(df['date_time']):
date_features.append(date2num(d))
date_features = np.array(date_features)
date_features_normalized = (date_features - np.min(date_features))/(np.max(date_features) - np.min(date_features))
Using the day, month, year as separate features. So, instead of considering the date as whole, we segregate. The motivation is that maybe there will be some relations between the output and a specific date, month, etc. Like, maybe the output suddenly increases in the summer season (specific months) or maybe on weekends (specific days)
I am looking to calculate the highest precision lat lon for the subsolar point, in a particular datetime moment, as is reasonably possible using pyephem, with the help of some other library(s) if they are needed.
Relevant context:
Anyone who has used pyephem, already knows that for certain calculations it requires certain setup values before computing body positions, those values including the datetime (epoch of the observation), the location of the observer, and of course, the body being investigated. Solutions for the subsolar point through the use of pyephem, that I have found online, show the time in utc as the time needed for the pyephem setup.
Remembering way back to my first exposure to astronomy, and to celestial navigation, utc is a variant of a mean day, compared to an actual solar day, where an actual solar day's duration throughout the year varies due to several factors of the nature of the earth's orbit. Because the length of an actual solar day varies throughout the year, for certain types of astronomical calculations, this requires the Equation of Time to more precisely map the actual solar day measurements to a mean and fixed 24 hour day system such as utc. Before the advent of sufficiently accurate 'pendulum movement' clock mechanisms, and now crystal controlled clock mechanisms, going back to when sundials were the accurate timepiece, the more sophisticated sundials included markings to apply a yearly approximation of this important Equation of Time, soon after it had been observed and definitively documented. Therefore, relevant to my question, since utc is a variant of mean day, and not the true solar day, but normalized to 24 hours exactly, there is this question now of how or if pyephem incorporates the Equation of Time in its right ascension solutions for the sun. At present, I imagine the EoT is required for accuracy, as I try to visualize the sun's position against the background of stars, as seen from the earth, as the earth revolves around the sun, with historically observed variations that are made available and useful and essential with the Equation of Time.
Summary then of my question:
If it is not necessary to explicitly enter an EoT value in pyephem, because it is not relevant for computing the most accurate subsolar point, please explain why. If it is relevant, as I presently think it is, please tell me if pyephem, in its right acension calculation of the sun (and other bodies), as a body, does in fact, apply the Equation of Time as appropriate. Does it do so transparently? Is there a way to input an explicit value for it, if such is known, an EoT value that might be more accurate or more up to date compared to what pyephem is using transparently?
Some initial research results that formed the question:
Upon doing a search through various search engines, I found several posts in topical forums that give what seems a very simple answer for finding the subsolar point. Finding the lattitude apears to be the less complicated part of the solution, being simply the computed declination. Finding the longitude is where the question arose in my thinking, and now I wonder if it is applicable for the declination as well, since using the properly precise time is essential for the most precise result of both declination(lat) and longitude of the subsolar point. I always applied the EoT from the Nautical Almanac, back when I was involved with celestial navigation.
Two links, specific to pyephem, present the same approach to the subsolar point solution. When the question(s) was first asked, Brandon Rhodes quickly presented the single line formula using pyephem's computing of the sun's right ascension. His was specifically the code for the longitude calculation in a more theoretical tone, without all the pyephem contextual details. Liam Kennedy presented a more complete context of python code, showing those additional pyephem details, so that one could 'copy and paste' the entire block of code, (needing only to add the import ephem and import datetime), and modify it as appropriate, which I also found to be a useful review. The code is from these links...
Computing sub-solar point
Confusion with using dec/ra to compute sub-lunar location
subsolar point:
Brandon's code
lon = body.ra - greenwich.sidereral_time()
Liam's code
sunLon = math.degrees(sun.ra - greenwich.sidereal_time() )
Nowhere in these two posts is there a mention of the Equation of Time, and yet a variant of mean day is being used as an input value here
greenwich.date = datetime.utcnow()
utc as a variant of mean day is EoT unaware, by its construction definition as a mean day, which then normally makes it a requirement to adjust it with the EoT for certain astronomical usefulness.
To further clarify this requirement, there are many navigation and astronomical references that go into considerable detail discussing it. But I will stick to refering to some forum posts such as the following:
https://forum.cosmoquest.org/showthread.php?55871-Finding-the-subsolar-point
specifically the post by grant hutchison 2007-mar-20, 04:33 pm
You can use the NOAA Solar Position Calculator, but it's kind of convoluted.
http://www.srrb.noaa.gov/highlights/sunrise/azel.html
note: the NOAA calculator, as of this writing, 2019-12-19, does have an input box where one is to enter the Equation of Time in minutes. That page has a link to a more updated calculator.
https://www.esrl.noaa.gov/gmd/grad/solcalc/
The more up to date page also calculates and displays the Equation of Time, clarifying its relevance. Now, continuing to quote Grant's post...
First, use the calculator to derive the Equation of Time and Solar Declination for the date and time you're interested in, at the location zero latitude and zero longitude, with no UTC offset.
The 2007 March equinox is at 21 March 00:08:30 UTC. Type that time and date into the calculator and, sure enough, you find the solar declination is zero: the sun is over the equator at that moment. For any other date and time, the solar declination will convert directly to the latitude of the subsolar point.
Now we need the longitude. First, work out the true solar time using the Equation of Time figure: it's -7.42 minutes in this case. That's the offset between the position of the mean sun and the real sun. Adding that figure to our UTC time tells us that the real sun is just 1.03 minutes past midnight (8.5-7.42) at the time of interest. Divide that figure by 60*24 (to get the fraction of a day) and multiply by 360 (to get degrees): that gives us 0.2575 degrees past midnight. So the sun will be on the noon meridian at 180-0.2575 degrees east = 179.7425 E. That's our longitude.
Combine the two, and the subsolar point is 0.0000N 179.7425E.
We can check that I haven't mixed my pluses and minuses by typing the derived coordinates of the subsolar point into the solar calculator (Lat 00:00:00, Lon -179:44:33), keeping the UTC offset at zero and the date and time at your time of interest, 21 March 00:08:30. That comes up with an Azimuth of zero and an Altitude of 89.98 degrees, confirming that we have the sun crossing the meridian within a couple of hundredths of a degree of directly overhead. Phew. It works, but it's a bit of a pain. Maybe someone can offer a calculator that will do more of the work for you.
And a followup post of his dated about an hour and a half later...
Some notes to the above, FWIW:
The difference between Dynamical Time and UTC this year is 65 seconds, so working from the Dynamical Time of the solstice we get the UTC time (to the nearest second) to be 00:07:25 UTC, which fits with G O R T's nearest-minute value, above.
The reason G O R T and I come up with a different subsolar longitude for the same time (00:07:00 UTC) is because of that pesky -7.42 minutes in the equation of time: although that time is after midnight at Greenwich, the real sun is still 42 seconds short of crossing the midnight line. That shifts the calculated subsolar point from the eastern to the western hemisphere. 7.42 minutes is equivalent to 1.855 degrees, which is exactly the difference between my calculated longitude of 179:53:42W and G O R T's of 178:15:00E.
My question is therefore based on this research, and based on my past experience with celestial navigation. I imagine that as vital as the Equation of Time might be to the problem, it would be incorporated into pyephem's calculation(s), since a mean day is input into pyephem's API. Seeing nowhere in these snippet solution postings where EoT is to be specified in the pyephem API, my assumption is that it would be internally and transparently implemented? I am not comfortable with this assumption, and so I have posted this question. Clarification would benefit the confidence of users, particularly newbies such as myself.
Update 12-20-2019:
I suspect the answer is yes, pyephem accounts for EoT, but it does not call it that? The way ephem, libastro, takes into account some other effect or relationship probably answers my question(s). I am reviewing:
https://rhodesmill.org/pyephem/radec
Needing to read it very slowly, while drawing some pictures, and waiting for an astronomy book so I can catch up on a very much misplaced education on this matter. I'm thinking that perhaps the term Equation of Time only has meaning in a narrow context of reconciling the solar day to a mean day metric, as experienced on earth, while pyephem solves in a broader context and uses more broadly applicable terminology, of which I need to be re-educated, which includes such resulting effects as the Equation of Time? Or I am only displaying my ignorance? Until I can more competently write my own answer, please do contribute any helpful comments or answers that can steer my study.
I think that your question, stated more briefly, is: does the libastro library that underlies PyEphem assume that the Earth’s orbit is a circle along which the Earth travels at a uniform rate? Because if it assumes a circular orbit and uniform rate for the Earth, then a correction — the Equation of Time — would need to appear for the fact that the Earth in fact varies its speed along its orbit.
I suggest that you can answer this question for yourself experimentally. If PyEphem assumes uniform circular motion for the Earth, then the number of degrees traveled by the Sun each day will be the same. Try looping over a long series of days. For the same time each day, ask the Sun for its right ascension and declination, and then use separation() to check the angle traveled between those points.
If the angle traveled by the Sun is the same each day, then PyEphem is modeling the Sun’s motion very poorly and you will need to apply an Equation of Time correction to get its true position.
But if the daily angle is varying — small in July, large in January — then PyEphem must be modeling the Earth’s motion more accurately. If you dig into the source code, you will find that its model is called the VSOP87 model of predicting where the Earth and Sun are. Your own experiments should show how the model behaves as the Sun travels the sky through the year.
I was unable to find the reference system that pyephem uses. I assume it is ICRS (International Celestial Reference System), is that correct?
The C library that PyEphem wraps was written in the 1990s, before the creation of the ICRS reference system, and therefore PyEphem only knows about the traditional equatorial coordinate system. The most details about the coordinate systems are at the page:
http://rhodesmill.org/pyephem/coordinates.html
You can see that equatorial coordinates created manually, as with other coordinates returned by the API, follow the rule that “If you do not specify an epoch, then J2000 is assumed.” So there is no concept within PyEphem of stepping outside all epoch-based coordinate systems and using an epoch-neutral system like ICRS instead.
We have following function is exist
j = ephem.Jupiter('1612/12/28')
n = ephem.Neptune('1612/12/28')
print j.ra, j.dec, j.mag
11:48:20.52 2:41:13.6 -1.96
My requirement is little different
I would like to know date by supplying angle is it possible?
Please reply.
Thanks
Harshad
There is no built-in function to, given an angle, find a date. Generally, when you have a problem like this that goes “backwards” from a circumstance that you need satisfied to the determination of the date on which that happens, you will need to try one date after another over and over again until you get out the angle that you are looking for.
There are several well-known techniques for finding the time at which a function returns a given angle, which are discussed here:
Using pyephem to calculate when a satellite crosses a Longitude
Are you able to imagine how you might apply the ideas in that answer to your own situation, since you are also looking for a particular angle (even though it comes from a different part of PyEphem)?