I am getting this issue while storing the data in influxdb. The logs are generating as data is stored but when it comes to influxdb I am unable to find the data in the measurement. I thought it may be due to database size limit but for other measurements the data is getting inserted and no issue with them. So, if you have any idea about how to check measurement size or any more ideas to solve this issue.
Thanks in advance.
If you are on InfluxDB v1.x, you could use influx_inspect's report-disk.
If you are on InfluxDB v2.x, you take advantage of the internal stats as following:
from(bucket: "yourBucket")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "storage_shard_disk_size")
|> filter(fn: (r) => r["_field"] == "gauge")
|> last()
I have an InfluxDB 4.5.0 database running on Home assistant 2022.8.7.
I want to plot two InfluxDB queries on the same Grafana 7.6.0 graph, but one series is timeshifted by +24hrs.
After several hours of research I see it it possible to timeshift all the series on a Grafana panel using the "Query Options" but I can find no way to timeshift just one of the series.
I note that there is a timeshift function in InfluxDB but am stumped as to how I can modify the query in Grafana to timeshift this by +24hrs
As an example, if the series I want to timeshift is given by the query
SELECT mean("value") FROM "kWh" WHERE ("entity_id" = 'energy_tomorrow') AND time >= now() - 7d and time <= now() GROUP BY time(5m) fill(linear)
is there anyway to modify this query to timeshift the result by +24h, or alternatively what other method is available to achieve this basic result in Grafana with InfluxDB ?
Thanks in advance.
After many hours of trying I've finally found a solution, and post for any others with the same problem.
There are two ways of connecting influxDB to Grafana on home assistant
InfluxQL : which is an SQL like query language and the default connection method
Flux : which is InfluxData’s functional data scripting language
It does not seem possible to timeshift with (1), but there is a function in (2) which allows timeshifting. So the solution is to add a new datasource in grafana, through the UI
Configuration : Data Sources : Add Data Source : InfluxDB
Give this new datasource a recognisable name (eg FluxQuery) and configure this datasource to use Flux instead of the default InfluxQL. Then when adding a new panel in grafana, providing you select the appropriate datasource (eg FluxQuery), then flux querying is enabled and timeshifting possible.
As an example, if the InfluxDB database has the name HomeAssistant which includes two database entity_id's called energy_today and energy_tomorrow. The following flux query plots energy_today timeshfited +24hrs so it overlays correctly on energy_tomorrow
Query A
from(bucket: "HomeAssistant/autogen")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "kWh")
|> filter(fn: (r) => r["entity_id"] == "energy_tomorrow")
Query B
from(bucket: "HomeAssistant/autogen")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> timeShift(duration: 24h)
|> filter(fn: (r) => r["_measurement"] == "kWh")
|> filter(fn: (r) => r["entity_id"] == "energy_today")
I have a dashboard in grafana (v 8.4.4) that uses InfluxDB with Flux query which looks something like this:
from(bucket: "landscape_sizing")
|> range(start: ${__from}, stop: ${__to})
|> filter(fn: (r) => r["_measurement"] == "old_snapshots")
If I select some range like Last 7 days or Last 90 days, I get no data in the dashboard. If I select absolute time rage and though with fixed dates/time data shows up. The fun part is this query used to work for a while, and as far as I'm aware there were no major changes on either Grafana or Influx side. Is there a way to check what the __from and __to variables in the query are interpred to?
You should be able to see that in Grafana, from the visual Inspect -> Query.
that should show you the actual query text
Not sure I got my title right, apologies.
Recently discovered influxdb 2.0 and grafana 7, vast improvement from previous version.
I wondered if something is possible to do, I have a system that posts to influxdb the time it took to do a task, is it possible to count the number of entries for the last 30 days or calendar month ideally and display it as a gauge or text on grafana?
the Flux syntax is not like anything I have seen before so no idea where to start and any obvious googling I have done doesn't seem to bare fruit
Could be I need to collect the data via python, work it out and post it to a new measurement, seems kludgey
Thanks
I finally figured this out
In order to count you need to group the data and then only keep the columns you need, for my table I grouped and kept the "serial" column, then ran a count on this. the Range -1mo is the previous calendar month
from(bucket: "provisioning")
|> range(start: -1mo, stop: now())
|> filter(fn: (r) => r["_measurement"] == "provision")
|> filter(fn: (r) => r["_field"] == "ttl")
|> group()
|> keep(columns: ["serial"])
|> count(column: "serial")
I'm writing a web app using Firestore that needs to be able to show "Today's Popular Posts" and I'm having trouble getting the queries right when considering users in different timezones.
The dates are stored in the DB as UTC 0, then adjusted to the current user's UTC offset in the client via Moment.js. This works correctly.
When adding a new post I use firebase.firestore.FieldValue.serverTimestamp() to store the current server timestamp in a field called timestamp, like so:
const collectionRef = db.collection('posts');
collectionRef.add({
name: "Test Post",
content: "Blah blah blah",
timestamp: firebase.firestore.FieldValue.serverTimestamp(),
likeCount: 0
});
Then on the server I have a Cloud Function that runs on create and adds another field to the document called datestamp which is the the UTC 0 timestamp, but adjusted so that the time is the beginning of the day. The function looks like this:
exports.updatePostDate = functions.firestore
.document('posts/{postID}')
.onCreate((event) => {
const db = admin.firestore();
const postRef = db.doc('post/'+event.params.postID);
const postData = event.data.data();
const startOfDay = moment(postData.timestamp).startOf('day').toDate();
return postRef.update({
datestamp: startOfDay
});
});
Storing a timestamp where the time is always the beginning of the day enables me to write a query like this for finding all posts and ordering by popularity on a given day:
const startOfDayUTC = moment.utc().startOf('day').toDate();
const postQuery = db.collection('posts')
.orderBy('likeCount', 'desc')
.orderBy('timestamp', 'desc')
.where('datestamp', '==', startOfDayUTC)
.limit(25);
The problem is, depending on the user's UTC offset, this can display posts with two different dates when parsing the post's timestamp field. So even though the query is correctly fetching all the posts where the datestamp is say, 2018-01-30T00:00:00Z, the timestamp's date might not be the same once parsed. Here's an example of two posts:
Post 2:
likeCount: 1
timestamp (UTC 0): 2018-01-30T06:41:58Z
timestamp (parsed to UTC-8): 2018-01-29T22:41:58-08:00
datestamp (UTC 0): 2018-01-30T00:00:00Z
Post 1:
likeCount: 0
timestamp (UTC 0): 2018-01-30T10:44:35Z
timestamp (parsed to UTC-8): 2018-01-30T02:44:35-08:00
datestamp (UTC 0): 2018-01-30T00:00:00Z
So you can see, while the posts have the same datestamp, after adjusting the timestamp to the local UTC, the timestamp fields can end up being on two different days.
If anyone has a solution to this I would be very grateful.
I think it is better to avoid functions in this case as you can perform compound queries now. You can simply use
query.where(date > lastMidnight).where(data < now).get().then(...)
so to speak to limit data which only belongs to one day and try to keep all your time variables in UTC 0 and just find the start point and the current time both client side and convert them to UTC0.
//get local time from midnight to now (local)
const now = new Date();
const lastMidnight = now.setHours(0,0,0,0);
//then convert those to UTC0 to pass on in your query to firestore
const lastMidNightUTC = new Date(lastMidnight + now.getTimezoneOffset() * 60000).toString();
const nowInUTC = new Date(now + now.getTimezoneOffset() * 60000).toString();
and you can get your data (remember you need to make an index or just run the query once and firebase SDK will generate a link to create the index in dev tools -> console , for you)
query.where(date > lastMidNightUTC).where(data < now).get().then(...)
I came up with a solution that I'm really not happy with... But it works!
The problem is fundamentally one post can be on more than one date, depending on the user's location. And since for this case we also want to order by a field other than timestamp we can't use a range query to select posts on a given date, because your first .orderBy must be on the field you're using a range query on (see Firestore docs).
My solution is to map localized datestamps to their corresponding UTC offset. The object contains every UTC offset as a key, and the post's datestamp in that offset's time.
An example post looks like this:
posts/{somepostid}
{
name: "Test Post",
content: "Blah blah blah",
timestamp: Mon Jan 29 2018 21:37:21 GMT-0800 (PST),
likeCount: 0,
utcDatemap: {
0: "2018-01-30,
...
-480: "2018-01-29",
...
}
}
The field utcDatemap is the the one we use in our queries now:
const now = moment();
const datestamp = now.format("YYYY-MM-DD");
const utcOffset = now.utcOffset();
const utcDatemapField = 'utcDatemap.'+utcOffset;
const postQuery = db.collection('posts')
.orderBy('likeCount', 'desc')
.orderBy('timestamp', 'desc')
.where(utcDatemapField, '==', datestamp)
.limit(25);
Now posts can show up on two different days, depending on where the user is querying from. And we can still convert the regular old timestamp to the user's local time on the client.
This is definitely not an ideal solution. For the above query I needed to create composite indexes for every single key in utcDatemap. I'm not sure what the rules of thumb are with composite indexes, but I'm thinking having 39 indexes for something simple like this is probably not great.
Additionally I checked it out using the roughSizeOfObject function from thomas-peter's answer on this post and the utcDatemap object, with all it's string datestamps clocked in at roughly 780 bytes, and it's not like 0.78kb is a lot, but you do need to be mindful of how much data you're transferring with a service like Firestore (0.78kb is a lot for a date).
I'm learning/reading up on Firestore and have Google'd to see how it deals with times, so discount my answer appropriately.
It looks as though Firestore converts times to UTC and stores them as its own Timestamp datatype. If so, then it's critical to know that this is a destructive conversion.
Though UTC is useful for comparing instants in time, it means that the wall-clock time as observed by the app user is lost forever. Some countries like the UK are in one of two timezones during the year, Daylight Savings Time and British Summer Time.
You can convert back to the user's observed time, but the problem is that the rules change over the years. You'd have to keep a record of all the different rule changes for all the timezones and countries of the world.
What was the time at the time?
The question is, what time did the user think an event happened ...at the time. This can have legal ramifications. You may need to go back through historic data to prove a person acted at a certain time as they observed it.
The solution is to capture the user's observed offset in an additional field. That way, you can always use this to convert.
Regarding the OPs problem, this seems somewhat philosophical for a web app. Does "today" mean the event, such as a post, must have happened within the user's Monday? Or just posts on today's date? Or posts within the last 24h?
An important thing to remember is that dates are the same all around the world, even when they begin and end at different instants.
What's Elvis got to do with all this?
Christmas Day is 25th everywhere. If I say something happened on Christmas Day and I'm in the USA, and then someone in Australia wants to see all the world's posts made on Christmas Day, then they need to query for posts where observedDate == 25th Dec.
Think about it. Such posts were all made on Christmas Day, even though it might have been Boxing Day for me in England at the instant that they posted.
Elvis died on 16th August. In the UK our radio stations don't all wait until it's the 16th in the timezone of the place of his death to start playing his records.
Another interesting one is whether something happened in Q1 or Q2 of a company's reporting year. Is a sale recognised as on the date at the point-of-sale in the store in New York, or in the database in LA?
The observed date is interesting.
My point is, think deeply about this and store both a normalised instant like UTC, but also the user's observed date, time and offset. Then you have all the information you'll need for the future.
Finally, consider adding observed year or week numbers, or day ordinals, H1/H2 for financial data, since it can be super useful for composing rapid queries, depending on your use-cases.