I have some smart meter data which shows gas and electricity meter readings at 30 min intervals for about two years, for 16000 households.
The date is stored in separate .mat files, with a datetime variable for the timestamp and a double variable for the actual data. Some of the data has gaps in, from a few hours to several days or weeks. I want to create a timeseries object containing all of the data and a continuous timestamp for the two year period, so that I can then interpolate the gaps.
Another option would be to use snychronize, but for this it seems the 16000 data series need to be in individual timeseries objects, which seems cumbersome.
I have tried this with timeseries objects and financial time series but cannot get all of the 16000 data series and corresponding timestamps into one time series object. When I try to add more than one series to an existing timeseries object, it is added "in series" rather than "in parallel" (i.e. data in the Data:1 column).
When I tried with a financial time series I had difficulties preparing the datetime data in a cell array.
Any ideas what the most efficient way to do this is?
Thanks
Russell
Depending on the version of matlab that you have the best idea would seem to be to use the table variable.
Tables can be used to store disparate objects so that you can have the date/time stamp as well as the meter readings in the same variable.
You can horizontally concatenate the tables (or otherwise join as you read them in so that you will now have a time series with a single date variable and the responses for each household.
Related
When I apply Month/Year to Cases or Deaths from my data, the values explode. For Cases it goes from approximately 48 million to over 1 billion, and for Deaths it goes from about 700 thousand to over 22 million. However, when I try the same thing with Initial Claims or the Stringency Index, my values remain correct. I'm trying to find the month over month percentage change by the way. And I'm using the Date column. I only select 2020 and 2021 in the filter for Year.
What I'm asking about is Sheet 21.
Link to workbook: https://public.tableau.com/app/profile/nilajah.rivers/viz/CoronaVirusProject_16323687296770/Sheet21
Your problem is that the data points are daily cumulative deaths. If you change the date aggregation to anything other than days, Tableau will default to summing the numbers for all the days in the month. This will give the wrong result, obviously.
If you want to show the correct total deaths or cases regardless of the time aggregation (months, days, weeks etc.) then you could use the New Case or New Death numbers plus a running sum table calculation. This will always give the correct total for the time period.
Table calculations will also allow automatic calculation of the period to period % change from the same data fields.
This is a common problem when working with datasets that offer pre-calculated aggregations. Tableau doesn't need that as it can dynamically calculate the aggregation of a field over any given time period but it is easy to forget which field has pre-aggregated data and which has raw data. Pre-aggregated fields assume a particular time period and can't be used for different time periods without disentangling that assumption (which is unnecessary if you also have the raw data (in this case daily new deaths/cases).
I have a report that uses a couple years of historical data and uses the Tableau forecasting to show expected values for the next three months. I would like to be able to save these values for a few reasons.
I need to do calculations on the forecasted values. Multiple different people will be using this forecast and will need different calculations. Some will need to calculate 70% of the forecasted value, and some will need to calculate different percentages. I'd like to have a parameter for them to enter the percent and be able to create a calculation using the parameter and the forecasted value.
I would like to be able to save forecasted values to later show the difference between what was forecasted and what actually happened. I understand you can export your data from your forecast and import that as another data source to show actuals vs forecast, but I need to do it all automatically with no manual intervention.
Is this possible? I haven't found any way to save the values or do calculations on the values that are forecasted. I am using Tableau 10.5
Tableau will never save data. Probably easiest to create the forecast values in the datasource and then report. It's very simple to have "actual" values and "forecast" values so you can compare
I have a Simulink block that calls a user-defined function. This function calls another function: thingspeakread.m (provided by the ThingSpeak Official Toolbox).
From here, I want two outcomes. Data (numeric type, vector), and timestamps (strings, vector).
[data,timestamps] = thingSpeakRead(___)
Now, I'm able to get the values of "Data" into a Scope without much of an issue (using coder.extrinsic('thingSpeakRead') and preallocating the variable to store such points). The issue I have is this: I need to make more sense of the data visualization, and for that purpose, I need to plot this "Data" against the information displayed in the timestamps vector. The timestamps varies per minute only.
What I'll try to do, but doesn't satisfies me a lot:
Convert the timestamp into a numeric value using datenum() using the format for 'mm/dd/yyyy HH:MM:SS'
This option sends back a huge number (which can be used to plot Data vs. Time indeed, but it doesn't look 'good' because the number is just too big and I don't like it).
I have thought about these too:
Convert date to Julian Date type.
Convert Hours, Minutes and Seconds into 3 different arrays.
But I don't see that getting me where I want (which is to plot Data vs. Time, being able to spot easily that the numbers displayed for "Time" are corresponding to an specific HH:MM:SS of a day).
Is there anything you can guys suggest, please? Thanks a lot in advance!
Edit 1: Can I use something like datetick() in Simulink?
What you are doing with converting to datenum is the only way to pass the "dates" down a Simulink signal.
There's no mechanism to display the time series, with dates displayed on the x-axis, without writing custom code.
If you don't need to display the time series as the simulation is running then just dump it to a mat file in your existing code, and generate the plot during post-processing.
If you want it to display as the simulation is running then you'll need to write a custom display block. This should be done as a Level-2 M-Code S-Function, but could be done using a MATLAB Function block. Either way you would input the datenum into the block and then convert the datenum back to a date - using something like datetime before generating the visualization, or afterwards using datetick.
I am using a MATLAB toolbox, specifically, https://uk.mathworks.com/matlabcentral/fileexchange/32882-armax-garch-k-sk-toolbox-estimation-forecasting-simulation-and-value-at-risk-applications
to insert data into the functions, the author defines a data matrix and then uses data(:,3) for the third column which represents a series.
I would like to do this put add data(:,3) lagged by one period.
My question: is there a way I can write something in Matlab that lags the dataset by one period which can be inserted into the function.
If I understand correctly, you would like to lag a series by one time period, with the time period being however you collect the data, for example, daily data, lag the series by one day.
If so you can use the lagmatrix
To provide an example,
LAGGEDX = lagmatrix(data(:,3),1)
This would lag your data(:,3) series by one day if it is daily data, you could then insert LAGGEDX in replace of data(:,3).
We have data that is submitted that is only YTD numbers. I'm wondering how I could display numbers that are subtracted along the Date field.
Ie, if I want to show the MTD movement on March. I will have to go March less February.
Now I know I can do this for individual measure fields. But having around 40+ measures seems a bit tedious.
http://kb.tableausoftware.com/articles/knowledgebase/creating-ytd-mtd-calculations
I tried to enter "Measure Values" but that is not a valid measure to put in the calculation.
Is there a way to set up a custom dimension?
Thanks,
Gem
After days of research, can't be done in tableau unless you want to labour for a week creating an almost cell by cell calculation. Data transformation in SQL will be a more feasible solution.
I had pivoted the data previously in SQL, so that I end up with 1 measure column instead of 40+. That enables you to minimise the calculation fields, so that you don't have to repeat all the calculation for individual measures.
Works well. Not for ratios though, as you will need to extract individual measures again so that you can divide them against each other. It's got pros and cons. Number of rows in the DB also multiplies.
Other solutions that preserves the table structure will be to use temp tables and do calculations on several temp tables.