I have file with a lot of data, for example here is a file with this data ->
time: 1 2 3 4 5 6 7 8 9 10
data: 1 0 1 1 1 1 1 0 1 0
but in my file I have dropped the multiple data going in sequence like this ->
time: 1 2 3 8 9 10
data: 1 0 1 0 1 0
If I run this data the result is this ->
My question is how to achieve result like in this picture shown with red arrows.
Simplier, how to repeat the value in not defined time steps (4,5,6,7 example above)
you can achive this by not droping the final (7 in this example) like this:
time: 1 2 3 7 8 9 10
data: 1 0 1 1 0 1 0
this way simulink will interpolate ones there.
Related
I have a date-level promotion data frame that looks something like this:
ID
Date
Promotions
Converted to customer
1
2-Jan
2
0
1
10-Jan
3
1
1
14-Jan
3
0
2
10-Jan
19
1
2
10-Jan
8
0
2
10-Jan
12
0
Now I want to see what were the number of promotions it took to convert someone into a customer
For eg., It took (2+3) promotions to convert ID 1 to the customer and (19) to convert ID 2 to the customer.
Eg.
ID
Date
1
5
2
19
I am unable to think of an idea to solve it. Can you please help me?
#Corralien and mozway have helped with the solution in Python. But I am unable to implement it in Pyspark because of the huge dataframe size (>1 TB).
You can use:
prom = (df.groupby('ID')['Promotions'].cumsum()
.where(df['Converted to customer'].eq(1))
.dropna().astype(int))
out = df.loc[prom.index, ['ID', 'Date']].assign(Promotion=prom)
print(out)
# Output
ID Date Promotion
1 1 10-Jan 5
3 2 10-Jan 19
Use one groupby to generate a mask to hide the rows, then one groupby.sum for the sum:
mask = (df.groupby('ID', group_keys=False)['Converted to customer']
.apply(lambda s: s.eq(1).shift(fill_value=False).cummax())
)
out = df[~mask].groupby('ID')['Promotions'].sum()
Output:
ID
1 5
2 19
Name: Promotions, dtype: int64
Alternative output:
df[~mask].groupby('ID', as_index=False).agg(**{'Number': ('Promotions', 'sum')})
Output:
ID Number
0 1 5
1 2 19
If you potentially have groups without conversion to customer, you might want to also aggregate the "" column as indicator:
mask = (df.groupby('ID', group_keys=False)['Converted to customer']
.apply(lambda s: s.eq(1).shift(fill_value=False).cummax())
)
out = (df[~mask]
.groupby('ID', as_index=False)
.agg(**{'Number': ('Promotions', 'sum'),
'Converted': ('Converted to customer', 'max')
})
)
Output:
ID Number Converted
0 1 5 1
1 2 19 1
2 3 39 0
Alternative input:
ID Date Promotions Converted to customer
0 1 2-Jan 2 0
1 1 10-Jan 3 1
2 1 14-Jan 3 0
3 2 10-Jan 19 1
4 2 10-Jan 8 0
5 2 10-Jan 12 0
6 3 10-Jan 19 0 # this group has
7 3 10-Jan 8 0 # no conversion
8 3 10-Jan 12 0 # to customer
you want to compute something by ID, so a groupby ID seems appropriate, e.g.
data.groupby("ID").apply(fct)
Now write a separate function agg_fct which computes the result for a
dataframe consisting of only one ID
Assuming data are ordered by Date, I guess that
def agg_fct(df):
index_of_conv = df["Converted to customer"].argmax()
return df.iloc[0:index_of_conv,df.columns.get_loc("Promotions")].sum()
would be fine. You might want to make some adjustments in case of a customer who has never been converted.
I have tried using the following function to derive a table consisting of 3 columns with one column data holding a list of an arbitrary schema.
fn:{
flip `time`data`id!(x`b;(x`a`b`c`d`e);x`a)
};
which works well on input with multiple rows i.e.:
q)x:flip `a`b`c`d`e!(5#enlist 5?10)
q)fn[`time`data`id!(x`b;(x`a`b`c`d`e);x`a)]
time data id
-----------------
8 8 5 2 8 6 8
5 8 5 2 8 6 5
2 8 5 2 8 6 2
8 8 5 2 8 6 8
6 8 5 2 8 6 6
However fails when using input with a single row i.e.
q)x:`a`b`c`d`e!5?10
q)fn[`time`data`id!(x`b;(x`a`b`c`d`e);x`a)]
time data id
------------
8 7 7
8 8 7
8 4 7
8 4 7
8 6 7
which is obviously incorrect.
One might fix this by using enlist i.e.
q)x:enlist `a`b`c`d`e!5?10
q)fn[`time`data`id!(x`b;(x`a`b`c`d`e);x`a)]
time| 8
data| 7 8 4 4 6
id | 7
Which is correct, however if one were to apply this in the function i.e.
fn:{
flip enlist `time`data`id!(x`b;(x`a`b`c`d`e);x`a)
};
...
time| 2 5 8 7 9
data| 2 5 8 7 9 2 5 8 7 9 2 5 8 7 9 2 5 8 7 9 2 5 8 7 9
id | 2 5 8 7 9
Which has the wrong format of data values.
My question here is how might one avert this conversion issue and derive the same field values whether the argument is a multi row or single row table.
Or otherwise what is the canonical implementation of this in kdb+/q
Thanks
Edit:
To clarify: my problem isn't necessarily with the data input as one could just apply enlist if it is only one row. My question pertains to how one might use enlist in the fn function to make single row input conform to the logic seen when using multi row tables. i.e. how to replace fn enlist input with fn data (how to make the function input agnostic) Thanks
Are you meaning to flip the data perpendicular to the rest of the table? Your 5 row example works because there are 5 rows and 5 columns. The single row doesn't work due to 1 row to 5 columns.
Correct me if I'm wrong but I think this is what you want:
fn:{([]time:x`b;data:flip x`a`b`c`d`e;id:x`a)};
--------------------------------------------------
t1:flip `a`b`c`d`e!(5#enlist til 5);
a b c d e
---------
0 0 0 0 0
1 1 1 1 1
2 2 2 2 2
3 3 3 3 3
4 4 4 4 4
fn[t1]
time data id
-----------------
0 0 0 0 0 0 0
1 1 1 1 1 1 1
2 2 2 2 2 2 2
3 3 3 3 3 3 3
4 4 4 4 4 4 4
--------------------------------------------------
t2:enlist `a`b`c`d`e!til 5;
a b c d e
---------
0 1 2 3 4
fn[t2]
time data id
-----------------
1 0 1 2 3 4 0
Note without the flip you get this:
([]time:t1`b;data:t1`a`b`c`d`e;id:t1`a)
time data id
-----------------
0 0 1 2 3 4 0
1 0 1 2 3 4 1
2 0 1 2 3 4 2
3 0 1 2 3 4 3
4 0 1 2 3 4 4
In this case the time is no longer in line with the data but it works because of 5 row and cols.
Edit - I can't think of a better way to convert a dictionary to a table when needed other than using count first in a conditional. Note if the first key is a nested list this wouldn't work
{ $[1 = count first x;enlist x;x] } `a`b`c`d`e!til 5
Note, your provided function doesn't work with this:
{
flip `time`data`id!(x`b;(x`a`b`c`d`e);x`a)
}{$[1 = count first x;enlist x;x]} `a`b`c`d`e!til 5
I want to plot a 3D histogram with Gnuplot using a style commonly used in Matlab.
The sequence of steps I follow are:
set palette defined ( 0 '#000090',\
1 '#000fff',\
2 '#0090ff',\
3 '#0fffee',\
4 '#90ff70',\
5 '#ffee00',\
6 '#ff7000',\
7 '#ee0000',\
8 '#7f0000')
set pm3d at s
set view map
splot 'test.dat' u 1:2:3
The data to make the plot is provided below. The resulting plot looks like this:
As you may see in the data below, most of the entries are zero which makes the plot very blue. In Matlab one can make this type of histograms and zero values have a white color. Just the points which are not zero have a color as in the palette above. I would like to have those points in white because that would emphasize the actual sampled region.
I wonder if we can do that in Gnuplot. I tried omitting the zeroes in the data file but it resulted in a pointy-corners plot.
Additionally, I modified the palette defining the zero value explicitly as here:
set palette defined ( 0 '#ffffff',\
1 '#000090',\
2 '#000fff',\
3 '#0090ff',\
4 '#0fffee',\
5 '#90ff70',\
6 '#ffee00',\
7 '#ff7000',\
8 '#ee0000',\
9 '#7f0000')
However, the borders of the sampled area look color violet:
Thanks.
1 1 1
1 2 0
1 3 0
1 4 0
1 5 0
1 6 0
1 7 0
1 8 0
2 1 0
2 2 0
2 3 1
2 4 2
2 5 3
2 6 0
2 7 0
2 8 0
3 1 0
3 2 0
3 3 2
3 4 10
3 5 15
3 6 2
3 7 0
3 8 0
4 1 0
4 2 0
4 3 0
4 4 5
4 5 2
4 6 1
4 7 0
4 8 0
5 1 0
5 2 0
5 3 0
5 4 3
5 5 2
5 6 0
5 7 0
5 8 0
6 1 0
6 2 0
6 3 0
6 4 2
6 5 0
6 6 0
6 7 1
6 8 0
7 1 0
7 2 0
7 3 0
7 4 0
7 5 0
7 6 0
7 7 0
7 8 0
8 1 0
8 2 0
8 3 0
8 4 0
8 5 0
8 6 0
8 7 0
8 8 0
Plotting with pm3d averages the data points. If you want to plot exactly the data points as matrix, you must plot with image. To have certain values in white, define them as undefined with 1/0:
set palette defined ( 0 '#000090',\
1 '#000fff',\
2 '#0090ff',\
3 '#0fffee',\
4 '#90ff70',\
5 '#ffee00',\
6 '#ff7000',\
7 '#ee0000',\
8 '#7f0000')
plot 'test.dat' u 1:2:($3 == 0 ? 1/0 : $3) with image notitle
Update
I just saw the answer above using image (I'll upvote that one), and that is probably the best way to accomplish what you asked for. In any case, I made a few tests with the data file and wanted to share just for completeness:
set term png
set out "tmp.png"
set palette defined ( 0 '#ffffff',\
1 '#000090',\
2 '#000fff',\
3 '#0090ff',\
4 '#0fffee',\
5 '#90ff70',\
6 '#ffee00',\
7 '#ff7000',\
8 '#ee0000',\
9 '#7f0000')
set xrange[0:8]
set yrange[0:8]
set pm3d explicit at s
set view map
set multiplot layout 2,2
splot 'test.dat' u 1:2:3 not w pm3d
splot 'test.dat' u 1:2:3 not w p pt 5 ps 5 pal
splot 'test.dat' u 1:2:($3==0 ? 1/0:$3) not w pm3d
Resulting in this:
I am trying to add/sum/merge two MAT files. Both the files contain only numeric values. Basically like this:
'file_1' contains
2 0 2 0 1 0...
3 0 9 0 2 0...
1 0 6 0 7 0...
...
'file_2.mat' contains
0 1 0 9 0 7 ...
0 5 0 5 0 8 ...
0 9 0 1 0 2 ...
...
i.e. in both files every alternate columns are zero. I want to merge them and form like this:
2 1 2 9 1 7...
3 5 9 5 2 8...
1 9 6 1 7 2...
...
and save this as a new mat file, 'file_3.mat'. And write this new file as an Image. How to do it?
Assuming that A contains the first matrix and B contains the second, you just need to do:
C = A + B
Example:
% create dummy values
A = magic(5);
A(:,1:2:5) = 0;
B = magic(5);
B(:,2:2:5) = 0;
C = A + B
This will output:
C =
17 24 1 8 15
23 5 7 14 16
4 6 13 20 22
10 12 19 21 3
11 18 25 2 9
I have this matrix and want to make all combinations of column composed square matrixes (8x8) composed from this data.
4 2 4 3 2 3 3 2 8 4 9 7 6 6 6
2 0 4 1 0 3 0 8 5 0 9 3 7 7 1
2 1 2 1 1 3 1 4 5 2 4 2 6 6 3
0 0 2 2 1 2 3 9 1 1 4 4 4 4 6
4 0 1 0 4 2 3 1 8 1 3 0 5 5 7
3 1 4 0 0 1 0 2 6 2 9 1 2 2 0
1 2 1 4 0 3 4 1 3 4 3 9 7 7 9
2 0 0 4 0 0 3 1 5 0 1 9 1 1 7
Even after reeding Matlab Loop of all combinations
I'm not really sure how to do all the matrix combinations and include the counter from the for loop in the name of the combination obtained in the itteration.
I called your matrix A.
p=nchoosek(1:15,8);
gives all the combinations of 8 numbers taken from 1 to 15. These represent the columns of the matrix A that you want.
There are now 3 ways to proceed. Firstly, using a for loop:
M=zeros(8,8,size(p,1));
for i=1:size(p,1)
M(:,:,i)=A(:,p(i,:));
end
which puts each 8x8 matrix into a larger 3D array. You would get out individual matrices by doing M(:,:,54), for example.
You can also create a cell array:
N=arrayfun(#(k) A(:,p(k,:)),1:size(p,1),'UniformOutput',false);
and get individual matrices by doing N{54}.
Finally, you could not precompute each matrix, and just pull out the appropriate columns when you need them. This may be the most efficient method if you don't reuse the matrices:
O=A(:,p(54,:));