SAS Placeholder value - merge

I am looking to have a flexible importing structure into my SAS code. The import table from excel looks like this:
data have;
input Fixed_or_Floating $ asset_or_liability $ Base_rate_new;
datalines;
FIX A 10
FIX L Average Maturity
FLT A 20
FLT L Average Maturity
;
run;
The original dataset I'm working with looks like this:
data have2;
input ID Fixed_or_Floating $ asset_or_liability $ Base_rate;
datalines;
1 FIX A 10
2 FIX L 20
3 FIX A 30
4 FLT A 40
5 FLT L 30
6 FLT A 20
7 FIX L 10
;
run;
The placeholder "Average Maturity" exists in the excel file only when the new interest rate is determined by the average maturity of the bond. I have a separate function for this which allows me to search for and then left join the new base rate depending on the closest interest rate. An example of this is such that if the maturity of the bond is in 10 years, i'll use a 10 year interest rate.
So my question is, how can I perform a simple merge, using similar code to this:
proc sort data = have;
by fixed_or_floating asset_or_liability;
run;
proc sort data = have2;
by fixed_or_floating asset_or_liability;
run;
data have3 (drop = base_rate);
merge have2 (in = a)
have1 (in = b);
by fixed_or_floating asset_or_liability;
run;
The problem at the moment is that my placeholder value doesn't read in and I need it to be a word as this is how the excel works in its lookup table - then I use an if statement such as
if base_rate_new = "Average Maturity" then do;
(Insert existing Function Here)
end;
so just the importing of the excel with a placeholder function please and thank you.
TIA.

I'm not 100% sure if this behaviour corresponds with how your data appears once you import it from excel but if I run your code to create have I get:
NOTE: Invalid data for Base_rate_new in line 145 7-13.
RULE: ----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8----+--
145 FIX L Average Maturity
Fixed_or_Floating=FIX asset_or_liability=L Base_rate_new=. _ERROR_=1 _N_=2
NOTE: Invalid data for Base_rate_new in line 147 7-13.
147 FLT L Average Maturity
Fixed_or_Floating=FLT asset_or_liability=L Base_rate_new=. _ERROR_=1 _N_=4
NOTE: SAS went to a new line when INPUT statement reached past the end of a line.
NOTE: The data set WORK.HAVE has 4 observations and 3 variables.
Basically it's saying that when you tried to import the character strings as numeric it couldn't do it so it left them as null values. If we print the table we can see the null values:
proc print data=have;
run;
Result:
Fixed_or_ asset_or_ Base_
Floating liability rate_new
FIX A 10
FIX L .
FLT A 20
FLT L .
Assuming this truly is what your data looks like then we can use the coalesce function to achieve your goal.
data have3 (drop = base_rate);
merge have2 (in = a)
have (in = b);
by fixed_or_floating asset_or_liability;
base_rate_new = coalesce(base_rate_new,base_rate);
run;
The result of doing this gives us this table:
Fixed_or_ asset_or_ Base_
ID Floating liability rate_new
1 FIX A 10
3 FIX A 10
2 FIX L 20
7 FIX L 20
4 FLT A 20
6 FLT A 20
5 FLT L 30
The coalesce function basically returns the first non-null value it can find in the parameters you pass to it. So when base_rate_new already has a value it uses that, and if it doesn't it uses the base_rate field instead.

Related

Sum in range until value change

I'am trying to use this formula to make it work
=ARRAYFORMULA(IF(ISDATE_STRICT(S2:S) ; (MATCH(MAX(AB2:AB),AB2:AB;0)-1) ; "" ))
If there is a date in Column "S" I want it to display the sum of the blanks that would appear if in Column "S" is text
=ARRAYFORMULA(IF(ISDATE_STRICT(S2:S) ; ArrayFormula(MATCH(FALSE ; ISBLANK(AB2:AB) ; 0)-1) ; "" ))
I've tried this one as well but I only get 0's as a result.
Any idea how I can make it work?
Here is the sample sheet.
https://docs.google.com/spreadsheets/d/19f5phXeAwXwrKbWz7njgbznmurOav72GUuo_5IGcbls/edit?usp=sharing
in Q2 use:
=ARRAYFORMULA(IF(ISBLANK(
I1:INDEX(I:I; ROWS(I:I)-1));
{N2:INDEX(N:N; ROWS(N:N))\
I1:INDEX(N:N; ROWS(N:N)-1)};
I1:INDEX(O:O; ROWS(O:O)-1)))
in X2 use:
=INDEX(LAMBDA(x; IFNA(VLOOKUP(x; QUERY(VLOOKUP(ROW(x);
IF(ISDATE_STRICT(x); {ROW(x)\x}); 2; 1);
"select Col1,count(Col1) group by Col1"); 2; 0)-1))
(Q2:INDEX(Q:Q; MAX((Q:Q<>"")*ROW(Q:Q)))))
UPDATE:
we start with column Q. we can take a range Q2:Q but that range contains a lot of empty rows. the next best thing is to check the last non-empty row and set it as the end of the range resulting in Q2:Q73. but static 73 won't do in case the dataset would grow or shrink so to get 73 dynamically we take the MAX of multiplication of Q:Q not being empty and row number of that case eg. Q:Q<>"" will output only TRUE or FALSE so what we are getting is
...
TRUE * 72 = 1 * 72 = 72
TRUE * 73 = 1 * 73 = 73
FALSE * 74 = 0 * 74 = 0
...
so the formula for getting Q2:Q73 is:
=Q2:INDEX(Q:Q; MAX((Q:Q<>"")*ROW(Q:Q)))
it could also be:
=INDEX(INDIRECT("Q2:Q"&MAX((Q:Q<>"")*ROW(Q:Q))))
but it's just long to type... next, we use the new LAMBDA function that allows us to reference cell/range/formula with a placeholder. simple LAMBDA syntax is:
=LAMBDA(x; x)(A1)
where x is A1 and we can do whatever we want with the 2nd (x) argument of LAMBDA like for example:
=LAMBDA(a, a+a*120-a/a)(A1)
you can think of it as:
LAMBDA(A1, A1+A1*120-A1/A1)(A1)
or as just:
=A1+A1*120-A1/A1
the issue here is that we repeat A1 4 times but with LAMBDA we do it only once. also, imagine if we would have 100 characters long formula instead of A1 so the final formula with lambda would be 300 characters shorter compared to "old way" formula.
back to our formula... x is the representation of Q2:Q73. now let's focus on VLOOKUP. basically, the idea here is that IF Q column contains a date we return that date, otherwise we return the last date from above. simply put:
=ARRAYFORMULA(VLOOKUP(ROW(Q2:Q73);
IF(ISDATE_STRICT(Q2:Q73); {ROW(Q2:Q73)\Q2:Q73}); 2; 1))
as you can see Y2, Y3 and Y4 are the same so all we need to do is to count them up and later take away one to exclude Q2 but include just Q3 and Q4 eg. 3-1=2. for that we use simple QUERY where the output is:
date count
30.06.2022 3
so all we need to do is to pair up dates from Q column to QUERY output for that we use the outer VLOOKUP where the output is as follows:
3
#N/A
#N/A
9
#N/A
#N/A
...
now is the right time for that -1 correction while we have these errors coz ERROR-1=ERROR and 3-1=2 so after this -1 correction the output is:
2
#N/A
#N/A
8
#N/A
#N/A
...
and all we need to do now is to hide errors with IFERROR and the output is column X

How do you merge lines in a single dataset with some duplicate values?

I am analyzing a medical record dataset where the patients were screened for STIs at 4 different times points. The data manager created a line per patient per STI for each time period. I want to merge the dataset so there is one line per patient at each time point with all of the diagnosed STI listed.
I created the new variables to capture each STI that would be listed under the Dx variable, but I can't figure out how to merge data within the same dataset so there is only one per patient at each timepoint.
data dx;
set dx;
if dx='ANOGENITAL WARTS (CONDYLOMATA ACUMINATA)' then MRWarts=1;
if dx='CHLAMYDIA' then MRCHLAMYDIA=1;
if dx='DYSPLASIA (ANAL, CERVICAL, OR VAGINAL)' then MRDYSPLASIA=1;
if dx='GONORRHEA' then MRGONORRHEA=1;
if dx='HEPATITIS B (HBV)' then MRHEPB=1;
if dx='HUMAN PAPILLOMAVIRUSES (HPV)-ANY MANIFESTATION' then MRHPV=1;
if dx='PEDICULOSIS PUBIS' then MRPUBIS=1;
if dx='SYPHILIS' then MRSYPHILIS=1;
if dx='TRICHOMONAS VAGINALIS' then MRTRICHOMONAS=1;
run;
Image of data structure I am looking for
taking the sample dataset that you provided in the image, you can use simple transpose for desired outcome.
data have;
input Pt_ID interval_round DX $10.;
datalines;
4 1 HIV
4 1 Warts
3 1 HIV
5 2 Chlamydia
;
run;
proc sort data=have1; by Pt_Id; run;
proc transpose data=have1 out=want(drop=_NAME_);
by Pt_Id;
id Dx;
var interval_round;
run;
proc print data=want; run;
Now this code will create all variables except interval_round, Say for example - a patient was screened for HIV in round 1 and Warts for round 2. Technically it should have only one row .. so how would you represent the interval_round then?

Reshaping and merging simulations in Stata

I have a dataset, which consists of 1000 simulations. The output of each simulation is saved as a row of data. There are variables alpha, beta and simulationid.
Here's a sample dataset:
simulationid beta alpha
1 0.025840106 20.59671241
2 0.019850549 18.72183088
3 0.022440886 21.02298228
4 0.018124857 20.38965861
5 0.024134726 22.08678021
6 0.023619479 20.67689981
7 0.016907209 17.69609466
8 0.020036455 24.6443037
9 0.017203175 24.32682682
10 0.020273349 19.1513272
I want to estimate a new value - let's call it new - which depends on alpha and beta as well as different levels of two other variables which we'll call risk and price. Values of risk range from 0 to 100, price from 0 to 500 in steps of 5.
What I want to achieve is a dataset that consists of values representing the probability that (across the simulations) new is greater than 0 for combinations of risk and price.
I can achieve this using the code below. However, the reshape process takes more hours than I'd like. And it seems to me to be something that could be completed a lot quicker.
So, my question is either:
i) is there an efficient way to generate multiple datasets from a single row of data without multiple reshape, or
ii) am I going about this in totally the wrong way?
set maxvar 15000
/* Input sample data */
input simulationid beta alpha
1 0.025840106 20.59671241
2 0.019850549 18.72183088
3 0.022440886 21.02298228
4 0.018124857 20.38965861
5 0.024134726 22.08678021
6 0.023619479 20.67689981
7 0.016907209 17.69609466
8 0.020036455 24.6443037
9 0.017203175 24.32682682
10 0.020273349 19.1513272
end
forvalues risk = 0(1)100 {
forvalues price = 0(5)500 {
gen new_r`risk'_p`price' = `price' * (`risk'/200)* beta - alpha
gen probnew_r`risk'_p`price' = 0
replace probnew_r`risk'_p`price' = 1 if new_r`risk'_p`price' > 0
sum probnew_r`risk'_p`price', mean
gen mnew_r`risk'_p`price' = r(mean)
drop new_r`risk'_p`price' probnew_r`risk'_p`price'
}
}
drop if simulationid > 1
save simresults.dta, replace
forvalues risk = 0(1)100 {
clear
use simresults.dta
reshape long mnew_r`risk'_p, i(simulationid) j(price)
keep simulation price mnew_r`risk'_p
rename mnew_r`risk'_p risk`risk'
save risk`risk'.dta, replace
}
clear
use risk0.dta
forvalues risk = 1(1)100 {
merge m:m price using risk`risk'.dta, nogen
save merged.dta, replace
}
Here's a start on your problem.
So far as I can see, you don't need more than one dataset.
The various reshapes and merges just rearrange what was first generated and that can be done within one dataset.
The code here in the first instance is for just one pair of values of alpha and beta. To simulate 1000 such, you would need 1000 times more observations, i.e. about 10 million, which is not usually a problem and to loop over the alphas and betas. But the loop can be tacit. We'll get to that.
This code has been run and is legal. It's limited to one alpha, beta pair.
clear
input simulationid beta alpha
1 0.025840106 20.59671241
2 0.019850549 18.72183088
3 0.022440886 21.02298228
4 0.018124857 20.38965861
5 0.024134726 22.08678021
6 0.023619479 20.67689981
7 0.016907209 17.69609466
8 0.020036455 24.6443037
9 0.017203175 24.32682682
10 0.020273349 19.1513272
end
local N = 101 * 101
set obs `N'
egen risk = seq(), block(101)
replace risk = risk - 1
egen price = seq(), from(0) to(100)
replace price = 5 * price
gen result = (price * (risk/200)* beta[1] - alpha[1]) > 0
bysort price risk: gen mean = sum(result)
by price risk: replace mean = mean[_N]/_N
Assuming now that you first read in 1000 values, here is a sketch of how to get the whole thing. This code has not been tested. That is, your dataset starts with 1000 observations; you then enlarge it to 10 million or so, and get your results. The tricksy part is using an expression for the subscript to ensure that each block of results is for a distinct alpha, beta pair. That's not compulsory; you could do it in a loop, but then you would need to generate outside the loop and replace within it.
local N = 101 * 101 * 1000
set obs `N'
egen risk = seq(), block(101)
replace risk = risk - 1
egen price = seq(), from(0) to(100)
replace price = 5 * price
egen sim = seq(), block(10201)
gen result = (price * (risk/200)* beta[ceil(_n/10201)] - alpha[ceil(_n/10201)]) > 0
bysort sim price risk: gen mean = sum(result)
by sim price risk: replace mean = mean[_N]/_N
Other devices used: egen to set up in blocks; getting the mean without repeated calls to summarize; using a true-or-false expression directly.
NB: I haven't tried to understand what you are doing, but it seems to me that the price-risk-simulation conditions define single values, so calculating a mean looks redundant. But perhaps that is in the code because you wish to add further detail to the code once you have it working.
NB2: This seems a purely deterministic calculation. Not sure that you need this code at all.

kdb+/q: Apply iterative procedure with updated variable to a column

Consider the following procedure f:{[x] ..} with starting value a:0:
Do something with x and a. The output is saved as the new version of a, and the output is returned by the function
For the next input x, redo the procedure but now with the new a.
For a single value x, this procedure is easily constructed. For example:
a:0;
f:{[x] a::a+x; :a} / A simple example (actual function more complicated)
However, how do I make such a function such that it also works when applied on a table column?
I am clueless how to incorporate this step for 'intermediate saving of a variable' in a function that can be applied on a column at once. Is there a special technique for this? E.g. when I use a table column in the example above, it will simply calculate a+x with a:0 for all rows, opposed to also updating a at each iteration.
No need to use global vars for this - can use scan instead - see here.
Example --
Generate a table -
q)t:0N!([] time:5?.z.p; sym:5?`3; price:5?100f; size:5?10000)
time sym price size
-----------------------------------------------
2002.04.04D18:06:07.889113280 cmj 29.07093 3994
2007.05.21D04:26:13.021438816 llm 7.347808 496
2010.10.30D10:15:14.157553088 obp 31.59526 1728
2005.11.01D21:15:54.022395584 dhc 34.10485 5486
2005.03.06D21:05:07.403334368 mho 86.17972 2318
Example with a simple accumilator - note, the function has access to the other args if needed (see next example):
q)update someCol:{[a;x;y;z] (a+1)}\[0;time;price;size] from t
time sym price size someCol
-------------------------------------------------------
2002.04.04D18:06:07.889113280 cmj 29.07093 3994 1
2007.05.21D04:26:13.021438816 llm 7.347808 496 2
2010.10.30D10:15:14.157553088 obp 31.59526 1728 3
2005.11.01D21:15:54.022395584 dhc 34.10485 5486 4
2005.03.06D21:05:07.403334368 mho 86.17972 2318 5
Say you wanted to get cumilative size:
q)update cuSize:{[a;x;y;z] (a+z)}\[0;time;price;size] from t
time sym price size cuSize
------------------------------------------------------
2002.04.04D18:06:07.889113280 cmj 29.07093 3994 3994
2007.05.21D04:26:13.021438816 llm 7.347808 496 4490
2010.10.30D10:15:14.157553088 obp 31.59526 1728 6218
2005.11.01D21:15:54.022395584 dhc 34.10485 5486 11704
2005.03.06D21:05:07.403334368 mho 86.17972 2318 14022
If you wanted more than one var passed through the scan, can pack more values into the first var, by giving it a more complex structure:
q)update cuPriceAndSize:{[a;x;y;z] (a[0]+y;a[1]+z)}\[0 0;time;price;size] from t
time sym price size cuPriceAndSize
--------------------------------------------------------------
2002.04.04D18:06:07.889113280 cmj 29.07093 3994 29.07093 3994
2007.05.21D04:26:13.021438816 llm 7.347808 496 36.41874 4490
2010.10.30D10:15:14.157553088 obp 31.59526 1728 68.014 6218
2005.11.01D21:15:54.022395584 dhc 34.10485 5486 102.1188 11704
2005.03.06D21:05:07.403334368 mho 86.17972 2318 188.2986 14022
#MdSalih solution is correct, I am just explaining here what could be the possible reason with global variable in your case and solution for that.
q) t:([]id: 1 2)
q)a:1
I think you might have been using it like this:
q) select k:{x:x+a;a::a+1;:x} id from t
output:
k
--
1
2
And a value is 2 which means function executed only once. Reason is we passed full id column list to function and (+) is atomic which means it operates on full list at once. In following ex. 2 will get added to all items in list.
q) 2 + (1;3;5)
Correct way to use it is 'each':
q)select k:{x:x+a;a::a+1;:x} each id from t
output:
k
--
2
3

Why does merge automatically set the latest dataset I created?

I'm having trouble using merge and I realized why : besides the table I want to merge, SAS seems to automatically add the latest table I created. The following code illustrates the issue :
DATA table1; /* to be merged dataset no 1*/
input X rep Y Z;
cards;
1 1 0 2
5 1 2 6
5 2 5 2
;
run;
proc sort; by x rep; run;
data table3; /* to be merged dataset no 2 */
input X;
cards;
1
5
5
10
10
15
;
run;
proc sort; by x; run;
data table3; /* rep stands for 'replicate' and makes sure there is no uniqueness issue */
set table3; by x;
retain rep;
if first.x then rep=0;
rep=rep+1; /*rep+1; */
run;
data table2; /*some other table having nothing to do with the merge*/
input Y W;
cards;
1 0
1 0
2 0
3 0
3 0
8 0
;
run;
data merge1;
merge table3 table1;
by x rep;
set nobs=n;
run;
When it is submitted, the log shows that the latest table created (table2) is somehow used to create merge1. Actually, table2 columns are added to what merge1 should be.
Trying to understand this, I found that this doesn't happen if I get rid of the set nobs=n; line in the definition of merge1.
I couldn't find why on the internet but I found several documents warning about how merge can be tricky (but for other reasons)...
Therefore, my questions are :
Why does this happen and how to fix it ? (I need nobs in my calculations) I would be able to escape the issue doing the merge and the following treatment in separated data steps but I would like to understand the whole thing and how to properly deal with it.
Is merge the best way to add values in only one column of a dataset ? (here, table1 column X is updated by table3, but Y and Z are not yet). (this question will be secondary if the first one is answered)
The set nobs=n statement is reading in table2 implicitly from &SYSLAST.
It's like doing
data table2 ;
/* some stuff */
run ;
data want ;
set ; /* implicity use &SYSLAST - table2 in this case - as input dataset */
run ;
I'm unsure what you intend to achieve with set nobs=n, but the merge datastep without set nobs=n will return Y and Z values based on the join criteria.
EDIT:
data merge1;
merge table3 table1 end=eof ;
by x rep;
if eof then call symputx('NOBS',_n_) ;
run;
data merge1 ;
set merge1 ;
NOBS = &NOBS ;
run ;
Output of merge1
X rep Y Z NOBS
1 1 0 2 6
5 1 2 6 6
5 2 5 2 6
10 1 6
10 2 6
15 1 6