I have this variable x_yyww, for every week the past 5 years. It can take the value 0, 1 or blank. I want to create a new line of variables which takes the value yyWww if the combination 1 and then 0 occurs, e.g. if x_0449=1 and x_0450=0. I then need to create this new set of variables, call them ovg_1 - ... ovg_n, which shows the specific date yyWww (04W50) when the event occurred, where n is the maximal number of times this combination has occurred for a given person. This is where I need help.
When I try to code this in SAS, i end up with all the variables ovg_1 ... ovg_n only taken the most recent value. Since my skills clearly are limited, I have been trying to do this with a fixed n=25...
%macro test(h) ;
...
%do i=2 %to 52;
%let j=%eval(&i-1);
%if &j lt 10 %then %let j=0&j;
%let l=%eval(&i);
%if &l lt 10 %then %let l=0&l;
....
%do n=1 %to 25
if x_&h&l="0" and x_&h&j="1" then do;
ovg_&n = intnx('week.1', mdy(1, 1, &h), &l); end;
format ovg_&k weekv5. ;
....
Can anyone help me out on this?
(I know I am missing some endpoint by this, and I have taken care of it in my original coding)
Try something like this:
/* create some test data */
data test(drop=j);
retain starts_at '01JAN1990'd; *** date event collection starts;
array x_{*} event1 - event1000; *** events;
array y_{*} date1 - date1000; *** date of events;
do person=1 to 10;
do j=1 to dim(x_); *** generate 1000 0s and 1s;
x_{j}=round(ranuni(person));
y_{j}=.;
end;
n=0;
do j=1 to dim(x_)-1; *** count 0s followed by 1s;
if x_{j}=0 and x_{j+1}=1 then do;
n+1;
y_{n}=starts_at+(j*7); *** calculate the date;
end;
end;
output;
format date1-date1000 weekv5.;
end;
run;
Related
So I have a range of datasets in a specific library. These datasets are named in the format DATASET_YYYYMM, with one dataset for each month. I am trying to append a range of these datasets based on user input for the date range. i.e. If start_date is 01NOV2019 and the end_date is 31JAN2020, I want to append the three datasets: LIBRARY.DATASET_201911, LIBRARY.DATASET_201912 and LIBRARY.DATASET_202001.
The range is obviously variable, so I can't simply name the datasets manually in a set function. Since I need to loop through the years and months in the date range, I believe a macro is the best way to do this. I'm using a loop within the SET statement to append all the datasets. I have copied my example code below. It does work in theory. But in practice, only if we are looping over the months of November and December. As the format of the dataset name has a two digit month, for Jan-Sept it will be 01-09. The month function returns 1-9 however, and of course a 'File DATASET_NAME does not exist' error is thrown.
Problem is I cannot figure out a way to get it to interpret the month with leading 0, without ruining functionality of another part of the loop/macro.
I have tried numerous approaches to format the number as z2, cannot get any to work.
i.e. Including PUTN functions in the DO line for quote_month as follows, it ignores the leading zero when generating the dataset name in the line below.
%DO quote_month = %SYSFUNC(IFN("e_year. = &start_year.,%SYSFUNC(PUTN(&start_month.,z2.)),1,.)) %TO %SYSFUNC(IFN("e_year. = &end_year.,%SYSFUNC(PUTN(&end_month.,z2.)),12,.));
Below is example code (without any attempt to reformat it to z2) - it will throw an error because it cannot find 'dataset_20201' because it is actually called 'dataset_202001'. The dataset called dataset_combined_example produces the desired output of the code by manually referencing the dataset names which it will be unable to do in practice. Does anyone know how to go about this?
DATA _NULL_;
FORMAT start_date end_date DATE9.;
start_date = '01NOV2019'd;
end_date = '31JAN2020'd;
CALL symput('start_date',start_date);
CALL symput('end_date',end_date);
RUN;
DATA dataset_201911;
input name $;
datalines;
Nov1
Nov2
;
RUN;
DATA dataset_201912;
input name $;
datalines;
Dec1
Dec2
;
RUN;
DATA dataset_202001;
input name $;
datalines;
Jan1
Jan2
;
RUN;
DATA dataset_combined_example;
SET dataset_201911 dataset_201912 dataset_202001;
RUN;
%MACRO get_table(start_date, end_date);
%LET start_year = %SYSFUNC(year(&start_date.));
%LET end_year = %SYSFUNC(year(&end_date.));
%LET start_month = %SYSFUNC(month(&start_date.));
%LET end_month = %SYSFUNC(month(&end_date.));
DATA dataset_combined;
SET
%DO quote_year = &start_year. %TO &end_year.;
%DO quote_month = %SYSFUNC(IFN("e_year. = &start_year.,&start_month.,1,.)) %TO %SYSFUNC(IFN("e_year. = &end_year.,&end_month.,12,.));
dataset_"e_year."e_month.
%END;
%END;
;
RUN;
%MEND;
%get_table(&start_date.,&end_date.);
You could do this using putn and z2. format.
%DO quote_year = &start_year. %TO &end_year.;
%DO quote_month = %SYSFUNC(IFN("e_year. = &start_year.,&start_month.,1,.)) %TO %SYSFUNC(IFN("e_year. = &end_year.,&end_month.,12,.));
dataset_"e_year.%sysfunc(putn("e_month.,z2.))
%END;
%END;
You can also do this using the metadata tables without having to resort to macro loops in the first place:
/* A few datasets to combine */
data
DATASET_201910
DATASET_201911
DATASET_201912
DATASET_202001
;
run;
%let START_DATE = '01dec2019'd;
%let END_DATE = '31jan2020'd;
proc sql noprint;
select catx('.', libname, memname) into :DS_LIST separated by ' '
from dictionary.tables
where
&START_DATE <=
case
when prxmatch('/DATASET_\d{6}/', memname)
then input(scan(memname, -1, '_'), yymmn6.)
else -99999
end
<= &END_DATE
and libname = 'WORK'
;
quit;
data combined_datasets /view=combined_datasets;
set &DS_LIST;
run;
The case-when in the where clause ensures that any other datasets present in the same library that don't match the expected naming scheme are ignored.
One key difference with this approach is that you will never end up attempting to read a dataset that doesn't exist if one of the expected datasets in your range is missing.
You can use the Z format to generate strings with leading zeros.
But your problem is much easier if you use SAS date functions and formats to generate the YYYYMM strings. Just use a normal iterative %DO loop to cycle the month offset from zero to the number of months between the two dates.
%macro get_table(start_date, end_date);
%local offset dsname ;
data dataset_combined;
set
%do offset=0 %to %sysfunc(intck(month,&start_date,&end_date));
%let dsname=dataset_%sysfunc(intnx(month,&start_date,&offset),yymmn6);
&dsname.
%end;
;
run;
%mend get_table;
Result:
445 options mprint;
446 %get_table(start_date='01NOV2019'd,end_date='31JAN2020'd);
MPRINT(GET_TABLE): data dataset_combined;
MPRINT(GET_TABLE): set dataset_201911 dataset_201912 dataset_202001 ;
MPRINT(GET_TABLE): run;
In a macro
Use INTNX to compute the bounds for a loop over date values. Within the loop:
Compute the candidate data set name according to specified lib, prefix and desired date value format. <yyyy><mm> is output by format yymmn6.
Use EXIST to check candidate data sets for existence.
Alternatively, do not check, but make sure to set OPTIONS NODSNFERR prior to combining. The setting will prevent errors when specifying a non-existent data set.
Update the loop index to the end of the month so the next increment takes the index to the start of the next month.
%macro names_by_month(lib=work, prefix=data_, start_date=today(), end_date=today(), format=yymmn6.);
%local index name;
%* loop over first-of-the-month date values;
%do index = %sysfunc(intnx(month, &start_date, 0)) %to %sysfunc(intnx(month, &end_date, 0));
%* compute month dependent name;
%let name = &lib..&prefix.%sysfunc(putn(&index,&format));
%* emit name if it exists;
%if %sysfunc(exist(&name)) or %sysfunc(exist(&name,VIEW)) %then %str(&name);
%* prepare index for loop +1 increment so it goes to start of next month;
%let index = %sysfunc(intnx(month, &index, 0, E));
%end;
%mend;
* example usage:
data combined_imports(label="nov2019 to jan2020");
set
%names_by_month(
prefix=import_,
start_date='01NOV2019'd,
end_date = '31JAN2020'd
)
;
run;
I am trying to compare two numberic value in a Macro.
But I keep getting the following message:
ERROR: A character operand was found in the %EVAL function or %IF condition where a numeric operand is required. The condition was: 0.2
ERROR: The %TO value of the %DO I loop is invalid.
ERROR: A character operand was found in the %EVAL function or %IF condition where a numeric operand is required. The condition was: 0.05
ERROR: The %BY value of the %DO I loop is invalid.
ERROR: The macro FAIL will stop executing.
My code is the following:
%macro fail;
%do i=0 %to 0.2 %by 0.05;
data failcrs;
set fail;
if f_p>=input(&i, 8.) then output;
run;
%end;
%mend failcrs;
f_p is a numeric variable.
What is wrong with my code? Please help.
Thank you so much!
Conditional tests in macro code (%if, %until, %while, etc) use %eval() macro function that only does integer arithmetic. This includes the increments and tests done in a %do-%to-%by loop.
To use floating point arithmetic you need to use the %sysvalf() macro function.
You could code your own increments to the loop counter.
%let i=0;
%do %while( %sysevalf( &I <= 0.2 ) );
...
%let i=%sysevalf(&i + 0.05);
%end;
Or make the loop counter an integer and use another macro variable to hold the fraction.
%do j=0 %to 20 %by 5 ;
%let i=%sysevalf(&j/100);
...
%end;
You have a couple of issues.
Macro loops work better with integers, but an easy workaround is a %DO %UNTIL loop instead.
Name on %MEND is different than on %MACRO
Invalid values for %DO %I loop.
Non-unique data set name, which means the output overwrites itself.
*fake data to work with;
data fail;
do f_p=0 to 0.2 by 0.01;
output;
end;
run;
%macro fail;
%let i=0;
%do %until(&i = 0.2); /*2*/
data failcrs_%sysevalf(&i*100); /*3*/
set fail;
if f_p>=&i then output;
run;
%let i = %sysevalf(&i + 0.05);
%end;
%mend fail; /*3*/
*test macro;
%fail;
The numbers in the comments align with the issues identified.
Try using best32. But why do you want to loop, when your dataset is overwritten for each loop. Please check log for each of step below. As at #Reeza in comments explains below you even do not an input statement
Options mprint;
/* do this*/
%macro fail;
%let i =15;
data failcrs;
set sashelp.class;
if age lt input(&i, best32.) then output;
run;
%mend fail;
%fail
/* dataset overwritten every time to finally pick up 15 as value check in the log*/
%macro fail1;
%do i = 1 %to 15;
data failcrs1;
set sashelp.class;
if age lt input(i, best32.) then output;
run;
%end;
%mend fail1;
%fail1
%macro fail;
%let i=0;
%do %until(&i = 0.2);
data failcrs;
set crse_grade_dist_fail;
if f_p>=&i then output;
run;
proc sql;
create table count_failclass as
select strm, count(class_nbr) as numfclass_%sysevalf(&i*100)
from failcrs
group by strm;
quit;
proc sql;
create table failfaculty as
select strm, instructor_id, instructor, count(class_nbr) as numfclass
from failcrs
group by strm, instructor_id, instructor;
quit;
proc sql;
create table count_failfaculty as
select strm, count(instructor) as numffaculty_%sysevalf(&i*100)
from failfaculty
group by strm;
quit;
data count_class_faculty;
set count_class_faculty;
set count_failclass;
set count_failfaculty;
run;
%let i = %sysevalf(&i + 0.05);
%end;
%mend fail;
Good thing is my data doesn't have f_p=0, all of them is greater than zero. Because I only want to count failed courses.
Documentation is written to be read, a simple search for "SAS 9.4 Macro Do" should explain it all -- start, stop and by are integers -- integers in the sense that whatever macro source expression in their place evaluates implicitly or explicitly to an integer at need time.
The macro you coded is a little strange. It will generate multiple data steps that all overwrite the same dataset. You might want to concentrate on not writing macro code first, and move to it when the need to have repetitive boilerplate code submitted. Writing good macro code means you have to think in terms of "will this generate appropriate source code and what side effect will these macro statements have in their resolution scope"
%DO, Iterative Statement
Syntax
%DO macro-variable=start %TO stop <%BY increment>;
text and macro language statements
%END;
Required Arguments
macro-variable
names a macro variable or a text expression that generates a macro
variable name. Its value functions as an index that determines the
number of times the %DO loop iterates. If the macro variable specified
as the index does not exist, the macro processor creates it in the
local symbol table.
You can change the value of the
index variable during processing. For example, using conditional
processing to set the value of the index variable beyond the stop
value when a certain condition is met ends processing of the loop.
startstop
specify integers or macro expressions that generate integers
to control the number of times the portion of the macro between the
iterative %DO and %END statements is processed.
The first time the
%DO group iterates, macro-variable is equal to start. As processing
continues, the value of macro-variable changes by the value of
increment until the value of macro-variable is outside the range of
integers included by start and stop.
increment
specifies an integer
(other than 0) or a macro expression that generates an integer to be
added to the value of the index variable in each iteration of the
loop. By default, increment is 1. Increment is evaluated before the
first iteration of the loop. Therefore, you cannot change it as the
loop iterates.
I have some data sets that I wrote some code to clean according to some methods according to some biological literature and then I want to split it into day and night (because they must be analyzed separately). It worked but now I need to do this for the full set which is WAY to many files for me to want to deal with one by one. So I am now trying to write a macro to split it into days and nights for me..
My data looks like so
Hour var1 var2 var3
1 123 90 100
2 122 99 108
...........
4 156 80 120
4 156 80 145
4 143 82 132
basically night has 1 obs per hour day 3. I also have this for many days.
Each dataset is named STUDYIDID#_first or STUDYID_ID#_last. I want to generate four datasets per dataset.
So MYID111_first would create: MYID111_first_day_var1, MYID111_first_day_var2, MYID111_first_night_var1 , and MYID111_first_night_var2.
I would then LIKE to append them into 4 datasets:
MYID_A_first_day_var1, MYID_A_first_day_var2, MYID_A_first_night_var1 , and MYID_A_first_night_var2.
MY CODE SO FAR:
%macro datacut(libname,worklib=work, grp = _A ,time1 = _night , time2 = _day type1 = _var1 , type2 = _var2);
%local num i;
proc datasets library=&libname memtype=data nodetails;
contents out=&worklib..temp1(keep=memname) data=_all_ noprint;
run;
data _null_;
set &worklib..temp1 end=final;
by memname notsorted;
if last.memname;
n+1;
call symput('ds'||left(put(n,8.)),trim(memname));
if final then call symput('num',put(n,8.));
run;
%do i=1 %to #
/* do the artifact removing method */
DATA &libname..&&ds&i;
SET &libname..&&ds&i;
PT_ID = '&ds&i' ;
IF var1< 60 OR var1> 230 then delete;
IF var2< 30 OR var2> 230 THEN delete;
IF var3< 60OR var3 > 135 THEN DELETE;
IF var2 > var1 then delete;
run;
/* get just the night values */
PROC SQL;
CREATE TABLE &libname..&&ds&i&time1 as
SELECT *
FROM &libname..&&ds&i
WHERE Hour BETWEEN 0 and 6 OR Hour BETWEEN 22 and 24
order by systolic
;
QUIT;
/* trim off the proper number of observations for variable 1 */
DATA &libname..&&ds&i&time1&type1;
SET &libname..&&ds&i&time1 end=eof;
IF _N_ =1 then delete;
if eof then delete;
run;
PROC append base= &libname..&&ds&time1&type1
data= &libname..&&ds&i&time1;
run;
QUIT;
%end;
%mend datacut;
%datacut(work)
Now the initial datastep works correctly but the later ones don't rename the data as planned. I get a bunch of datasets called Ds10_night_var1 with the wrong field names (memtype, nodetails, data)
I get the warning:
WARNING: Apparent symbolic reference DS1_NIGHT not resolved.
NOTE: Line generated by the macro variable "TIME1".
1 work.&ds1_night
-
22
200
ERROR 22-322: Expecting a name.
ERROR 200-322: The symbol is not recognized and will be ignored.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE SQL used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds
WARNING: Apparent symbolic reference DS1_NIGHT_SYS not resolved.
22: LINE and COLUMN cannot be determined.
NOTE 242-205: NOSPOOL is on. Rerunning with OPTION SPOOL might allow recovery of the LINE and
COLUMN where the error has occurred.
ERROR 22-322: Syntax error, expecting one of the following: a name, a quoted string, /, ;,
_DATA_, _LAST_, _NULL_.
201: LINE and COLUMN cannot be determined.
NOTE: NOSPOOL is on. Rerunning with OPTION SPOOL might allow recovery of the LINE and COLUMN
where the error has occurred.
ERROR 201-322: The option is not recognized and will be ignored.
So I want the right names for my file AND my datasets to actually have data I and I don't understand why they don't.
As you know, you write a macro variable as & followed by its name, optionally followed by .. With this . you can explicitly end the macro variable reference, so you can use a macro variable as a prefix, like in
%let prefix = fore;
Aspect = &prefix.Ground;
which evaluates to Aspect = foreGround;. And that is why
%let myLib = abc;
%let mymember = xyz;
data &myLib.&myMember;
is an error, as it evaluates to data abcxyz;, and you must write
data &myLib..&myMember;
** or as I prefer **;
data &myLib..&myMember.;
to get data abc.xyz;.
For the case you need macro variables to create macro variable names from, SAS allows to write double ampersands &&, which evaluate to a single & and continues evaluating until all ampersands are consumed. So suppose
%let i = 1;
%let ds1 = myData;
%let time = _nigth;
This is how SAS evaluates &&ds&i&time1 :
&&ds&i&time1
&ds1_night
ERROR because a macro variable ds1_night is not defined
This is how SAS evaluates &&ds&i..&time1. :
&&ds&i..&time1.
&ds1._night
myData_night
I need some assistance with splitting a large SAS dataset into smaller datasets.
Each month I'll have a dataset containing a few million records. This number will vary from month to month. I need to split this dataset into multiple smaller datasets containing 250,000 records each. For example, if I have 1,050,000 records in the original dataset then I need the end result to be 4 datasets containing 250,000 records and 1 dataset containing 50,000 records.
From what I've been looking at it appears this will require using macros. Unfortunately I'm fairly new to SAS (unfamiliar with using macros) and don't have a lot of time to accomplish this. Any help would be greatly appreciated.
Building on Joe's answer, maybe you could try something like this :
%MACRO SPLIT(DATASET);
%LET DATASET_ID = %SYSFUNC(OPEN(&DATASET.));
%LET NOBS = %SYSFUNC(ATTRN(&DATASET__ID., NLOBS));
%LET NB_DATASETS = %SYSEVALF(&NOBS. / 250000, CEIL);
DATA
%DO I=1 %TO &NB_DATASETS.;
WANT&I.
%END;;
SET WANT;
%DO I=1 %TO &NB_DATASETS.;
%IF &I. > 1 %THEN %DO; ELSE %END; IF _N_ LE 2.5E5 * &I. THEN OUTPUT WANT&I.;
%END;
RUN;
%MEND SPLIT;
You can do it without macros at all, if you don't mind asking for datasets that may not exist, and have a reasonable bound on things.
data want1 want2 want3 want4 want5 want6 want7 want8 want9;
if _n_ le 2.5e5 then output want1;
else if _n_ le 5e5 then output want2;
else if _n_ le 7.5e5 then output want3;
... etc....
run;
Macros would make that more efficient to program and cleaner to read, but wouldn't change how it actually runs in reality.
You can do it without macros, using CALL EXECUTE(). It creates SAS-code as text strings and then executes it, after your "manually written" code completed.
data _null_;
if 0 then set have nobs=n;
do i=1 to ceil(n/250000);
call execute (cats("data want",i)||";");
call execute ("set have(firstobs="||(i-1)*250000+1||" obs="||i*250000||");");
call execute ("run;");
end;
run;
The first result on Google is from the SAS User Group International (SUGI)
These folks are your friends.
The article is here:
http://www2.sas.com/proceedings/sugi27/p083-27.pdf
The code is:
%macro split(ndsn=2);
data %do i = 1 %to &ndsn.; dsn&i. %end; ;
retain x;
set orig nobs=nobs;
if _n_ eq 1
then do;
if mod(nobs,&ndsn.) eq 0
then x=int(nobs/&ndsn.);
else x=int(nobs/&ndsn.)+1;
end;
if _n_ le x then output dsn1;
%do i = 2 %to &ndsn.;
else if _n_ le (&i.*x)
then output dsn&i.;
%end;
run;
%mend split;
%split(ndsn=10);
All you need to do is replace the 10 digit in "%split(ndsn=10);" with the number you require.
In Line 4, "set orig nobs=nobs;", simply replace orig with your dataset name.
Hey presto!
A more efficient option, if you have room in memory to store one of the smaller datasets, is a hash solution. Here's an example using basically what you're describing in the question:
data in_data;
do recid = 1 to 1.000001e7;
datavar = 1;
output;
end;
run;
data _null_;
if 0 then set in_data;
declare hash h_out();
h_out.defineKey('_n_');
h_out.defineData('recid','datavar');
h_out.defineDone();
do filenum = 1 by 1 until (eof);
do _n_ = 1 to 250000 until (eof);
set in_data end=eof;
h_out.add();
end;
h_out.output(dataset:cats('file_',filenum));
h_out.clear();
end;
stop;
run;
We define a hash object with the appropriate parameters, and simply tell it to output every 250k records, and clear it. We could do a hash-of-hashes here also, particularly if it weren't just "Every 250k records" but some other criteria drove things, but then you'd have to fit all of the records in memory, not just 250k of them.
Note also that we could do this without specifying the variables explicitly, but it requires having a useful ID on the dataset:
data _null_;
if 0 then set in_data;
declare hash h_out(dataset:'in_data(obs=0)');
h_out.defineKey('recid');
h_out.defineData(all:'y');
h_out.defineDone();
do filenum = 1 by 1 until (eof);
do _n_ = 1 to 250000 until (eof);
set in_data end=eof;
h_out.add();
end;
h_out.output(dataset:cats('file_',filenum));
h_out.clear();
end;
stop;
run;
Since we can't use _n_ anymore for the hash ID due to using the dataset option on the constructor (necessary for the all:'y' functionality), we have to have a record ID. Hopefully there is such a variable, or one could be added with a view.
Here is a basic approach. This requires manual adjustment of the intervals, but is easy to understand.
* split data;
data output1;
set df;
if 1 <= _N_ < 5 then output;
run;
data output2;
set df;
if 5 <= _N_ < 10 then output;
run;
data output3;
set df;
if 10 <= _N_ < 15 then output;
run;
data output4;
set df;
if 15 <= _N_ < 22 then output;
run;
I have a table called term_table containing the below columns
comp, term_type, term, score, rank
I go through every observation and at each obs, I want to store the value of variable rank to a macro variable called curr_r. The code I created below does not work
Data Work.term_table;
input Comp $
Term_type $
Term $
Score
Rank
;
datalines;
comp1 term_type1 A 1 1
comp2 term_type2 A 2 10
comp3 term_type3 A 3 20
comp4 term_type4 B 4 20
comp5 term_type5 B 5 40
comp6 term_type6 B 6 100
;
Run;
%local j;
DATA tmp;
SET term_table;
LENGTH freq 8;
BY &by_var term_type term;
RETAIN freq;
CALL SYMPUT('curr_r', rank);
IF first.term THEN DO;
%do j = 1 %to &curr_r;
do some thing
%end;
END;
RUN;
Could you help me to solve the problem
Thanks a lot
Hung
The call symput statement does create the macro var &curr_r with the value of rank, but it is not available until after the data step.
However, I don't think you need to create the macro var &curr_r. I don't think a macro is needed at all.
I think the below should work: (Untested)
DATA tmp;
SET term_table;
LENGTH freq 8;
BY &by_var term_type term;
RETAIN freq;
IF first.term THEN DO;
do j = 1 to rank;
<do some thing>
end;
END;
RUN;
If you needed to use the rank from a prior obs, use the LAG function.
Start=Lag(rank);
To store each value of RANK in a macro variable, the below will do that:
Proc Sql noprint;
select count(rank)
into :cnt
from term_table;
%Let cnt=&cnt;
select rank
into :curr_r1 - :curr_r&cnt
from term_table;
quit;