Related
So I have a range of datasets in a specific library. These datasets are named in the format DATASET_YYYYMM, with one dataset for each month. I am trying to append a range of these datasets based on user input for the date range. i.e. If start_date is 01NOV2019 and the end_date is 31JAN2020, I want to append the three datasets: LIBRARY.DATASET_201911, LIBRARY.DATASET_201912 and LIBRARY.DATASET_202001.
The range is obviously variable, so I can't simply name the datasets manually in a set function. Since I need to loop through the years and months in the date range, I believe a macro is the best way to do this. I'm using a loop within the SET statement to append all the datasets. I have copied my example code below. It does work in theory. But in practice, only if we are looping over the months of November and December. As the format of the dataset name has a two digit month, for Jan-Sept it will be 01-09. The month function returns 1-9 however, and of course a 'File DATASET_NAME does not exist' error is thrown.
Problem is I cannot figure out a way to get it to interpret the month with leading 0, without ruining functionality of another part of the loop/macro.
I have tried numerous approaches to format the number as z2, cannot get any to work.
i.e. Including PUTN functions in the DO line for quote_month as follows, it ignores the leading zero when generating the dataset name in the line below.
%DO quote_month = %SYSFUNC(IFN("e_year. = &start_year.,%SYSFUNC(PUTN(&start_month.,z2.)),1,.)) %TO %SYSFUNC(IFN("e_year. = &end_year.,%SYSFUNC(PUTN(&end_month.,z2.)),12,.));
Below is example code (without any attempt to reformat it to z2) - it will throw an error because it cannot find 'dataset_20201' because it is actually called 'dataset_202001'. The dataset called dataset_combined_example produces the desired output of the code by manually referencing the dataset names which it will be unable to do in practice. Does anyone know how to go about this?
DATA _NULL_;
FORMAT start_date end_date DATE9.;
start_date = '01NOV2019'd;
end_date = '31JAN2020'd;
CALL symput('start_date',start_date);
CALL symput('end_date',end_date);
RUN;
DATA dataset_201911;
input name $;
datalines;
Nov1
Nov2
;
RUN;
DATA dataset_201912;
input name $;
datalines;
Dec1
Dec2
;
RUN;
DATA dataset_202001;
input name $;
datalines;
Jan1
Jan2
;
RUN;
DATA dataset_combined_example;
SET dataset_201911 dataset_201912 dataset_202001;
RUN;
%MACRO get_table(start_date, end_date);
%LET start_year = %SYSFUNC(year(&start_date.));
%LET end_year = %SYSFUNC(year(&end_date.));
%LET start_month = %SYSFUNC(month(&start_date.));
%LET end_month = %SYSFUNC(month(&end_date.));
DATA dataset_combined;
SET
%DO quote_year = &start_year. %TO &end_year.;
%DO quote_month = %SYSFUNC(IFN("e_year. = &start_year.,&start_month.,1,.)) %TO %SYSFUNC(IFN("e_year. = &end_year.,&end_month.,12,.));
dataset_"e_year."e_month.
%END;
%END;
;
RUN;
%MEND;
%get_table(&start_date.,&end_date.);
You could do this using putn and z2. format.
%DO quote_year = &start_year. %TO &end_year.;
%DO quote_month = %SYSFUNC(IFN("e_year. = &start_year.,&start_month.,1,.)) %TO %SYSFUNC(IFN("e_year. = &end_year.,&end_month.,12,.));
dataset_"e_year.%sysfunc(putn("e_month.,z2.))
%END;
%END;
You can also do this using the metadata tables without having to resort to macro loops in the first place:
/* A few datasets to combine */
data
DATASET_201910
DATASET_201911
DATASET_201912
DATASET_202001
;
run;
%let START_DATE = '01dec2019'd;
%let END_DATE = '31jan2020'd;
proc sql noprint;
select catx('.', libname, memname) into :DS_LIST separated by ' '
from dictionary.tables
where
&START_DATE <=
case
when prxmatch('/DATASET_\d{6}/', memname)
then input(scan(memname, -1, '_'), yymmn6.)
else -99999
end
<= &END_DATE
and libname = 'WORK'
;
quit;
data combined_datasets /view=combined_datasets;
set &DS_LIST;
run;
The case-when in the where clause ensures that any other datasets present in the same library that don't match the expected naming scheme are ignored.
One key difference with this approach is that you will never end up attempting to read a dataset that doesn't exist if one of the expected datasets in your range is missing.
You can use the Z format to generate strings with leading zeros.
But your problem is much easier if you use SAS date functions and formats to generate the YYYYMM strings. Just use a normal iterative %DO loop to cycle the month offset from zero to the number of months between the two dates.
%macro get_table(start_date, end_date);
%local offset dsname ;
data dataset_combined;
set
%do offset=0 %to %sysfunc(intck(month,&start_date,&end_date));
%let dsname=dataset_%sysfunc(intnx(month,&start_date,&offset),yymmn6);
&dsname.
%end;
;
run;
%mend get_table;
Result:
445 options mprint;
446 %get_table(start_date='01NOV2019'd,end_date='31JAN2020'd);
MPRINT(GET_TABLE): data dataset_combined;
MPRINT(GET_TABLE): set dataset_201911 dataset_201912 dataset_202001 ;
MPRINT(GET_TABLE): run;
In a macro
Use INTNX to compute the bounds for a loop over date values. Within the loop:
Compute the candidate data set name according to specified lib, prefix and desired date value format. <yyyy><mm> is output by format yymmn6.
Use EXIST to check candidate data sets for existence.
Alternatively, do not check, but make sure to set OPTIONS NODSNFERR prior to combining. The setting will prevent errors when specifying a non-existent data set.
Update the loop index to the end of the month so the next increment takes the index to the start of the next month.
%macro names_by_month(lib=work, prefix=data_, start_date=today(), end_date=today(), format=yymmn6.);
%local index name;
%* loop over first-of-the-month date values;
%do index = %sysfunc(intnx(month, &start_date, 0)) %to %sysfunc(intnx(month, &end_date, 0));
%* compute month dependent name;
%let name = &lib..&prefix.%sysfunc(putn(&index,&format));
%* emit name if it exists;
%if %sysfunc(exist(&name)) or %sysfunc(exist(&name,VIEW)) %then %str(&name);
%* prepare index for loop +1 increment so it goes to start of next month;
%let index = %sysfunc(intnx(month, &index, 0, E));
%end;
%mend;
* example usage:
data combined_imports(label="nov2019 to jan2020");
set
%names_by_month(
prefix=import_,
start_date='01NOV2019'd,
end_date = '31JAN2020'd
)
;
run;
I am trying to change the length of the variables based on a list that I have and the code seems to work but the desired output is not achieved. here is the code:
%macro LEN();
Proc sql ;
select count(name) into: varnum from variab;
select name into: varname1-:varname%trim(%left(&varnum)) from Variab;
select length3 into: len from Length;
Quit;
%do i=1 %to &varnum;
data Zero;
length &&varname&i $ &&len&i.;
set desti.test;
length _numeric_ 4.;
format _numeric_ 12.2;
run;
%end;
%mend;
It gives a warning
WARNING: Multiple lengths were specified for the variable fscadl1 by
input data set(s). This can cause truncation
of data.
and it doesnt change the length of the variable. what is wrong in this code?
Are you trying to change a list of variables in one dataset? You're repeating the entire data step for each iteration, but only writing to a constant destination, which is inconsistent.
Probably what you want is:
Proc sql ;
select count(name) into: varnum from variab;
select name into: varname1-:varname%trim(%left(&varnum)) from Variab;
select length3 into: len from Length;
Quit;
%macro set_len(varnum=);
%do i=1 %to &varnum;
length &&varname&i $ &&len&i.;
%end;
%mend;
data Zero;
%set_len(&varnum);
set desti.test;
length _numeric_ 4.;
format _numeric_ 12.2;
run;
Note that you'd need to define &&len&i as you're not doing that currently.
The warning messages suggests that it is working. SAS started throwing that warning when you truncate a variable. You can suppress the warning message with the VARLENCHK option.
Below works:
options varlenchk=nowarn;
data want;
length name $ 3;
set sashelp.class;
length _numeric_ 4;
run;
If your code isn't working, I would turn on MPRINT to see make sure your macro is generating the SAS code you expect.
Lets's say I have a bunch of variables named the same way and I'd like to recode them and add a prefix to each (the variables are all numeric).
In Stata I would do something like (let's say the variables start with eq)
foreach var of varlist eq* {
recode var (1/4=1) (else=0), pre(r_)
}
How can I do this in SAS? I'd like to use the %DO macros, but I'm not familiar with them (I want to avoid SQL). I'd appreciate if you could include comments explaining each step!
SAS syntax for this would be easier if your variables are named using numeric suffix. That is, if you had ten variables with names of eq1, eq2, .... , eq10, then you could just use variable lists to define both sets of variables.
There are a number of ways to translate your recode logic. If we assume you have clean variables then we can just use a boolean expression to generate a 0/1 result. So if 4 and 5 map to 1 and the rest map to 0 you could use x in (4,5) or x > 3 as the boolean expresson.
data want;
set have;
array old eq1-eq10 ;
array new r_eq1-r_eq10 ;
do i=1 to dim(old);
new(i) = old(i) in (4,5);
end;
run;
If you have missing values or other complications you might want to use IF/THEN logic or a SELECT statement or you could define a format you could use to convert the values.
If your list of names is more random then you might need to use some code generation, such as macro code, to generate the new variable names.
Here is one method that use the eq: variable list syntax in SAS that is similar to the syntax of your variable selection before. Use PROC TRANSPOSE on an empty (obs=0) version of your source dataset to get a dataset with the variable names that match your name pattern.
proc transpose data=have(obs=0) out=names;
var eq: ;
run;
Then generate two macro variables with the list of old and new names.
proc sql noprint ;
select _name_
, cats('r_',_name_)
into :old_list separated by ' '
, :new_list separated by ' '
from names
;
quit;
You can then use the two macro variables in your ARRAY statements.
array old &old_list ;
array new &new_list ;
You can do this with rename and a dash indicating which variables you want to rename. Note the following only renames the col variables, and not the other one:
data have;
col1=1;
col2=2;
col3=3;
col5=5;
other=99;
col12=12;
run;
%macro recoder(dsn = , varname = , prefix = );
/*select all variables that include the string "varname"*/
/*(you can change this if you want to be more specific on the conditions that need to be met to be renamed)*/
proc sql noprint;
select distinct name into: varnames
separated by " "
from dictionary.columns where memname = upcase("&dsn.") and index(name, "&varname.") > 0;
quit;
data want;
set have;
/*loop through that list of variables to recode*/
%do i = 1 %to %sysfunc(countw(&varnames.));
%let this_varname = %scan(&varnames., &i.);
/*create a new variable with desired prefix based on value of old variable*/
if &this_varname. in (1 2 3) then &prefix.&this_varname. = 0;
else if &this_varname. in (4 5) then &prefix.&this_varname. = 1;
%end;
run;
%mend recoder;
%recoder(dsn = have, varname = col, prefix = r_);
PROC TRANSPOSE will give you good flexibility with regards to the way your variables are named.
proc transpose data=have(obs=0) out=vars;
var col1-numeric-col12;
copy col1;
run;
proc transpose data=vars out=revars(drop=_:) prefix=RE_;
id _name_;
run;
data recode;
set have;
if 0 then set revars;
array c[*] col1-numeric-col12;
array r[*] re_:;
call missing(of r[*]);
do _n_ = 1 to dim(c);
if c[_n_] in(1 2 3) then r[_n_] = 0;
else if c[_n_] in(4 5) then r[_n_] = 1;
else r[_n_] = c[_n_];
end;
run;
proc print;
run;
It would be nearly trivial to write a macro to parse almost that exact syntax.
I wouldn't necessarily use this - I like both the transpose and the array methods better, both are more 'SASsy' (think 'pythonic' but for SAS) - but this is more or less exactly what you're doing above.
First set up a dataset:
data class;
set sashelp.class;
age_ly = age-1;
age_ny = age+1;
run;
Then the macro:
%macro do_count(data=, out=, prefix=, condition=, recode=, else=, var_start=);
%local dsid varcount varname rc; *declare local for safety;
%let dsid = %sysfunc(open(&data.,i)); *open the dataset;
%let varcount = %sysfunc(attrn(&dsid,nvars)); *get the count of variables to access;
data &out.; *now start the main data step;
set &data.; *set the original data set;
%do i = 1 %to &varcount; *iterate over the variables;
%let varname= %sysfunc(varname(&dsid.,&i.)); *determine the variable name;
%if %upcase(%substr(&varname.,1,%length(&var_start.))) = %upcase(&var_start.) %then %do; *if it matches your pattern then recode it;
&prefix.&varname. = ifn(&varname. &condition., &recode., &else.); *this uses IFN - only recodes numerics. More complicated code would work if this could be character.;
%end;
%end;
%let rc = %sysfunc(close(&dsid)); *clean up after yourself;
run;
%mend do_count;
%do_count(data=class, out=class_r, var_start=age, condition= > 14, recode=1, else=0, prefix=p_);
The expression (1/4=1) means values {1,2,3,4} should be recoded into
1.
Perhaps you do not need to make new variables at all? If have variables with values 1,2,3,4,5 and you want to treat them as if they have only two groups you could do it with a format.
First define your grouping using a format.
proc format ;
value newgrp 1-4='Group 1' 5='Group 2' ;
run;
Then you can just use a FORMAT statement in your analysis step to have SAS treat your five level variable as it if had only two levels.
proc freq ;
tables eq: ;
format eq: NEWGRP. ;
run;
I have some data sets that I wrote some code to clean according to some methods according to some biological literature and then I want to split it into day and night (because they must be analyzed separately). It worked but now I need to do this for the full set which is WAY to many files for me to want to deal with one by one. So I am now trying to write a macro to split it into days and nights for me..
My data looks like so
Hour var1 var2 var3
1 123 90 100
2 122 99 108
...........
4 156 80 120
4 156 80 145
4 143 82 132
basically night has 1 obs per hour day 3. I also have this for many days.
Each dataset is named STUDYIDID#_first or STUDYID_ID#_last. I want to generate four datasets per dataset.
So MYID111_first would create: MYID111_first_day_var1, MYID111_first_day_var2, MYID111_first_night_var1 , and MYID111_first_night_var2.
I would then LIKE to append them into 4 datasets:
MYID_A_first_day_var1, MYID_A_first_day_var2, MYID_A_first_night_var1 , and MYID_A_first_night_var2.
MY CODE SO FAR:
%macro datacut(libname,worklib=work, grp = _A ,time1 = _night , time2 = _day type1 = _var1 , type2 = _var2);
%local num i;
proc datasets library=&libname memtype=data nodetails;
contents out=&worklib..temp1(keep=memname) data=_all_ noprint;
run;
data _null_;
set &worklib..temp1 end=final;
by memname notsorted;
if last.memname;
n+1;
call symput('ds'||left(put(n,8.)),trim(memname));
if final then call symput('num',put(n,8.));
run;
%do i=1 %to #
/* do the artifact removing method */
DATA &libname..&&ds&i;
SET &libname..&&ds&i;
PT_ID = '&ds&i' ;
IF var1< 60 OR var1> 230 then delete;
IF var2< 30 OR var2> 230 THEN delete;
IF var3< 60OR var3 > 135 THEN DELETE;
IF var2 > var1 then delete;
run;
/* get just the night values */
PROC SQL;
CREATE TABLE &libname..&&ds&i&time1 as
SELECT *
FROM &libname..&&ds&i
WHERE Hour BETWEEN 0 and 6 OR Hour BETWEEN 22 and 24
order by systolic
;
QUIT;
/* trim off the proper number of observations for variable 1 */
DATA &libname..&&ds&i&time1&type1;
SET &libname..&&ds&i&time1 end=eof;
IF _N_ =1 then delete;
if eof then delete;
run;
PROC append base= &libname..&&ds&time1&type1
data= &libname..&&ds&i&time1;
run;
QUIT;
%end;
%mend datacut;
%datacut(work)
Now the initial datastep works correctly but the later ones don't rename the data as planned. I get a bunch of datasets called Ds10_night_var1 with the wrong field names (memtype, nodetails, data)
I get the warning:
WARNING: Apparent symbolic reference DS1_NIGHT not resolved.
NOTE: Line generated by the macro variable "TIME1".
1 work.&ds1_night
-
22
200
ERROR 22-322: Expecting a name.
ERROR 200-322: The symbol is not recognized and will be ignored.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE SQL used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds
WARNING: Apparent symbolic reference DS1_NIGHT_SYS not resolved.
22: LINE and COLUMN cannot be determined.
NOTE 242-205: NOSPOOL is on. Rerunning with OPTION SPOOL might allow recovery of the LINE and
COLUMN where the error has occurred.
ERROR 22-322: Syntax error, expecting one of the following: a name, a quoted string, /, ;,
_DATA_, _LAST_, _NULL_.
201: LINE and COLUMN cannot be determined.
NOTE: NOSPOOL is on. Rerunning with OPTION SPOOL might allow recovery of the LINE and COLUMN
where the error has occurred.
ERROR 201-322: The option is not recognized and will be ignored.
So I want the right names for my file AND my datasets to actually have data I and I don't understand why they don't.
As you know, you write a macro variable as & followed by its name, optionally followed by .. With this . you can explicitly end the macro variable reference, so you can use a macro variable as a prefix, like in
%let prefix = fore;
Aspect = &prefix.Ground;
which evaluates to Aspect = foreGround;. And that is why
%let myLib = abc;
%let mymember = xyz;
data &myLib.&myMember;
is an error, as it evaluates to data abcxyz;, and you must write
data &myLib..&myMember;
** or as I prefer **;
data &myLib..&myMember.;
to get data abc.xyz;.
For the case you need macro variables to create macro variable names from, SAS allows to write double ampersands &&, which evaluate to a single & and continues evaluating until all ampersands are consumed. So suppose
%let i = 1;
%let ds1 = myData;
%let time = _nigth;
This is how SAS evaluates &&ds&i&time1 :
&&ds&i&time1
&ds1_night
ERROR because a macro variable ds1_night is not defined
This is how SAS evaluates &&ds&i..&time1. :
&&ds&i..&time1.
&ds1._night
myData_night
I am trying to create a new variable by assigning a format to an existing variable. I'm doing this from within a macro. I'm getting the following error: ": Expecting a format name." Any thoughts on how to resolve? Thanks!
/* macro to loop thru a list of vars and execute a code block on each. This is working fine. */
%macro iterlist
(
code =
,list =
)
;
%*** ASSIGN EACH ITEM IN THE LIST TO AN INDEXED MACRO VARIABLE &&ITEM&I ;
%let i = 1;
%do %while (%cmpres(%scan(&list., &i.)) ne );
%let item&i. = %cmpres(%scan(&list., &i.));
%let i = %eval((&i. + 1);
%end;
%*** STORE THE COUNT OF THE NUMBER OF ITEMS IN A MACRO VARIABLE: &CNTITEM;
%let cntitem = %eval((&i. - 1);
%*** EXPRESS CODE, REPLACING TOKENS WITH ELEMENTS OF THE LIST, IN SEQUENCE;
%do i = 1 %to &cntitem.;
%let codeprp = %qsysfunc(tranwrd(&code.,?,%nrstr(&&item&i..)));
%unquote(&codeprp.)
%end;
%mend iterlist;
/* set the list of variables to iterate thru */
%let mylist = v1 v2 v3 v4;
/* create a contents table to look up format info to assign in macro below*/
proc contents data=a.recode1 noprint out=contents;
run;
/* macro to create freq and chisq tables for each var */
%macro runfreqs (variabl = );
proc freq data=a.recode1 noprint ;
tables &variabl.*improved /out=&variabl._1 chisq;
output out=&variabl.chisq n pchi ;
run;
/* do some more stuff with the freq tables, then grab format for variable from contents */
data _null_;
set contents;
if name="&variabl." then CALL SYMPUT("classformat", format);
run;
data &variabl._3;
length classvalue $ 30 ;
set &variabl._2; ;
/* output a new var using the macro variable for format that we pulled from contents above. Here's where the error occurs. */
classvalue=put(class, %quote(&classformat.));
run;
%mend runfreqs;
* run the macro, iterating thru var list and creating freq tables;
%ITERLIST(list = &mylist., code = %nrstr(%runfreqs(variabl = ?);));
Just guessing, the line
classvalue=put(class, %quote(&classformat.));
should be
classvalue=put(class, &classformat..);
Two points because one is "eaten" by macro processor to mark end of macro variable name, the second one is needed to complete format name.
I believe you won't need %quote() in your case - format name cannot contain strings quoted by %quote().
EDIT: Again not tried, just based on the code I see you also need to change CALL SYMPUT("classformat", format);
to CALL SYMPUTX("classformat", format);
CALL SYMPUTX() is advanced version of CALL SYMPUT(), it removes trailing blanks in macro variable value while the original version keeps blanks. Effectively this will be same as your solution, just simpler.
So the problem is indeed with extra blanks between format name and the period.
No idea why this works and vasja's idea wouldn't, but the problem was clearly with the period on the end of the format name (or perhaps some extra white space?). I changed the data step to add the period before the SYMPUT call:
data _null_;
set contents;
myformat=catt(format,'.');
if name="&variabl." then CALL SYMPUT("classformat", myformat);
run;