SAS macro do loop-- import multiple flat files - import

I have a list of 17 flat files that I'm trying to import into different data sets. All of the files have the same data step, so I'm trying to write a do while loop to import all the files.
I've been trying to adapt some code from here without success:
http://www.sas.com/offices/europe/uk/support/sas-hints-tips/tips-enterprise-csv.html
http://support.sas.com/documentation/cdl/en/mcrolref/61885/HTML/default/viewer.htm#a000543785.htm
I'm getting an error that says the %do statement is not valid in open code. Here is my code:
% let path1 = 'c:\path1'
% let path2 = 'c:\path2'
...
% let pathN = 'c:\pathN'
%let n=1;
%do %while (&n <= 17);
%let pathin = &path&n;
data retention&n;
infile &pathin;
<data step-->
run;
%let n=%eval(&n+1);
%end;
I've tested the data step outside of the do-while loop and it works fine for 1 file at a time using the %let pathin = &path&n code. The code still writes the datafile for the 1st data set; but, I need to be able to loop through all the files and can't figure out how. Sorry if this is a novice question; I'm just learning SAS.
Thanks,
-Alex

Welcome to SAS programming! The error message you got is a clue. "Open code" refers to statements that are executed directly by the SAS system. A %do statment is part of the SAS Macro Language, not "normal" SAS. A %let statement can be executed in open code and is use to create a macro variable (distinct from a compiled macro).
Compiled SAS macros are created by code that appears between the %macro and %mend statements. For example, using your code:
%macro run_me;
%let n=1;
%do %while (&n <= 17);
%let pathin = &path&n;
data retention&n;
infile &pathin;
<data step-->
run;
%let n=%eval(&n+1);
%end;
%mend;
But all that does is define/compile the macro. To execute it, you must issue the statement %run_me;. Note that the name run_me was just a name I made up.
For more info, please consult the SAS Macro Reference, especially the introductory section.

To convert your progma to macro, turn your macro variables declared by LET statement into macro arguments:
%macro readfile(n, pathin);
data retention&n;
infile &pathin;
<data step-->
run;
%mend;
A data step to repetitively call your macro.
Here the data included in CARDS statement, but also can be read from some table via SET statement.
The macro call is performed via call execute routine.
data _null_;
length path $200 stmt $250;
input path;
stmt = catt('%readfile(', putn(_N_, 3. -L), path, ')');
call execute(stmt);
cards;
c:\file1.txt
c:\file2.txt
c:\file3.txt
;
run;

Related

Output SAS macro to file or email attachment

I have some problems with getting basic file information send to me in an email.
From various scripts on the internet I have ended up with the following macro which works for outputting the information I want.
%macro list_files(dir);
%local filrf rc did memcnt name i;
%let rc=%sysfunc(filename(filrf,&dir));
%let did=%sysfunc(dopen(&filrf));
%let ymd = %sysfunc(putn(%sysfunc(today()).,yymmdd6.));
%if &did eq 0 %then %do;
%put Directory &dir cannot be open or does not exist;
%return;
%end;
%do i = 1 %to %sysfunc(dnum(&did));
%let name=%qsysfunc(dread(&did,&i));
%if %qscan(%qscan(&name,1,_,b),1,.) eq &ymd. %then %do;
%let rc=%sysfunc(filename(fref,&dir\&name));
%let fid=%sysfunc(fopen(&fref));
%let CreatedDT=%qsysfunc(finfo(&fid,Create Time));
%let ModifiedDT=%qsysfunc(finfo(&fid,Last Modified));
%put &dir\&name,&CreatedDT,&ModifiedDT;
%let fid=%sysfunc(fclose(&fid));
%let rc=%sysfunc(filename(fref));
%end;
%end;
%let rc=%sysfunc(dclose(&did));
%let rc=%sysfunc(filename(filrf));
%mend list_files;
%list_files(C:\logs\);
Notice: The macro does expect a certain file name format in order to work. If I for example have the following log files:
ProgramA_210215.log
ProgramA_210214.log
ProgramA_210213.log
ProgramB_210214.log
ProgramB_210213.log
ProgramC_210215.log
ProgramC_210214.log
ProgramC_210213.log
... and I run this on 2021-02-15, then I only get:
ProgramA_210215.log
ProgramC_210215.log
... which is perfect for my needs.
My problem is, that I can't figure out how to either send the output to me as an email or dump the output in a file, which I can then attach to an email.
Notice 2: I initially wrote the macro as "normal data calls"(?) and everything worked except for the CreateDT and ModifiedDT, which were static for all files. The macro version at least works and shows the correct (and different) CreateDT and ModifiedDT.
What I am trying to achieve in the end is, to send me an email with an attachment of todays log files and their creation datetime and last modified datetime.
I guess I'm just missing a simple step, but can't figure out which :(.
Calling all of those functions using %SYSFUNC() is just making it harder to to do what you want. Just call the functions in a data step instead so that you have an actual dataset with the files. Then you can use any of multiple methods to direct the list where ever you want, including sending an email.
I am pretty sure that FINFO() always returns character strings so make sure that the target variables are long enough for the value returned. Note that the information available from FINFO() depends on the operating system where SAS is running. So make sure the names you use are appropriate for your SAS session.
%let dir=c:\logs\;
data file_list;
length filrf fref $8 name $256 ymd $6 CreatedDT ModifiedDT $40;
ymd = put(today(),yymmdd6.);
rc=filename(filrf,"&dir");
did=dopen(filrf);
if did=0 then do;
put "Directory &dir cannot be open or does not exist.";
stop;
end;
do i = 1 to dnum(did);
name=dread(did,i);
if scan(scan(name,1,'_','b'),1,'.') eq ymd then do;
rc=filename(fref,"&dir\"||name);
fid=fopen(fref);
CreatedDT=finfo(fid,'Create Time');
ModifiedDT=finfo(fid,'Last Modified');
output;
fid=fclose(fid);
rc=filename(fref);
end;
end;
rc=dclose(did);
rc=filename(filrf);
keep name CreatedDT ModifiedDT ;
run;

SAS macro loop to read csv files with proc import

I have a directory of csv files, each with names that begin with the letter m and end with a number. There are twelve files - m6 to m17.
I'd like to read them in and process them as separate data sets. I've written two macros attempting to do so. Macro1 works. Macro2 breaks. I would prefer Macro2 if I can get it to work, to avoid unnecessary bits like my creation of %rawfiles, invocation of %sysfunc, etc.
Macro 1:
%let rawcsv = C:\ALL\dat\;
%let rawfiles = m6 m7 m8 m9 m10 m11 m12 m13 m14 m15 m16 m17;
%macro1;
%do i = 1 %to %sysfunc(countw(&rawfile));
%let rawfile = %scan(&rawfiles, &i);
proc import datafile="&&rawcsv.&&rawfile.csv"
out=&rawfile replace
dbms=csv;
guessingrows=500;
run;
%end;
%mend;
%macro1;
Macro 2:
%let rawcsv = C:\ALL\dat\;
%macro macro2(first=6, last=19);
%do i=&first. %to &last. %by 1;
proc import datafile="&&rawcsv..m&&i.csv"
out=m&i replace
dbms=csv;
guessingrows=500;
run;
%end;
%mend;
%macro2;
%macro2 is my bad imitation of this solution. It returns the following errors:
MPRINT(MACRO2): proc import datafile="C:\ALL\dat\m.6.csv" out=m.6 replace
dbms=csv;
MPRINT(MACRO2): ADLM;
MPRINT(MACRO2): guessingrows=500;
MPRINT(MACRO2): run;
ERROR: Library name is not assigned. /*repeats this error 14 times, once per file*/
Two questions:
What am I missing in %macro2?
Do you see a better solution that I am not using? The files are structured differently and not stackable, just a heads up.
From your log we can see a period is being inserted into the output dataset name. Just remove that extra period in your macro definition.
MPRINT(MACRO2): proc import datafile="C:\ALL\dat\m.6.csv" out=m.6 replace dbms=csv;
The extra & in the code is probably confusing you. When the macro processor sees two & it converts them to one and then reprocesses the string to further resolve the resulting macro variable references.
The period after a macro variable name is not required when the macro processor can tell that the name has ended. But the periods are needed in some places.
One place in your code is where it is required to make sure the macro processor knows where the name ends (the macro variable is named readcsv not readcsvm ). Another is where you want to place an actual period after the value of a macro variable. You will need to place two periods there since the first will be used by the macro processor when it evaluates the macro variable value.
In this version of macro2 I have removed the periods after the macro variable names in the places where they are not required just to emphasize the places where the period is required.
%let rawcsv = C:\ALL\dat\;
%macro macro2(first, last);
%local i ;
%do i=&first %to &last ;
proc import dbms=csv
datafile="&rawcsv.m&i..csv"
out=m&i replace
;
guessingrows=500;
run;
%end;
%mend macro2;
%macro2(first=6, last=19)
Small typo here, you need to use an & in front of LAST not the %.
%do i=&first. %to %last. %by 1;
Should be:
%do i=&first. %to &last. %by 1;
Unless you're using a separate macro called last to determine your end of the loop. But in that case you likely wouldn't also have a parameter called last.
If you're looking for alternate options I usually recommend reading all at once using a data step or CALL EXECUTE instead of macro loops as they're infinitely easier to debug in my opinion.
https://communities.sas.com/t5/SAS-Communities-Library/How-do-I-write-a-macro-to-import-multiple-text-files-that-have/ta-p/223627

Passing SAS dataset column as macro parameter

I have a SAS dataset with values
yyyymm
201605
201606
201607
201608
201609
I am trying to find a way to pass these values one at a time to macro such that
do while dataset still has value
%macro passdata(yyyymm);
end
How can I do this in SAS. Can someone please help with a sample/code snippet.
As mentioned by the prior comment, a way to pass parameters is through call execute routine. Note that this must be done in datastep environment. The lines are read from the set you input.
You can input multiple variables. Just add more variables in '||' separators. Note that the variables may have a lot of whitespaces in them. (==Do comparisons with care.)
Here is a small sample code. Tested.
data start_data;
input date_var ;
datalines;
201605
201606
201607
201608
201609
;
run;
%macro Do_stuff(input_var);
%put 'Line generates value ' &input_var;
%mend do_stuff;
data _null_;
set start_data;
call execute('%do_stuff('||date_var||')' );
run;
Try this example and try modifying to meet your needs... From the "source" dataset we can use call symput() to assign a macro token to each observation (differentiated by the SAS automatic dataset variable n so My_token1, My_token2, etc.) Once you have a set of macro variables defined, just loop through them! This program will print all the individual records from source to the SAS log:
data source;
do var=1000 to 1010;
output;
end;
run;
data _null_;
set source;
call symput(compress("My_token"||_n_),var);
run;
%put &my_token1 &my_token4;
%Macro neat;
%do this=1 %to 11;
*Check the log.;
%put &&My_token&this;
%end;
%mend;
%neat;

resolve macro variables for saving/naming a dataset in SAS

I have problem saving a dataset using macro variables to a desired directory.
Basically, I want to save the dataset "_est" to library "sret" according to the values of &var and &age. I wrote the following code:
%let var=k;
%let age=2;
...
...
data sret.est_&var&age._b3;
set _est;
run;
What I want is a dataset named as "est_k2_b3.sas7bdat" in "sret". But what the code gives me is a dataset "est_k2.sas7bdat" saved in the folder I want and another dataset "_b3" in the working library. Both datasets are identical. I'm quite puzzled how to solve this.
As itzy pointed out you have a space after "2" that splits your dataset name in two.
I can replicate the issue only defining the macro variable age with a call symput:
data _null_;
age='2 ';
call symput('age',age);
run;
If this is the case you can solve it by removing the space in the data step with a strip(), using a call symputx() (to be used with numbers) or re-declaring your variable after the data step with a %let, that automatically removes spaces:
%let age= &age.;
Had a very similar problem. Somehow a space is added until you use strip(). Here's the example below.
data test;
input numdays;
datalines;
31
;
%macro monthly(months);
%let count=%sysfunc(countw(&months.));
%do i=1 %to &count.;
%let value=%qscan(&months.,&i,%str(,));
%let month=%sysfunc(strip(&value.));
%put &value.;
%put &month.;
data value_&value.;
set test;
run;
data month_&month.;
set test;
run;
%end;
%mend;
%monthly(%str(oct,jan));

How to write macro for importing multiple excel files(xlsx) in sas and append it

I am having some 50 excel files (xlsx format) to be imported to sas and then need to append it for analysis. All the excel files header are same i.e., the variable names are same for all the file. I need macro for importing and appending all of them at a time rather than importing all the files one by one and then later append it. Your help is much appreciated.
The other issue with the excel file is that there is a blank column next between the variable name and data points. I have written a code remove it using data step but came we write this also in the macro while importing.
Data XXX.yyy;
Set XXX.yyy;
if missing(coalesceC(of ASC Brand Cdesc1 Cust_ DGM Desc Family Grp1 High_Level_Product_Desc
Issf Name Prod_Desc Product__Code RVP SA_Desc Terr_ UOM Yr
)) and missing(coalesce(of Acc Int_Margin M_Cost Mth Net_Sales Sls__ Uts )) then delete;
run;
It sounds as though your existing code already does what you need it to do. I doubt there will be much of a performance gain from attempting to import all 50 files in one data step (which is possible via dde, but rather fiddly).
If your existing code is set up to process just one hard-coded file, I'd suggest using it to write a simple macro that takes one excel file as input, imports that file, and appends it to the master dataset. Then you can call the macro 50 times.
e.g. You could write the macro as something like this, incorporating all the relevant bits of your code, and replacing all references to specific files with macro variables:
%macro import_and_append(excel_file,base_dataset);
proc import datafile = "&excel_file" dbms = excel out = t_import;
run;
proc append base = &base_dataset data = t_import;
run;
proc datasets lib = work nolist nowarn;
delete t_import;
run;
quit;
%mend;
Then you can call the macro like so:
%import_and_append(c:\excel_file_01.xls,work.master_dataset)
Another way to do this would be to use the Excel LIBNAME Engine. You declare a library to each of your files, and then read all the sheets in 1 Data Step.
In this example, I have 2 workbooks (Book1.xlsx and Book2.xlsx) in C:\temp. All data is in Sheet1. 3 variables -- X, Y, and Z. Modify as needed for your purpose.
data files;
format file $12.;
input file $;
datalines;
Book1.xlsx
Book2.xlsx
;
run;
%macro read_excel(dir,outdata,files);
data _null_;
set &files end=last;
call execute("libname t" || strip(put(_n_,8.)) || " excel '&dir\" || strip(file) || "';");
if last then
call symput("n",_n_);
run;
data &outdata;
set
%do i=1 %to &n;
t&i.."Sheet1$"n
%end;
;
a = sum(x,y,z);
if missing(a) then delete;
run;
%do i=1 %to &n;
libname t&i clear;
%end;
%mend;
%read_excel(c:\temp,data_from_excel,files);