I need to download a lots of data from Ftp server. Now I want to use matlab for this task
My ftp address is
http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2008.03.17/
Filenames are:
"MOD09GA.A2008077.h23v05.005.2008080122921.hdf" "MOD09GA.A2008077.h22v05.005.2008080122814.hdf "
Data will get saved in a folder in drive E.
I want to download the file using Matlab. In this way I want to download data from the ftp server.
Waiting for your kind help.
Thanks in Advance
thank you for your code.but there is a problem.i just need 2data that are below: "MOD09GA.A2008077.h23v05.005.2008080122921.hdf" "MOD09GA.A2008077.h22v05.005.2008080122814.hdf "
but i need just this ID of that data, to eliminate the extra letters:
"MOD09GA.A2008077.h23v05.hdf"
"MOD09GA.A2008077.h22v05.hdf "
I want to have 2 data through 313 data. how can do that?
please help.
Technically, you're downloading using HTTP, not FTP.
You can have Matlab copy the contents of the URL to a local file:
url = 'http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2008.03.17/MOD09GA.A2008077.h23v05.005.2008080122921.hdf'
urlwrite(url, 'tmp.hdf');
then to see what's in the file:
hdfinfo('tmp.hdf')
As dpwe already said, you are not technically downloading via FTP.
However here is a solution tailored to your case, which first gets all the filenames you want, namely the ".*hdf" files. It then just loops over all found *.hdf files and downloads them to 'localPath'.
This is definitely not the easiest or cleanest way to do this, but it works and it should fit your needs, I hope.
% URL you want to grab from - has to end with a /
URL = 'http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2008.03.17/';
% Local path on your machine
localPath = 'E:/myfolder/';
% Read html contents and parse file names with ending *.hdf
urlContents = urlread(URL);
ret = regexp(urlContents, '"\S+.hdf"', 'match');
% Loop over all files and download them
for k=1:length(ret)
filename = ret{k}(2:end-1);
filepathOnline = strcat(URL, filename);
filepathLocal = fullfile(localPath, filename);
urlwrite(filepathOnline, filepathLocal);
end
Related
I would like to save variables as mat files on s3. The example on the official site shows "tall table" only. Maybe I can use the "system" command overstep MATLAB but I am looking for a straight forward solution.
Any suggestion?
It does look like save does not support saving to remote filesystems.
You can, however, write matrices, cells, tables and timetables.
An example which uses writetable:
LastName = {'Smith';'Johnson';'Williams';'Jones';'Brown'};
Age = [38;43;38;40;49];
T = table(Age,LastName)
writetable(T,'s3://.../table.txt')
Note:
To write to a remote location, filename must contain the full path of
the file specified as a uniform resource locator (URL) of the form:
scheme_name://path_to_file/my_file.ext
To obtain the right URL of the bucket, you can navigate to the contents of the s3 bucket, select a file in there, choose Copy path and remove the name of the file (e.g table.txt).
The alternative is, as you mentioned, a system call:
a = rand(5);
save('matExample','a');
system('aws s3api put-object --bucket mybucket --key=s3mat.mat --body=matExample.mat')
the mat file matExample.mat is saved as s3.mat on the server.
As with others who have posted in the past, I cannot figure out to download a csv.gz file from a URL in Python 3.7. I see posts but they only post a 2kb file.
I am a 100% newbie using Python. What follows is the code for one file that I am trying to obtain. I can't even do that. The final goal would be to request all files that start with 2019* using python. Please try the code below to save the file. As others stated, the file is just a name without the true content - Ref: Downloading a csv.gz file from url in Python
import requests
url = 'https://public.bitmex.com/?prefix=data/trade/20191026.csv.gz'
r = requests.get(url, allow_redirects=True)
open('20191026.csv.gz', 'wb').write(r.content)
Yields:
Out[40]:
1245
I've tried "wget" and urllib.request along with "urlretrieve" also.
I wish I could add a screenshot or attach a file. The file created is 2kb and not even a csv.gz file. But the true file that I can download from a web browser is 78mb. The file is 20191026.csv.gz not that it matters as they all do the same thing. The location is https://public.bitmex.com/?prefix=data/trade/
Again, if you know of a way to obtain all the files using a filter such that 2019*csv.gz would be fantastic.
You are trying to download the files from https://public.bitmex.com/?prefix=data/trade/.
To achieve your final goal of download all the files starting from 2019* you have to do in 3 steps
1) you read the content of https://public.bitmex.com/?prefix=data/trade/
2) convert the content into an list, from that filter out the file names which starting from 2019.
3) from the result list try to download the csv using the example which you referring.
Hope this approach will help you
Happy coding.
I was assigned a matlab assignment where I was given 25000 pictures of cats and dogs all stored in one folder. My question is how can I use the imagedatastore function on matlab to store these files into one single variable containing two labels (cats and dogs). Each image stored in the file follow the following format:
cat.1.png,
cat.2.png,
.....,
cat.N.png,
dog.1.png,
dog.2.png,
.....,
dog.N.png,
Ideally I think labeling them based on image name would probably be the best approach to this. How ever I've tired doing this using various implementations methods but I keep failing. Any advice on this would be greatly appreciated!
The steps for both image data stores are the same:
Find all the image files with a matching name with dir.
Rebuild the full path to these files with fullfile.
Create the image data store with the files.
My code assumes that you are running the script in the same folder in which images are located. Here is the code:
cats = dir('cat.*.png');
files_cats = fullfile({cats.folder}.', {cats.name}.');
imds_cats = imageDatastore(files_cats);
dogs = dir('dog.*.png');
files_dogs = fullfile({dogs.folder}.', {dogs.name}.');
imds_dogs = imageDatastore(files_dogs);
You could also use the short path:
imds_cats = imageDatastore('cat.*.png');
imds_dogs = imageDatastore('dog.*.png');
If you want to use a single image data store and split files into categories within it (without using folder names, since all your files seem to be located in the same directory):
cats = dir('cat.*.png');
cats_labs = repmat({'Cat'},numel(cats),1);
dogs = dir('dog.*.png');
dogs_labs = repmat({'Dog'},numel(dogs),1);
labs = [cats_labs; dogs_labs];
imds = imageDatastore({'cat.*.png' 'dog.*.png'},'Labels',labs);
I am working on an experiment that take a lot of data samples and save it to different video files. For example, I take data and save it to a file called "slowmotion.avi" and then I take another set of data and save it to another file called "normalspeed.avi".
I am trying to find a way that I could change the name of file being saved without editing the code. The way I am using right now makes me have to open the code and change the name of the file directory within the code and then save the code. I want to make it a lot faster and easier to use.
I have tried the following code to fix this problem but it doesn't work.
graph=input('Graph of experiment: ');
vidObj = VideoWriter('%3.1f.avi\n',graph);
.....
Hope I didn't confuse you.
A possible solution:
graph=input('Graph of experiment: ','s');
vidObj = VideoWriter([graph,'.avi']);
The 's' in the input() function indicates that the expected input is a string, and [graph,'.avi'] simply concatenates both strings.
I have written a Matlab script to perform some analysis on audio samples recorded at different locations. I have a mobile app that records audio and stores it on a web server. Is there a way I can access this file from Matlab as an input to the script? The url and individual file names will be available, I assume there is a Matlab command that uses this information.
Thanks,
Try something like this:
str = urlread('http://stackoverflow.com');