First time poster! Im trying to get all of the .csv files linked on this website https://promo.betfair.com/betfairsp/prices however when i use the wget function it doesnt actually get the files (creates a prices file which opens as txt of the website).
wget -r --no-parent -e robots=off https://promo.betfair.com/betfairsp/prices
Any ideas or help would be greatly appreciated!
Thanks!
I suggest there is a way to download all the csv files from the link that you have provided.
use the developer console to run a JavaScript to obtain all the href links in the site.
NB : since all the links in this particular site is of the csv file.we will be only getting the csv files link.
var urls = [];
for(var i = document.links.length; i --> 0;)
if(document.links[i].hostname === location.hostname)
urls.push(document.links[i].href);
now we have a js array with the links (urls) convert it to json for downloading as text file
function download(content, fileName, contentType) {
var a = document.createElement("a");
var file = new Blob([content], {type: contentType});
a.href = URL.createObjectURL(file);
a.download = fileName;
a.click();
}
download(urls, 'json.txt', 'text/plain');
Format the text file so that we can perform a batch wget operaion on it.
wget -i json.txt to download the csv files
Related
I want to use a JSON file as temp data saving and edit it.
this is how I get data from local, I want to get file from internet and edit it.
var mydata = json.decode(await rootBundle.loadString('/assets/testjson.json'));
thanks in advance.
As with others who have posted in the past, I cannot figure out to download a csv.gz file from a URL in Python 3.7. I see posts but they only post a 2kb file.
I am a 100% newbie using Python. What follows is the code for one file that I am trying to obtain. I can't even do that. The final goal would be to request all files that start with 2019* using python. Please try the code below to save the file. As others stated, the file is just a name without the true content - Ref: Downloading a csv.gz file from url in Python
import requests
url = 'https://public.bitmex.com/?prefix=data/trade/20191026.csv.gz'
r = requests.get(url, allow_redirects=True)
open('20191026.csv.gz', 'wb').write(r.content)
Yields:
Out[40]:
1245
I've tried "wget" and urllib.request along with "urlretrieve" also.
I wish I could add a screenshot or attach a file. The file created is 2kb and not even a csv.gz file. But the true file that I can download from a web browser is 78mb. The file is 20191026.csv.gz not that it matters as they all do the same thing. The location is https://public.bitmex.com/?prefix=data/trade/
Again, if you know of a way to obtain all the files using a filter such that 2019*csv.gz would be fantastic.
You are trying to download the files from https://public.bitmex.com/?prefix=data/trade/.
To achieve your final goal of download all the files starting from 2019* you have to do in 3 steps
1) you read the content of https://public.bitmex.com/?prefix=data/trade/
2) convert the content into an list, from that filter out the file names which starting from 2019.
3) from the result list try to download the csv using the example which you referring.
Hope this approach will help you
Happy coding.
Product CSV created in var folder but it's not getting the download. Or CSV file shows 302 error.
I tried all the solutions given for not working export. is there any other way?
1.expected result: It should download a CSV file
2.actual result: It only generates a file in the var folder but not getting a download.
You have to remove deprecated "modules_disable_output" values from core_config_data,
Before run below query please take backup of core_config_data table.
DELETE FROM core_config_data WHERE path LIKE
"advanced/modules_disable_output/%";
And clear cache
Hope it will work for you.
I have form with fields like
'name, firstname, street, zip, city and image'.
The image was an upload
field. Everything works fine, i can upload the image as Filereferrence. But the image was uploaded every time into "fileadmin/". I want to upload the image into "/uploads/<extensionname>" or "fileadmin/user_uploads/<extensionname>".
The part where the upload and file moving was realized look like.
$storageRepository = $this->objectManager->get('TYPO3\\CMS\\Core\\Resource\\StorageRepository');
$storage = $storageRepository->findByUid('1');
$fileData = array();
$fileData['name'] = $_FILES['tx_oaevents_eventslisting']['name']['image'][0];
$fileData['type'] = $_FILES['tx_oaevents_eventslisting']['type']['image'][0];
$fileData['tmp_name'] = $_FILES['tx_oaevents_eventslisting']['tmp_name']['image'][0];
$fileData['size'] = $_FILES['tx_oaevents_eventslisting']['size']['image'][0];
$newFileObject = $storage->addFile(
$fileData['tmp_name'], $storage->getRootLevelFolder(), $fileData['name']
);
This line deines the storage folder form database:
$storage = $storageRepository->findByUid('1');
which is fileadmin as default. But what is the best way to change/modify this storage folder destination?
If you want to specify the destination folder for your upload, I suggest you take a look at the FAL File Upload example from Helmut Hummel. For his example, there is also a blogpost which describes how the file upload works and why you should use a TypeConverter for FAL file upload in TYPO3 6.2+
I have implemented the solution several times and it works just great, since it covers a lot of scenarios you may run into when implementing a file upload (e.g. validation, error handling, file replacement if file exists)
I need to download a lots of data from Ftp server. Now I want to use matlab for this task
My ftp address is
http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2008.03.17/
Filenames are:
"MOD09GA.A2008077.h23v05.005.2008080122921.hdf" "MOD09GA.A2008077.h22v05.005.2008080122814.hdf "
Data will get saved in a folder in drive E.
I want to download the file using Matlab. In this way I want to download data from the ftp server.
Waiting for your kind help.
Thanks in Advance
thank you for your code.but there is a problem.i just need 2data that are below: "MOD09GA.A2008077.h23v05.005.2008080122921.hdf" "MOD09GA.A2008077.h22v05.005.2008080122814.hdf "
but i need just this ID of that data, to eliminate the extra letters:
"MOD09GA.A2008077.h23v05.hdf"
"MOD09GA.A2008077.h22v05.hdf "
I want to have 2 data through 313 data. how can do that?
please help.
Technically, you're downloading using HTTP, not FTP.
You can have Matlab copy the contents of the URL to a local file:
url = 'http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2008.03.17/MOD09GA.A2008077.h23v05.005.2008080122921.hdf'
urlwrite(url, 'tmp.hdf');
then to see what's in the file:
hdfinfo('tmp.hdf')
As dpwe already said, you are not technically downloading via FTP.
However here is a solution tailored to your case, which first gets all the filenames you want, namely the ".*hdf" files. It then just loops over all found *.hdf files and downloads them to 'localPath'.
This is definitely not the easiest or cleanest way to do this, but it works and it should fit your needs, I hope.
% URL you want to grab from - has to end with a /
URL = 'http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2008.03.17/';
% Local path on your machine
localPath = 'E:/myfolder/';
% Read html contents and parse file names with ending *.hdf
urlContents = urlread(URL);
ret = regexp(urlContents, '"\S+.hdf"', 'match');
% Loop over all files and download them
for k=1:length(ret)
filename = ret{k}(2:end-1);
filepathOnline = strcat(URL, filename);
filepathLocal = fullfile(localPath, filename);
urlwrite(filepathOnline, filepathLocal);
end