Wget - downloading all files in a webpage? - wget

Question

I'm using this wget command to download all the .fits files from that URL:
wget -r -np -nd -l inf -A fits https://archive.stsci.edu/missions/tess/ete-6/tid/00/000/000/057/
This is based on an adaptation of this answer.
All I'm getting is a directory structures that mirrors the URI on the website all the way down to /057/, but there's no file.
If I add -nd, then I only get a robot.txt files which isn't very instructive, but still no files.
What am I not getting about how to use wget for this?
EDIT: based on Turgbek's answer below, I do see that the robot.txt file from that website actually has /missions/ in the "Disallow"... maybe this is what is preventing me from using the wget command? Is that the source of the problem? How can I get around that?

Answer 1

In robots.txt there's a statement:
Disallow: /missions/
Which your requested files are in. Since the url builds up as /missions/tess/ete-6/tid/00/000/000/057/ I believe that robots.txt is blocking you.
I've saved two of the files from that URL in my Raspberry Pi and ran a local test without robots.txt. With this command:
wget -r -np -nd -l inf -A fits 192.168.1.250/test/
It worked as intended and I've received both of the files.
--2018-05-03 23:46:51-- http://192.168.1.250/test/tess2019128220341-0000000005712108-0016-s_lc.fits
Reusing existing connection to 192.168.1.250:80.
HTTP request sent, awaiting response... 200 OK
Length: 2090880 (2.0M)
Saving to: `192.168.1.250/test/tess2019128220341-0000000005712108-0016-s_lc.fits'
100%[==============================================================================>] 2,090,880 3.77M/s in 0.5s
2018-05-03 23:46:51 (3.77 MB/s) - `192.168.1.250/test/tess2019128220341-0000000005712108-0016-s_lc.fits' saved [2090880/2090880]
--2018-05-03 23:46:51-- http://192.168.1.250/test/tess2019128220341-0000000005715814-0016-s_lc.fits
Reusing existing connection to 192.168.1.250:80.
HTTP request sent, awaiting response... 200 OK
Length: 2090880 (2.0M)
Saving to: `192.168.1.250/test/tess2019128220341-0000000005715814-0016-s_lc.fits'
100%[==============================================================================>] 2,090,880 4.61M/s in 0.4s
2018-05-03 23:46:52 (4.61 MB/s) - `192.168.1.250/test/tess2019128220341-0000000005715814-0016-s_lc.fits' saved [2090880/2090880]