Pages

Get subdirs from wget and getting past robot disallows

I’m going to a place where there is no net, and was thinking of reading the metro manual. A naive approach

$ wget -r https://metro.dev.java.net/guide/

gets me nowhere however, as wget stops pretty fast and leaves a file that states

User-agent: CEE
Disallow:
User-agent: *
Disallow: /

I can do ‘-e robots=off’ mixed with some stuff from  Voelkers excellent wget examples:

wget -P [...]

secure copy which continues where you left off if interrupted

If I’m at liberty, I’ll in 10/10 cases be ssh based when communicating between hosts. Secure, easy, terminal with graphics (ssh -X), copy (scp) , remote filesystem (sshfs) and only one port to keep track of. Large copy operations are a bit annoying over a normal scp however, since if the copy is interrupted for [...]