nsa-dl

[cwade12c]

Download all of the released NSA documents (continuously updating) with two scripts. Very hacky, but gets the job done. DEPENDS ON LYNX. (Why? Because I'm lazy)

$ apt install lynx

 

nsadl.sh

#!/bin/bash

echo 'Scraping links from Primary Sources...'
lynx -dump "https://www.eff.org/nsa-spying/nsadocs" | grep "https://www.eff.org/document" | awk '/http/{print $2}' > links
echo 'Done. Links saved as "links.txt"'
echo 'Downloading .pdf documents using "links.txt" -- this may take awhile...'
while read line
do
    name=$line
    sh scraper.sh $name
done < links
echo 'All done!'

scraper.sh

#!/bin/bash          
STR="`wget --quiet -O - $1 | grep -Eo 'https://www.eff.org/files/[0-9]+/[^"]+\.pdf';`"
wget --no-clobber --quiet $STR

 

Usage:

$ sh nsadl.sh; echo 'Have fun!'

 

[ryoh]

I'm liking the bash scripts. programmers' duct tape FTW!

[killab]

Any way there could be another version of this script that uses tor as a socks5 proxy using proxychains to grab the doc's in a more low key way?

Could the editing of the config for proxychains also be automated into the script? Really all the end user would need to do at that point is open up the tor browser bundle, not to mention they could edit the script to grab anything from any site with a  bit more privacy.

Just a thought.

EFF FTW!

[cwade12c]

2 minutes ago, killab said:

Real men donate to the EFF!

I'd actually like to have HaxMe become a potential Luminary || Innovator level donor in the future.

https://www.eff.org/thanks

 

[killab]

Wow that was bad timing.

For the record I did say that despite my editing of my post.

:)

[cwade12c]

18 minutes ago, killab said:

Any way there could be another version of this script that uses tor as a socks5 proxy using proxychains to grab the doc's in a more low key way?

Could the editing of the config for proxychains also be automated into the script? Really all the end user would need to do at that point is open up the tor browser bundle, not to mention they could edit the script to grab anything from any site with a  bit more privacy.

Just a thought.

EFF FTW!

Hmmm....I will think about this a little bit and see what I can whip up. I think I can make the script accept an option for proxychains, and if the option is passed, then prepend proxychains to the wget commands. For the native SOCKS support, could probably just use curl instead of wget, as I don't think wget supports SOCKS proxies.

Thanks for the tips!

[killab]

The addy for tor when using it as a socks5 would be:

 

127.0.0.1  9150

 

Basically all you would need to do is edit this in proxychains.conf, from my understanding.

When trying to wget on a .onion I have had issues (grabbing PHP configs of DNM's FTW),

because 1. you dont know the IP of the hidden service, and 2. wget will try to use DNS (obviously) to

locate the site, this does not works so well in tor, but with a clearnet site, it's possible.

 

The only thing the end user would have to do is run the tor web browser before attempting this or install tor manual via terminal.