Jump to content

nsa-dl


cwade12c
 Share

Recommended Posts

Download all of the released NSA documents (continuously updating) with two scripts. Very hacky, but gets the job done. DEPENDS ON LYNX. (Why? Because I'm lazy)

$ apt install lynx

 

nsadl.sh

#!/bin/bash

echo 'Scraping links from Primary Sources...'
lynx -dump "https://www.eff.org/nsa-spying/nsadocs" | grep "https://www.eff.org/document" | awk '/http/{print $2}' > links
echo 'Done. Links saved as "links.txt"'
echo 'Downloading .pdf documents using "links.txt" -- this may take awhile...'
while read line
do
    name=$line
    sh scraper.sh $name
done < links
echo 'All done!'

scraper.sh

#!/bin/bash          
STR="`wget --quiet -O - $1 | grep -Eo 'https://www.eff.org/files/[0-9]+/[^"]+\.pdf';`"
wget --no-clobber --quiet $STR

 

Usage:

$ sh nsadl.sh; echo 'Have fun!'

 

  • I Like This! 2
Link to comment
Share on other sites

Any way there could be another version of this script that uses tor as a socks5 proxy using proxychains to grab the doc's in a more low key way?

Could the editing of the config for proxychains also be automated into the script? Really all the end user would need to do at that point is open up the tor browser bundle, not to mention they could edit the script to grab anything from any site with a  bit more privacy.

Just a thought.

EFF FTW!

  • I Like This! 1
Link to comment
Share on other sites

18 minutes ago, killab said:

Any way there could be another version of this script that uses tor as a socks5 proxy using proxychains to grab the doc's in a more low key way?

Could the editing of the config for proxychains also be automated into the script? Really all the end user would need to do at that point is open up the tor browser bundle, not to mention they could edit the script to grab anything from any site with a  bit more privacy.

Just a thought.

EFF FTW!

Hmmm....I will think about this a little bit and see what I can whip up. I think I can make the script accept an option for proxychains, and if the option is passed, then prepend proxychains to the wget commands. For the native SOCKS support, could probably just use curl instead of wget, as I don't think wget supports SOCKS proxies.

Thanks for the tips!

Link to comment
Share on other sites

The addy for tor when using it as a socks5 would be:

 

127.0.0.1  9150

 

Basically all you would need to do is edit this in proxychains.conf, from my understanding.

When trying to wget on a .onion I have had issues (grabbing PHP configs of DNM's FTW),

because 1. you dont know the IP of the hidden service, and 2. wget will try to use DNS (obviously) to

locate the site, this does not works so well in tor, but with a clearnet site, it's possible.

 

The only thing the end user would have to do is run the tor web browser before attempting this or install tor manual via terminal.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...