Short: Htdump v0.9 - the http protocol dumper tool Author: louise@louise.amiga.hu Type: dev/gg Architecture: m68k-amigaos Uploaded: louise@louise.amiga.hu (LouiSe) Url: http://arak.cs.hro.nl/~ren/linux/ Ported by LouiSe more info and other AMIGA ports at: http://louise.amiga.hu ----------------------------------------- HTDUMP Copyright 2000 by Ren Hoek (ren@arak.cs.hro.nl) Purpose ------- A program to use in a script to easily dump http content in a file. The program will accept links in the standard URL form and will support tweaking options as specifying the referrer address and supporting resuming of files. Importance is put on being sure files get transferred correctly without errors and being able to tweak headers to overcome anti-leeching procedures on websites. Offcourse it's a valuable tool too for web developers like me who work a lot of CGI's.. Ok most of the info is moved into htdump itself.. Try a 'htdump --help | less' and read. Or try the rather complex example right below. On the website there are more examples available. Don't hesitate to email me if you have any questions. (Note.. I need to make this pretty.. and more mainstream perhaps.. dunno.. let's see if anybody reads this first *grin*) Example ------- Netscape has released version 4.72 and this time everybody is allowed to download the 128-bit strong encryption version, except the usual 'evil' people. To this end Netscape has devised a method to protect the download of their program. Since I'm Dutch, I can download this program and don't have to worry about probable domain name checks to see if I'm from an 'evil' country. There are 2 reasons why I wanted to see how this particulair situation worked, because I can't use a download manager like Getright and Netscape froze up on me after it downloaded 89% of the new install version. *grr* And also because of all the check and weird checks, it's a good test of htdump So let's see if we can make it download.. First, let's see the page where we can download it (line below is 1 line, I cut is in bits so you can read it on 80 char monitors) https://wwwus.netscape.com/usdl-bin/pdms_dnstest.cgi? PRODUCT=communicator4.72-win32-en-complete-128& COMPONENTS=CLIENT&TEMPLATES=NSCP If we click 'Agree' a download is automatically started. View the source and see:
     
If we check out the info for this page you will see Form 1: Action URL: https://wwwus.netscape.com/usdl-bin/pdms_promise2.cgi Encoding: application/x-www-form-urlencoded (default) Method: Post Ok now we have a target and we know what we have to post. Only the hidden things get send to the cgi if I'm not mistaken.. SUBMIT's and BUTTON's don't get send.. So make the following command (cut up again, it's 1 line) htdump -d -P "PRODUCT=communicator4.72-win32-en-complete-128&COMPONENTS=CLIENT" https://wwwus.netscape.com/usdl-bin/pdms_promise2.cgi -u "Mozilla/4.05 [en] (Win95; U ;Nav)" I had to fidget around to get it to work though.. it turned out that Netscape also checks on User Agent.. Leave it out and you get an internal server error back. :) So I just inserted something nice :) And we get this result: ------------------------------------- Username: [] Password: [] B64_pass: [] Host : [wwwus.netscape.com] Port : [443] Path : [usdl-bin/pdms_promise2.cgi] Debug : [1] ------------------------------------- POST /usdl-bin/pdms_promise2.cgi HTTP/1.0 User-Agent: Mozilla/4.05 [en] (Win95; U ;Nav) Content-type: application/x-www-form-urlencoded Content-length: 64 PRODUCT=communicator4.72-win32-en-complete-128&COMPONENTS=CLIENT ----------------------------------- SSL connection using RC4-MD5 header: [HTTP/1.1 302 Moved Temporarily] header: [Server: Netscape-Enterprise/3.6] header: [Date: Sun, 12 Mar 2000 00:35:14 GMT] header: [Content-type: text/html] header: [Set-cookie: NSCP-US-DOWNLOAD=1IhWuzjK5kMAAAAASLm0goK_Rm UVRvydzyKhqQ== ; expires=Sunday, 12-Mar-00 04:35:15 GMT ; domain=.netscape.com; secure] header: [Refresh: 0; URL=http://home.netscape.com/download/thanks.html] header: [Location: https:pdms_pick2.cgi/cc32d472.exe?product= communicator4.72-win32-en-complete-128&ext=.exe] header: [] ------------------------------------- Content-length: 0 Errorcode: 302 ------------------------------------- Ok as you can see we get a load of data.. what is interesting is the Cookie and the redirection we get. So a new target (Again this is 1 line, just cut up to make it readable) htdump "https://wwwus.netscape.com/usdl-bin/pdms_pick2.cgi/cc32d472.exe? product=communicator4.72-win32-en-complete-128&ext=.exe" -c "NSCP-US-DOWNLOAD=1IhWuzjK5kMAAAAASLm0goK_RmUVRvydzyKhqQ==" -f cc32i472.exe & This will make it download in the background. Come back a few hours later and it will be there complete. I tried to use Ranges (-R) but it still spit out the same. It's a dumb script that doesn't understand resume unfortunatly.. But even if my primary goal failed I learned of the 'Location' header which I never seen before :) You learn something every day. I think you should be able to use htdump efficiently now. Use the debug option to see what is going on, and experiment. Have fun Ren Hoek (ren@arak.cs.hro.nl)