can this scraper plugin scrape a website that requires a password set via js?

Started by wowza, November 27, 2012, 04:07:23 AM

wowza

I have a membership to a site that needs a username/password.

Everything in the site, once I am logged in sets me as a master seller with resell rights, so I have 100% permission to download the zip files from the site. That is what I paid for.

the problem is there is about 1700 pages, and I do not want to sit here for 2 days clicking button after button to download all of the zip files, that is why i got this scraper program.

Here is the site:
http://bit.ly/U92AiI

So can this scraper plugin let me login and then simply download all the zip files?
* I do not want anything else but the zips, no need *

thanks

Webhelpforums

TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

wowza

I am so confused.

Can you do me one favor and try to use the link for the website I gave in my first post and let me know how I am supposed to login to that site using the website download tool?

I followed the guides the best I could but I do not understand which type of authentication I am supposed to use for that website.

If you could just let me know what settings to add to your program, then I can simply add in the username and password I have and see if it works.

Or I could just send you the username/pass, tried that but can't pm via your profile, so not sure how.

Your plugin looks great, but it's hard to figure out and this seems like a basic use case scenario.

thanks so much

wowza

Here is the form data


<form method="post" id="loginForm" name="loginForm">


<input type="text" maxlength="60" size="30" name="nicknameLogin">
<input type="password" maxlength="60" size="30" name="password">
<a onmouseout="this.className='button-ltblue'; " onblur="this.className='button-ltblue';" onclick="javascript:document.getElementById('loginForm').submit();" onmouseover="javascript:this.className='button-black';" href="javascript:void(0);" class="button-ltblue">
<a onmouseout="this.className='button-ltblue'; " onblur="this.className='button-ltblue';" onclick="javascript:document.getElementById('loginForm').reset();" onmouseover="javascript:this.className='button-black';" href="javascript:void(0);" class="button-ltblue">

<input type="hidden" value="http://website.com/99products/members/index.php" name="redirectMember">
<input type="hidden" value="         Login         " name="login">
<input type="hidden" value="login" name="action">
<input type="hidden" value="1" name="retry">

</form>

Webhelpforums

Using the method outlined in section:
Configure Login by Using FireFox Live HTTP Headers

Described at help page
http://www.microsystools.com/products/website-download/help/website-scan-login/

The configuration should be something like this:

Login Path (post form):
/99products/members/index.php

Login paramaters (post form):
nicknameLogin=YOURUSERNAME&password=YOURPASSWORD&redirectMember=http%3A%2F%2Fdownloadplrproducts.com%2F99products%2Fmembers%2Findex.php&login=+++++++++Login+++++++++&action=login&retry=1

(obviously you need to correct YOURUSERNAME and YOURPASSWORD in above)


However, without actual user/pass it is hard to test, but there is a good chance above works.

If you continue to have problems you can try email support:
http://www.microsystools.com/home/contact.php

Note #1:
Both A1 Website Download and A1 Website Scraper are invidividual software programs (not "plugins")

Note #2:
This thread is going to be moved to A1 Websie Download section later :)
(A1 Website Scraper is for extracting structured website data into CSV files / databases)
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac