can this scraper plugin scrape a website that requires a password set via js?

Started by wowza, November 27, 2012, 04:07:23 AM

wowza

I have a membership to a site that needs a username/password.

Everything in the site, once I am logged in sets me as a master seller with resell rights, so I have 100% permission to download the zip files from the site. That is what I paid for.

the problem is there is about 1700 pages, and I do not want to sit here for 2 days clicking button after button to download all of the zip files, that is why i got this scraper program.

Here is the site:
http://bit.ly/U92AiI

So can this scraper plugin let me login and then simply download all the zip files?
* I do not want anything else but the zips, no need *

thanks

Webhelpforums

TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

wowza

I am so confused.

Can you do me one favor and try to use the link for the website I gave in my first post and let me know how I am supposed to login to that site using the website download tool?

I followed the guides the best I could but I do not understand which type of authentication I am supposed to use for that website.

If you could just let me know what settings to add to your program, then I can simply add in the username and password I have and see if it works.

Or I could just send you the username/pass, tried that but can't pm via your profile, so not sure how.

Your plugin looks great, but it's hard to figure out and this seems like a basic use case scenario.

thanks so much

wowza

Here is the form data


<form method="post" id="loginForm" name="loginForm">


<input type="text" maxlength="60" size="30" name="nicknameLogin">
<input type="password" maxlength="60" size="30" name="password">
<a onmouseout="this.className='button-ltblue'; " onblur="this.className='button-ltblue';" onclick="javascript:document.getElementById('loginForm').submit();" onmouseover="javascript:this.className='button-black';" href="javascript:void(0);" class="button-ltblue">
<a onmouseout="this.className='button-ltblue'; " onblur="this.className='button-ltblue';" onclick="javascript:document.getElementById('loginForm').reset();" onmouseover="javascript:this.className='button-black';" href="javascript:void(0);" class="button-ltblue">

<input type="hidden" value="http://website.com/99products/members/index.php" name="redirectMember">
<input type="hidden" value="         Login         " name="login">
<input type="hidden" value="login" name="action">
<input type="hidden" value="1" name="retry">

</form>

Webhelpforums

Using the method outlined in section:
Configure Login by Using FireFox Live HTTP Headers

Described at help page
http://www.microsystools.com/products/website-download/help/website-scan-login/

The configuration should be something like this:

Login Path (post form):
/99products/members/index.php

Login paramaters (post form):
nicknameLogin=YOURUSERNAME&password=YOURPASSWORD&redirectMember=http%3A%2F%2Fdownloadplrproducts.com%2F99products%2Fmembers%2Findex.php&login=+++++++++Login+++++++++&action=login&retry=1

(obviously you need to correct YOURUSERNAME and YOURPASSWORD in above)


However, without actual user/pass it is hard to test, but there is a good chance above works.

If you continue to have problems you can try email support:
http://www.microsystools.com/home/contact.php

Note #1:
Both A1 Website Download and A1 Website Scraper are invidividual software programs (not "plugins")

Note #2:
This thread is going to be moved to A1 Websie Download section later :)
(A1 Website Scraper is for extracting structured website data into CSV files / databases)
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper