can this scraper plugin scrape a website that requires a password set via js?

  • 4 Replies
  • 2563 Views
*

wowza

  • Newbie
  • *
  • 3
  • +0/-0
    • View Profile
I have a membership to a site that needs a username/password.

Everything in the site, once I am logged in sets me as a master seller with resell rights, so I have 100% permission to download the zip files from the site. That is what I paid for.

the problem is there is about 1700 pages, and I do not want to sit here for 2 days clicking button after button to download all of the zip files, that is why i got this scraper program.

Here is the site:
http://bit.ly/U92AiI

So can this scraper plugin let me login and then simply download all the zip files?
* I do not want anything else but the zips, no need *

thanks

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1385
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
What you actually want is A1 Website Download to download all ZIP files
http://www.microsystools.com/products/website-download/

First check the general tutorial:
http://www.microsystools.com/products/website-download/help/websites-download-tutorial/

Then see this tutorial on how to limit download to specific types of files including e.g. .zip:
http://www.microsystools.com/products/website-download/help/website-gallery-image-downloader/
 
Then try the two login methods described in help:
http://www.microsystools.com/products/website-download/help/website-scan-login/
MicrosysTools.com | Website and SEO Software for webmasters | A1 Sitemap Generator, A1 Website Analyzer etc.

*

wowza

  • Newbie
  • *
  • 3
  • +0/-0
    • View Profile
I am so confused.

Can you do me one favor and try to use the link for the website I gave in my first post and let me know how I am supposed to login to that site using the website download tool?

I followed the guides the best I could but I do not understand which type of authentication I am supposed to use for that website.

If you could just let me know what settings to add to your program, then I can simply add in the username and password I have and see if it works.

Or I could just send you the username/pass, tried that but can't pm via your profile, so not sure how.

Your plugin looks great, but it's hard to figure out and this seems like a basic use case scenario.

thanks so much

*

wowza

  • Newbie
  • *
  • 3
  • +0/-0
    • View Profile
Here is the form data

Code: [Select]
<form method="post" id="loginForm" name="loginForm">


<input type="text" maxlength="60" size="30" name="nicknameLogin">
<input type="password" maxlength="60" size="30" name="password">
<a onmouseout="this.className='button-ltblue'; " onblur="this.className='button-ltblue';" onclick="javascript:document.getElementById('loginForm').submit();" onmouseover="javascript:this.className='button-black';" href="javascript:void(0);" class="button-ltblue">
<a onmouseout="this.className='button-ltblue'; " onblur="this.className='button-ltblue';" onclick="javascript:document.getElementById('loginForm').reset();" onmouseover="javascript:this.className='button-black';" href="javascript:void(0);" class="button-ltblue">

<input type="hidden" value="http://website.com/99products/members/index.php" name="redirectMember">
<input type="hidden" value="         Login         " name="login">
<input type="hidden" value="login" name="action">
<input type="hidden" value="1" name="retry">

</form>

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1385
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Using the method outlined in section:
Configure Login by Using FireFox Live HTTP Headers

Described at help page
http://www.microsystools.com/products/website-download/help/website-scan-login/

The configuration should be something like this:

Login Path (post form):
Code: [Select]
/99products/members/index.php
Login paramaters (post form):
Code: [Select]
nicknameLogin=YOURUSERNAME&password=YOURPASSWORD&redirectMember=http%3A%2F%2Fdownloadplrproducts.com%2F99products%2Fmembers%2Findex.php&login=+++++++++Login+++++++++&action=login&retry=1
(obviously you need to correct YOURUSERNAME and YOURPASSWORD in above)


However, without actual user/pass it is hard to test, but there is a good chance above works.

If you continue to have problems you can try email support:
http://www.microsystools.com/home/contact.php

Note #1:
Both A1 Website Download and A1 Website Scraper are invidividual software programs (not "plugins")

Note #2:
This thread is going to be moved to A1 Websie Download section later :)
(A1 Website Scraper is for extracting structured website data into CSV files / databases)
« Last Edit: November 28, 2012, 09:55:02 AM by Webhelpforums »
MicrosysTools.com | Website and SEO Software for webmasters | A1 Sitemap Generator, A1 Website Analyzer etc.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper