Webmaster Forums - Website and SEO Help

Microsys Products and Webmaster Tools => A1 Website Scraper => Topic started by: gjackson123 on February 20, 2012, 07:40:40 AM

Title: Does A1 Website Scrapper support dynamic parameter passing?
Post by: gjackson123 on February 20, 2012, 07:40:40 AM
 :)Dear  A1 Website Scrapper forum members,

I would like to extract a comprehensive list of real estate data from certain website before generating the data in CSV / XML, or both. However, I am not sure whether A1 Website Scrapper could do the job or not. This web page provides the following selections prior to query the website for answer:

( i ) Address,Suburbs, Postcodes, or Regions,
( ii ) Property Type (Pull down menu),
( iii ) Bedrooms (Pull down menu),
( iv ) Min Price (Pull down menu),
( v) Max Price (Pull down menu),

As a result, can A1 Website Scrapper provide advanced filtering configuration / settings to manually supply these data to achieve the desire query? Moreover, does it support some form of automation using APIs to read the content of these fields possibly from an external file for instance, to dynamically populate these fields to come up with all combinations of fields to achieve a comprehensive set of data?

Thanks a lot,

George
Title: Re: Does A1 Website Scrapper support dynamic parameter passing?
Post by: Webhelpforums on February 21, 2012, 06:51:27 AM
Concerning which pages to crawl/analyze for links see:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/ (http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/)

Then combine it with which pages you want to scrape for data:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/ (http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/)

You can also have A1 Website Scraper combine elements in forms and try visit them as URLs. (Option found if you switch off Easy mode" and then check "Scan website - Crawler options".)

So concerning the URLs themselves and which of them to scrape, it is very flexible.

...

However, the data extraction itself is still done though regular expression which may or may not be suitable for your needs. See:
http://www.microsystools.com/products/website-scraper/help/scrape-websites-tutorial/ (http://www.microsystools.com/products/website-scraper/help/scrape-websites-tutorial/)

You can also have A1 Website Scraper combine elements in forms and try visit them as URLs. (Option found if you switch off Easy mode" and then check "Scan website - Crawler options".)
Title: Re: Does A1 Website Scrapper support dynamic parameter passing?
Post by: gjackson123 on February 29, 2012, 10:13:02 PM
 :)Hi,

Thanks for your guidance. I am still taking some time to become familiarize with this product before getting back to you.

George
Title: Re: Does A1 Website Scrapper support dynamic parameter passing?
Post by: gjackson123 on March 02, 2012, 02:03:17 AM
Hi,

I am wondering whether you could provide some guidance on how to quickly grasp and make use of this product since I am still a little overwhelmed with all the configuration settings in the Advanced mode, which is the minimum functionalies needed for the job at hand:

( i ) Need to go through all states (New South Wales, Northern Territory, Queensland, South Australia, Tasmania, Victoria, Western Australia) in alphabetical from www.postcodes-australia.com.
( ii ) Get the following suburb details from each of the states using Sydney as an example, where the URL is
      http://www.postcodes-australia.com/areas/nsw/sydney/sydney:
     
      ( a ) Suburb name
      ( b ) Direct URL (optional)
      ( c ) Latitude
      ( d ) Locality
      ( e ) Nearest urban centre
      ( f ) Dialing code
      ( g ) Time zone
      ( h ) Nearest airport
      ( i ) Nearest train station
      ( j ) Nearest hotels
      ( k ) Near by features
     
I am hoping to write a wrapper in Java that will pass these details as parameter to A1 Website Scraper and let it carry out the
data extraction.

Does regular expression in A1 Website Scraper zero-width / length word boundaries (\b) and negation to narrow down the search to
specific string, there by getting exactly what one is looking for?

Hope that this can be a feedback to development if the current version does not support the functionalities I am looking for.

Many thanks,

George
Title: Re: Does A1 Website Scrapper support dynamic parameter passing?
Post by: Webhelpforums on March 03, 2012, 07:14:55 AM
I suggest you:

- Download latest version. (Each version usually includes a new scrape demo/example.)
- Check the latest instance of scrape websites tutorial. (The online help is often updated.)

If I recall correctly, there should be no problem using the regex constructs you mention, but if you have problems getting something to work, do feel free to email support directly at:
http://www.microsystools.com/home/contact.php (http://www.microsystools.com/home/contact.php)

(Be sure to include your existing test-project and make it as simple as possible to demonstrate the problem you have. You can always expand it later when the problem has been resolved.)