Thanks for the reply. A combination of what you've suggested should eventually get the job done. However, I'm having a bit of trouble scanning individual directories due to the .NET nature of the site.
For example, for the Asus subdirectory, there'll be something of a doorway page such as:
www.mywebsite.com/Asus.aspx?manid=1&catid=1
The above page will then link to many pages within the Asus subdirectory, such as:
www.mywebsite.com/Asus/Series.aspx?seriesid=1&manid=1&catid=1
I guess the issue is that the Asus subdirectory and all of the pages contained in it don't actually exist without being linked to from the Asus doorway page. The pages are created dynamically and don't exist within a static directory structure on the server.
Also, is there a way of limiting the scan so that it will only scan descending subdirectory levels, to avoid it reaching the root directory again and trying to crawl the whole site?