Webmaster Forums - Website and SEO Help

Microsys Products and Webmaster Tools => A1 Sitemap Generator => Topic started by: cwallace on January 06, 2019, 11:47:04 AM

Title: Robot.txt - pretty much empty after 'creation'
Post by: cwallace on January 06, 2019, 11:47:04 AM: When I went to 'create robot.txt' it created a file that only has a sinlge line for sitemap entries and it refers to a 'video' format one...which I don't even have video on my site so why it would take that one and not the base index I don't know...

This is what I get:

User-agent: *

Sitemap: http://<domain name>.com/sitemap-video-OSH.xml

I have 15 or more sitemap files...I would expect to see a bigger list or as I have found with some research a main 'index' sitemap to include only...

Anyway...I would like to create a nice valid robot.txt file, but I keep getting nothing for the most part.

Thanks!!
Title: Re: Robot.txt - pretty much empty after 'creation'
Post by: cwallace on January 06, 2019, 01:07:09 PM: I also noticed several created sitemaps are HUGE in content...one has nearly 500k entries and google is kicking it back telling me to break it down to 50k per file...

However, I am not sure that google interprets those sitemaps and are even valid in the sitemaps for google...

Can you point me to where I might sort out WHAT files should be uploaded and entered into the google search concole sitemap submitter??

Does that make sense??

I do 'create all' and it creates 35 sitemap files...I upload them and then submit to google.

A lot of them have errors...should I not be using them or are these errors in my configuration or setup before sending?

Things like missing XML tag (i.e. there was no URL tag as there was no content in a specific 'video' sitemap). Should I just delete it?

This is still one the best software purchases I have ever made in regard to SEO!!

Thanks in advance...
Chris
Title: Re: Robot.txt - pretty much empty after 'creation'
Post by: Webhelpforums on January 13, 2019, 08:32:08 AM: Hi,

Sorry for the delay (email support was still within 24 hours though, but the forum took an unintended backseat for a while)

There are many of the sitemap file types you should not submit to Google, e.g. HTML sitemaps but there are quite a few more that are not meant to be submitted to Google (as they used for entirely different purposes)

if you simply submit all generated then you will get lots of errors and warnings about XML etc. :)

Concerning sitemaps differences see this:
https://www.microsystools.com/products/sitemap-generator/help/website-sitemap-kinds-comparison/ (https://www.microsystools.com/products/sitemap-generator/help/website-sitemap-kinds-comparison/)

But if you do not have video or image content you should simply generate and upload a standard XML sitemap.

I suggest you do not use "Build all", but instead select XML sitemap in the dropdown of choices and click "Build selected". (And maybe clear the output directory first.) That way you avoid uploading old/wrong files to Google Webmaster Tools.

If your website is large, A1 Sitemap Generator will default to split it across multiple files including creating a sitemap index file:
https://www.microsystools.com/products/sitemap-generator/help/xml-sitemaps-page-limit/ (https://www.microsystools.com/products/sitemap-generator/help/xml-sitemaps-page-limit/)

From your robots.txt question I suspect you may have configured output paths wrongly (but I may be wrong - I will need to see the project file) - for reference here is the online help:
https://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-disallow-urls/ (https://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-disallow-urls/)

If I can see your project file, I will have a much better chance of helping, so please do this:

1) Make sure you are using a recent version of the software
2) Email info@microsystools.com your project file ("File | Save As... Which generates a ".ini" file you can send)
3) Report examples of errors in Google Webmaster Tools if they are still there (Note: sometimes resubmitting and waiting a day helps) - the more specific the better
Title: Re: Robot.txt - pretty much empty after 'creation'
Post by: cwallace on May 13, 2019, 01:18:55 PM: Hello. I have been absent a while, but have sent an e-mail in regard to this post. I am sure something might trigger activity again. :)

I am obviously in no rush. Still using the software as I have...just made some changes and have done some testing and benchmarking stuff.

Thanks!!
Chris
Title: Re: Robot.txt - pretty much empty after 'creation'
Post by: Webhelpforums on May 20, 2019, 05:23:55 AM: Hi Chris,

We emailed some days ago back and forth - in my last email I asked for further information - did you miss it by any chance?
Title: Re: Robot.txt - pretty much empty after 'creation'
Post by: cwallace on May 21, 2019, 02:13:19 PM: Checking now.

Chris
Title: Re: Robot.txt - pretty much empty after 'creation'
Post by: cwallace on May 21, 2019, 02:19:33 PM: Checked my e-mail and even spam and saw nothing. I am assuming coming from the info@micr...

Chris
Title: Re: Robot.txt - pretty much empty after 'creation'
Post by: Webhelpforums on May 25, 2019, 02:39:27 PM: See you replied to my emails, so we will continue there :)