Developer Needs Help Creating a Huge Sitemap for 2,000,000 URLs

A website designer asked for advice in a large online forum. They have a client who is in a technical and industrial niche. They sell industrial parts that are specific yet similar. There are more than two million SKUs in the client's inventory. The developer's challenge is to make a site map. Because of the number of pages, the developer has to do it manually.

They have a list of URLs by product and category pulled through API. Site maps can only have up to 50,000 URLs. The developer has 37 Excel spreadsheets with 50,000 URLs each. They are using Screaming Frog to sort through the sheets and make site maps out of the crawl. They then send it to FTP and GSC. They want to know if this approach is any good.

So far, the developer is about one-third of the way done. The URLs have been getting indexed as planned. However, they are worried about the size of the site map. They want to make sure they are not making errors that could cause problems in the future.

Some of the fields cause uncertainty for the developer. The primary one causing questions is the priority field. The URLs all have priority level one by default. The developer wonders if they should be making category and static pages priority one along with blog posts and putting products as priority two. The developer adds that because the products are so specific, customers often search for them by SKU. For this reason, they want the product pages to be ranked and indexed by the Google crawler.

One person replied to the developer with a suggestion to get the site map directly from the CMS. Another person suggested getting the site map from the API that the development team already used for the project. A different person told the original poster they should refer to the database itself for the site map. An individual also suggested the developer take a look at a dynamic site map. One person's thought was that doing this manually means the developer does not know their stuff. For more information click here

10 Best SEO is the internets best rankings directory for SEO Agencies. Our experience in the industry ensures an accurate list. Best SEO Agencies & SEO Agency Awards are fastly becoming the largest and most well known SEO awards

SEO Consulting

We won't optimize your site, but we will help you find the right online marketer for your needs.


Sign up below to get updates from our site!