fix!: deprecate disallowNonIndexableRoutes

[harlan-zw]

🔗 Linked issue

❓ Type of change

• 📖 Documentation (updates to the documentation or readme)
• 🐞 Bug fix (a non-breaking change that fixes an issue)
• 👌 Enhancement (improving an existing functionality)
• ✨ New feature (a non-breaking change that adds functionality)
• 🧹 Chore (updates to the build process or auxiliary tools and libraries)
• ⚠️ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

The disallowNonIndexableRoutes config adds any ignored routes within your routeRules to /robots.txt.

export default defineNuxtConfig({
  routeRules: {
    '/secret/**': { robots: false },
  }
})

# /robots.txt
Disallow: /secret/

This feature is inherently broken as robot meta tags (or X-Robots-Header) and robots.txt are for different controls.
We can't automatically infer one from the other, for more context see Controlling Web Crawlers but tldr; we may want robots to still visit a page but not index it.

This PR changes the default behavior to false for this config and sets it to deprecated. While this is marked as a breaking change, this should have no effect on your end SEO as the page will still be disabled from indexing