make docs LLM friendly

[chartotu19]

Actual Situation

I use cursor IDE to assist with my development. In my typical workflow, I @ the docs page to add knowledge to LLMs for better code generation. Most popular coding models work really well with markdown but docs website have only html content.

There is a proposal to serve llms.txt, llms-full.txt and .md extension for each docs page - https://llmstxt.org/. To make websites LLM friendly.

Docs companies like mintlify offer this out of the box as a feature. Also, there are public directories [1] [2] that show 112 projects are using it already. Some examples:

• Agentic framework for AI Apps - crewai project, https://docs.crewai.com/llms.txt
• Langfuse - lib for LLM evals, traces, observability project, https://langfuse.com/llms.txt
• duckdb - in-memory olap DB - project, https://duckdb.org/duckdb-docs.md
• dontenvx - replacement of dotenv - project, https://dotenvx.com/llms.txt

Desired Situation

New endpoints to assist LLM agents to better parse copier docs.

https://copier.readthedocs.io/llms.txt
https://copier.readthedocs.io/llms-full.txt
https://copier.readthedocs.io/en/stable/reference/cli/index.md
...
...

Proposed solution

Add a Github Action to generate these markdown files, llms.txt prior to docs deployment.