Cache warm-up

[chesio]

Let internal crawling mechanism trigger cache files creation instead of site visitors.

[chesio]

Things to keep in mind:

1. Crawling should not overload the web server.
2. Some URLs are more important than others.
3. Individual cache entries are never removed automatically, they can only be removed "by hand" via Cache Viewer interface, WP-CLI command or deletion from disk.
4. Request variants should be supported as much as possible.

Points 1. and 2. inherently require some priority management (a queue of some sort). I would argue that individual URLs can be grouped as follows (sorted by descending priority):

1. Front-page - is_front_page()
2. Blog-page - is_home()
3. Static pages - is_page()
4. Posts (including CPT, excluding attachments) - is_single()
5. Taxonomy pages - is_tax()

Groups 4 and 5 futhermore contain subgroups:

• in case of group 4: built-in posts and (optional) custom post types
• in case of group 5: built-in category and tags taxonomy and (optional) custom taxonomies

Subgroups should have priority too, although here no obvious default sorting exists. In case of a blog website, blog posts would have higher priority than custom post types; In case of portfolio website with a news section, portfolio custom post type would have higher priority than blog posts.

Finally, items within every (sub)group should have priority too. I assume it could be (most of the time) sort order based on particular database columns: date in case of blog posts, menu order in case of pages etc.

Summary

Given all the above, the crawling queue can be treated as static list. To get next set of URLs to crawl, one would only need to maintain a pointer of some kind like "Nth static page should be cached next". This pointer would then be reset every time the cache is cleared.

Since individual cache entries are only removed manually (see point 3. above), I would argue that they can be set for immediate recrawl. Alternatively the interface could provide an option for that (like "Remove and recrawl").

Implementation notes:

1. The ordering of groups, subgroups and items must be strict.
2. The ordering of groups, subgroups and items should be filterable.
3. It should be possible to exclude certain group, subgroup or item.

Open questions:

1. Should there be some indications of crawling progress? Is this feasible?
2. How to proceed with request variants? Does wp_remote_get() support cookies?

[chesio]

Another thing to consider: some pages (URLs in general) are excluded from caching (like WooCommerce cart page). Crawler should handle such URLs gracefully.

[chesio]

Seems to be a large enough feature to warrant major version bump.

[chesio]

There are 3-rd party solutions to this problem, probably worth further investigation:

• https://wordpress.org/plugins/warm-cache/
• https://github.com/khromov/sitemap-cache-warmer

[chesio]

Things to keep in mind:
1. Crawling should not overload the web server.
2. Some URLs are more important than others.

One elegant solution that fits these criteria is to integrate cache warming with a statistics plugin like Statify and only warm up the cache with M most-visited URLs from last N days (where M and N are respective Statify options).