Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy load post meta #4216

Open
wants to merge 28 commits into
base: trunk
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
e77768b
Lazy load post meta
spacedmonkey Feb 14, 2023
a9e45a1
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Mar 9, 2023
b5f297e
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Mar 10, 2023
efcdbc7
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Mar 29, 2023
85ad4ae
Rebase
spacedmonkey Mar 29, 2023
9f19cac
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Apr 21, 2023
77fa22e
Fix
spacedmonkey Apr 21, 2023
753379d
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey May 23, 2023
093b224
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Jul 7, 2023
ef8e797
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Jul 13, 2023
b2e72e1
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Jun 12, 2024
657aebb
Update callback name and implement lazyload_post_meta function
spacedmonkey Jun 12, 2024
ee9e65e
Update expected database queries in tests
spacedmonkey Jun 12, 2024
14aec68
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Oct 9, 2024
d6d9934
Add filter for REST API menu read access check
spacedmonkey Oct 9, 2024
8f33b9a
Update post meta cache function call
spacedmonkey Oct 9, 2024
e61d194
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Nov 17, 2024
55b32c3
Revert these changes.
spacedmonkey Nov 17, 2024
4330c69
Apply suggestions from code review
spacedmonkey Nov 17, 2024
4f90281
Merge branch 'trunk' into fix/lazy-load-post-meta
spacedmonkey Dec 3, 2024
046cdf5
Add $lazy_load_post_meta parameter to WP_Query and use it in post con…
spacedmonkey Dec 3, 2024
cbe16f1
Add $lazy_load_post_meta parameter in wp_get_post_revisions.
spacedmonkey Dec 3, 2024
dc7344a
Fix tests.
spacedmonkey Dec 4, 2024
a077f46
Improve tests.
spacedmonkey Dec 4, 2024
f4ec367
Fix tests again.
spacedmonkey Dec 4, 2024
59a3aee
Add tests.
spacedmonkey Dec 4, 2024
a6b8d34
wp_dashboard_recent_posts lazy load post meta.
spacedmonkey Dec 4, 2024
4e3a8ec
Apply suggestions from code review
spacedmonkey Jan 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions src/wp-admin/includes/dashboard.php
Original file line number Diff line number Diff line change
Expand Up @@ -976,14 +976,15 @@ function wp_dashboard_site_activity() {
*/
function wp_dashboard_recent_posts( $args ) {
$query_args = array(
'post_type' => 'post',
'post_status' => $args['status'],
'orderby' => 'date',
'order' => $args['order'],
'posts_per_page' => (int) $args['max'],
'no_found_rows' => true,
'cache_results' => true,
'perm' => ( 'future' === $args['status'] ) ? 'editable' : 'readable',
'post_type' => 'post',
'post_status' => $args['status'],
'orderby' => 'date',
'order' => $args['order'],
'posts_per_page' => (int) $args['max'],
'no_found_rows' => true,
'cache_results' => true,
'lazy_load_post_meta' => true,
'perm' => ( 'future' === $args['status'] ) ? 'editable' : 'readable',
);

/**
Expand Down
39 changes: 39 additions & 0 deletions src/wp-includes/class-wp-metadata-lazyloader.php
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@ public function __construct() {
'filter' => 'get_comment_metadata',
'callback' => array( $this, 'lazyload_meta_callback' ),
),
'post' => array(
'filter' => 'get_post_metadata',
'callback' => array( $this, 'lazyload_post_meta' ),
),
'blog' => array(
'filter' => 'get_blog_metadata',
'callback' => array( $this, 'lazyload_meta_callback' ),
Expand Down Expand Up @@ -163,6 +167,41 @@ public function lazyload_comment_meta( $check ) {
return $this->lazyload_meta_callback( $check, 0, '', false, 'comment' );
}

/**
* Lazy-loads post meta for queued posts.
*
* This method is public so that it can be used as a filter callback. As a rule, there
* is no need to invoke it directly.
*
* @since x.x.x
*
* @param mixed $check The `$check` param passed from the 'get_*_metadata' hook.
* @param int $object_id ID of the object metadata is for.
* @param string $meta_key Unused.
* @param bool $single Unused.
* @param string $meta_type Type of object metadata is for. Accepts 'post', 'comment', 'term', 'user',
* or any other object type with an associated meta table.
* @return mixed In order not to short-circuit `get_metadata()`. Generally, this is `null`, but it could be
* another value if filtered by a plugin.
*/
public function lazyload_post_meta( $check, $object_id, $meta_key, $single, $meta_type ) {
if ( empty( $this->pending_objects[ $meta_type ] ) ) {
return $check;
}

$object_ids = array_keys( $this->pending_objects[ $meta_type ] );
if ( $object_id && ! in_array( $object_id, $object_ids, true ) ) {
return $check;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole method is a line for line duplication of lazyload_meta_callback() except for this line. Why is the logic different here, and could this be simplified by adding a check for $meta_type to return the $check rather than adding the $object_id to the array of IDs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joemcgill This is the secret sause of this change. Unlike other meta types like, site, comment and term, post meta gets called ALOT. So to stop every call to post meta loading this lazy load queue, this only loads if the queue if an item is in the post meta queue.

For, if I have 10 items of post meta loaded and 5 items in the lazy load queue, if you request an post id in the load meta, you DONT want the queue items to load.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spacedmonkey I think the logic here should be slightly different. If the object ID passed to get_post_meta() is already cached, then it makes sense not to load the queued data. If the object ID is not cached, then a database call will be made and it makes sense to load the queued data.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peterwilsoncc that is not how the other meta types work. IDs are put in the queue. If the id is already loaded, when wp_cache_get_multiple, is called, if it is only object is not cache is loaded.

I don't see lots of cases where an id in the queue but also primed. I expect that ids that might be used will be added to the queue, for example attachment meta or post parent meta.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spacedmonkey I realise that is not how it works for other meta data (potentially a bug), but you are proposing that post meta works differently.

However, the code in it's current form will trigger a database query for get_post_meta( $uncached_post_id ) without priming other data in the queue. As the idea of lazy loading meta data is to ride another database query, it seems we should do that here even if the uncached object is not queued.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peterwilsoncc So you are saying, we should not have special behavour for post meta, it should work like other meta? That every time get_post_meta is called it should clear the queue? If that is the case, as post meta is called so much, then there is no performance benefit. The queue would be cleared so much, that we might as well not bother.

It is less about saving database queries I guess. It useful in the following use case.

I have loaded 5 posts and there meta. But I have also loaded, all those post parents, ( say for revisions or attachments ). Also add the parents post meta to the queue, as it MAY get used. In cases, where it is, load them all at once, in other cases the items in the queue may never get loaded, saving a database query. There are LOTs of places in core, where post meta is loaded, because it MAY be used a plugin, ( example if you have a filter of the title to change it a value loaded for post meta ). But core itselve is not using this post meta. Once we have a queue, where it says maybe load post meta, we can stop priming post meta in lots of places where it is not required by core, but turning off the priming might effect sites with plugins that except the post meta to be loaded.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That every time get_post_meta is called it should clear the queue? If that is the case, as post meta is called so much, then there is no performance benefit. The queue would be cleared so much, that we might as well not bother.

No, this is not what I am saying.

What I am saying is that IF a database query is required then the queue should be cleared. For example:

Cached post meta IDs: 1, 2, 3, 4
Queued post meta IDs: 5, 6, 7, 8

get_post_meta( 1 ) would not clear the queue as it does not require a database call
get_post_meta( 10 ) does require a database call so would use the opportunity to clear the queue and prime the cache for the queued meta data.

}

update_meta_cache( $meta_type, $object_ids );

// No need to run again for this set of objects.
$this->reset_queue( $meta_type );

return $check;
}

/**
* Lazy-loads meta for queued objects.
*
Expand Down
28 changes: 28 additions & 0 deletions src/wp-includes/class-wp-query.php
Original file line number Diff line number Diff line change
Expand Up @@ -648,6 +648,7 @@ public function fill_query_vars( $query_vars ) {
* @since 5.3.0 Introduced the `$meta_type_key` parameter.
* @since 6.1.0 Introduced the `$update_menu_item_cache` parameter.
* @since 6.2.0 Introduced the `$search_columns` parameter.
* @since 6.8.0 Introduced the `$lazy_load_post_meta` parameter.
*
* @param string|array $query {
* Optional. Array or string of Query parameters.
Expand Down Expand Up @@ -784,6 +785,9 @@ public function fill_query_vars( $query_vars ) {
* disable cache priming for term meta, so that each
* get_term_meta() call will hit the database.
* Defaults to the value of `$update_post_term_cache`.
* @type bool $lazy_load_post_meta Whether to lazy-load post meta. Setting to false will
* disable cache priming for post meta, so that each
* get_post_meta() call will hit the database.
* @type int $w The week number of the year. Default empty. Accepts numbers 0-53.
* @type int $year The four-digit year. Default empty. Accepts any four-digit year.
* }
Expand Down Expand Up @@ -1967,6 +1971,12 @@ public function get_posts() {
$q['update_post_meta_cache'] = true;
}

if ( ! isset( $q['lazy_load_post_meta'] ) ) {
$q['lazy_load_post_meta'] = false;
spacedmonkey marked this conversation as resolved.
Show resolved Hide resolved
} elseif ( $q['lazy_load_post_meta'] ) {
$q['update_post_meta_cache'] = false;
spacedmonkey marked this conversation as resolved.
Show resolved Hide resolved
}

if ( ! isset( $q['post_type'] ) ) {
if ( $this->is_search ) {
$q['post_type'] = 'any';
Expand Down Expand Up @@ -3243,6 +3253,10 @@ public function get_posts() {
$this->post_count = count( $this->posts );
$this->set_found_posts( $q, $limits );

if ( $q['lazy_load_post_meta'] ) {
wp_lazyload_post_meta( $this->posts );
}

if ( $q['cache_results'] && $id_query_is_cacheable ) {
$cache_value = array(
'posts' => $this->posts,
Expand Down Expand Up @@ -3281,6 +3295,10 @@ public function get_posts() {
// Prime post parent caches, so that on second run, there is not another database query.
wp_cache_add_multiple( $post_parents_cache, 'posts' );

if ( $q['lazy_load_post_meta'] ) {
wp_lazyload_post_meta( $post_ids );
}

if ( $q['cache_results'] && $id_query_is_cacheable ) {
$cache_value = array(
'posts' => $post_ids,
Expand Down Expand Up @@ -3535,6 +3553,7 @@ public function get_posts() {
'update_post_meta_cache' => $q['update_post_meta_cache'],
'update_post_term_cache' => $q['update_post_term_cache'],
'lazy_load_term_meta' => $q['lazy_load_term_meta'],
'lazy_load_post_meta' => $q['lazy_load_post_meta'],
)
);

Expand Down Expand Up @@ -3592,6 +3611,11 @@ public function get_posts() {
wp_queue_posts_for_term_meta_lazyload( $this->posts );
}

if ( $q['lazy_load_post_meta'] ) {
$post_ids = wp_list_pluck( $this->posts, 'ID' );
wp_lazyload_post_meta( $post_ids );
}

return $this->posts;
}

Expand Down Expand Up @@ -3692,6 +3716,9 @@ public function the_post() {
$post_ids = array_filter( $post_ids );
if ( $post_ids ) {
_prime_post_caches( $post_ids, $this->query_vars['update_post_term_cache'], $this->query_vars['update_post_meta_cache'] );
if ( $this->query_vars['lazy_load_post_meta'] ) {
wp_lazyload_post_meta( $post_ids );
}
}
$post_objects = array_map( 'get_post', $this->posts );
update_post_author_caches( $post_objects );
Expand Down Expand Up @@ -4880,6 +4907,7 @@ protected function generate_cache_key( array $args, $sql ) {
$args['cache_results'],
$args['fields'],
$args['lazy_load_term_meta'],
$args['lazy_load_post_meta'],
$args['update_post_meta_cache'],
$args['update_post_term_cache'],
$args['update_menu_item_cache'],
Expand Down
3 changes: 2 additions & 1 deletion src/wp-includes/nav-menu.php
Original file line number Diff line number Diff line change
Expand Up @@ -807,7 +807,8 @@ function update_menu_item_cache( $menu_items ) {
}

if ( ! empty( $post_ids ) ) {
_prime_post_caches( $post_ids, false );
_prime_post_caches( $post_ids, false, false );
wp_lazyload_post_meta( $post_ids );
}

if ( ! empty( $term_ids ) ) {
Expand Down
18 changes: 17 additions & 1 deletion src/wp-includes/post.php
Original file line number Diff line number Diff line change
Expand Up @@ -2616,6 +2616,21 @@ function get_post_meta( $post_id, $key = '', $single = false ) {
return get_metadata( 'post', $post_id, $key, $single );
}

/**
* Queue post meta for lazy-loading.
*
* @since 6.3.0
*
* @param array $post_ids List of post IDs.
*/
function wp_lazyload_post_meta( array $post_ids ) {
if ( empty( $post_ids ) ) {
return;
}
$lazyloader = wp_metadata_lazyloader();
$lazyloader->queue_objects( 'post', $post_ids );
}

/**
* Updates a post meta field based on the given post ID.
*
Expand Down Expand Up @@ -7706,7 +7721,8 @@ function update_post_parent_caches( $posts ) {
$parent_ids = array_unique( array_filter( $parent_ids ) );

if ( ! empty( $parent_ids ) ) {
_prime_post_caches( $parent_ids, false );
_prime_post_caches( $parent_ids, false, false );
wp_lazyload_post_meta( $parent_ids );
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,9 @@ static function ( $format ) {
// Force the post_type argument, since it's not a user input variable.
$args['post_type'] = $this->post_type;

// Lazy load post meta.
$args['lazy_load_post_meta'] = true;

/**
* Filters WP_Query arguments when querying posts via the REST API.
*
Expand Down
7 changes: 4 additions & 3 deletions src/wp-includes/revision.php
Original file line number Diff line number Diff line change
Expand Up @@ -678,9 +678,10 @@ function wp_get_post_revisions( $post = 0, $args = null ) {
$args = array_merge(
$args,
array(
'post_parent' => $post->ID,
'post_type' => 'revision',
'post_status' => 'inherit',
'post_parent' => $post->ID,
'post_type' => 'revision',
'post_status' => 'inherit',
'lazy_load_post_meta' => true,
)
);

Expand Down
1 change: 1 addition & 0 deletions tests/phpunit/includes/abstract-testcase.php
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ protected function reset_lazyload_queue() {
$lazyloader = wp_metadata_lazyloader();
$lazyloader->reset_queue( 'term' );
$lazyloader->reset_queue( 'comment' );
$lazyloader->reset_queue( 'post' );
$lazyloader->reset_queue( 'blog' );
}

Expand Down
19 changes: 11 additions & 8 deletions tests/phpunit/tests/post/nav-menu.php
Original file line number Diff line number Diff line change
Expand Up @@ -267,13 +267,13 @@ public function test_update_menu_item_cache_primes_posts() {

wp_cache_delete( $post_id, 'posts' );
$action = new MockAction();
add_filter( 'update_post_metadata_cache', array( $action, 'filter' ), 10, 2 );
add_action( 'metadata_lazyloader_queued_objects', array( $action, 'action' ) );

update_menu_item_cache( $query_result );

$args = $action->get_args();
$last = end( $args );
$this->assertSameSets( array( $post_id ), $last[1], '_prime_post_caches() was not executed.' );
$this->assertSameSets( array( $post_id ), $last[0], 'wp_lazyload_post_meta() was not executed.' );
}

/**
Expand Down Expand Up @@ -336,17 +336,20 @@ public function test_wp_get_nav_menu_items_cache_primes_posts() {
wp_cache_delete_multiple( $post_ids, 'posts' );
wp_cache_delete_multiple( $post_ids, 'post_meta' );

$action = new MockAction();
add_filter( 'update_post_metadata_cache', array( $action, 'filter' ), 10, 2 );
$action1 = new MockAction();
$action2 = new MockAction();
add_filter( 'update_post_metadata_cache', array( $action1, 'filter' ), 10, 2 );
add_action( 'metadata_lazyloader_queued_objects', array( $action2, 'action' ) );

$start_num_queries = get_num_queries();
wp_get_nav_menu_items( $this->menu_id, array( 'nopaging' => false ) );
$queries_made = get_num_queries() - $start_num_queries;
$this->assertSame( 7, $queries_made, 'Only does 7 database queries when running wp_get_nav_menu_items.' );
$this->assertSame( 6, $queries_made, 'Only does 6 database queries when running wp_get_nav_menu_items.' );

$args = $action->get_args();
$this->assertSameSets( $menu_nav_ids, $args[0][1], '_prime_post_caches() was not executed.' );
$this->assertSameSets( $post_ids, $args[2][1], '_prime_post_caches() was not executed.' );
$args1 = $action1->get_args();
$args2 = $action2->get_args();
$this->assertSameSets( $menu_nav_ids, $args1[0][1], '_prime_post_caches() was not executed.' );
$this->assertSameSets( $post_ids, $args2[0][0], 'lazy_load_post_meta() was not executed.' );
}

/**
Expand Down
4 changes: 4 additions & 0 deletions tests/phpunit/tests/query/cacheResults.php
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,10 @@ public function data_query_cache_duplicate() {
'update_menu_item_cache' => false,
),
),
'lazy load post meta' => array(
'query_vars1' => array( 'lazy_load_post_meta' => true ),
'query_vars2' => array( 'lazy_load_post_meta' => false ),
),
);
}

Expand Down
102 changes: 102 additions & 0 deletions tests/phpunit/tests/query/lazyLoadPostMeta.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
<?php

/**
* @group query
* @group meta
*/
class Tests_Lazy_Load_Post_Meta extends WP_UnitTestCase {
/**
* Post IDs.
*
* @var int[]
*/
private static $post_ids = array();

public static function wpSetUpBeforeClass( WP_UnitTest_Factory $factory ) {
// Register CPT for use with shared fixtures.
register_post_type( 'wptests_pt' );

self::$post_ids = $factory->post->create_many( 5, array( 'post_type' => 'wptests_pt' ) );
foreach ( self::$post_ids as $post_id ) {
update_post_meta( $post_id, 'foo', 'bar' );
}
}

/**
* @dataProvider data_lazy_load_post_meta
* @ticket 57496
*/
public function test_lazy_load_post_meta( $query_args ) {
wp_cache_delete_multiple( self::$post_ids, 'posts' );
wp_cache_delete_multiple( self::$post_ids, 'post_meta' );
$action1 = new MockAction();
$action2 = new MockAction();
add_filter( 'update_post_metadata_cache', array( $action1, 'filter' ), 10, 2 );
add_action( 'metadata_lazyloader_queued_objects', array( $action2, 'action' ) );

new WP_Query( $query_args );

$args1 = $action1->get_args();
$args2 = $action2->get_args();
$last = end( $args2 );
$this->assertSameSets( self::$post_ids, $last[0], 'wp_lazyload_post_meta() was not executed.' );
$this->assertSameSets( array(), $args1, 'update_meta_cache() was executed.' );
$num_queries = get_num_queries();
get_post_meta( self::$post_ids[0], 'foo', true );
$this->assertSame( $num_queries + 1, get_num_queries(), 'wp_lazyload_post_meta() was not executed.' );
$args1 = $action1->get_args();
$last = end( $args1 );
$this->assertSameSets( self::$post_ids, $last[1], 'update_meta_cache() was not executed.' );
}

/**
* Provides test data for lazy loading post metadata.
*
* @return array
*/
public function data_lazy_load_post_meta() {
return array(
'lazy load post meta' => array(
array(
'post_type' => 'wptests_pt',
'lazy_load_post_meta' => true,
),
),
'lazy load post meta fields id' => array(
array(
'post_type' => 'wptests_pt',
'fields' => 'ids',
'lazy_load_post_meta' => true,
),
),
'lazy load post meta fields id=>parent' => array(
array(
'post_type' => 'wptests_pt',
'fields' => 'id=>parent',
'lazy_load_post_meta' => true,
),
),
'lazy load post meta - update_post_meta_cache true' => array(
array(
'post_type' => 'wptests_pt',
'update_post_meta_cache' => true,
'lazy_load_post_meta' => true,
),
),
'lazy load post meta - update_post_meta_cache false' => array(
array(
'post_type' => 'wptests_pt',
'update_post_meta_cache' => false,
'lazy_load_post_meta' => true,
),
),
'lazy load post meta - cache_results false' => array(
array(
'post_type' => 'wptests_pt',
'cache_results' => false,
'lazy_load_post_meta' => true,
),
),
);
}
}
Loading
Loading