Composable queries: Allow queries as data sources

[chriszarate]

Allow queries to provide other queries as data sources. This allows queries compose each other, reducing code duplication and allowing access to "nested collections" within the query response.

For example, you may wish to write one query to fetch data for a product, and another query to fetch a collection of images for that product, in order to render a gallery. Instead of duplicating a large amount of code between the two queries, you can use the product query as the data source for the query that gets the images for that product.

Since query responses are always cached in-memory (even if they are not cached in object cache), queries are not re-executed when composed.

[smithjw1]

@chriszarate can you add an example? I'm not exactly sure how I would code this up.

I think you would register two blocks: a "product" block with a standard query and a "product images" block with a data source of the first query (the one tied to the product block).

Would the child query mapping be against the parent query's output schema or the shape of the data returned to the parent query? In other words, can the child grab data returned in the parent query but not output?

Could the block registered to the child query be inserted outside the parent block? Would parent overrides filter down to the child?

This is very cool, but introduces a few new failure modes and complications that we want to document and try to prevent in the editor.

[alecgeatches]

can you add an example? I'm not exactly sure how I would code this up.

Same! I'd like to test this with some of our nested data sources.

[chriszarate]

I think you would register two blocks: a "product" block with a standard query and a "product images" block with a data source of the first query (the one tied to the product block).

Yes, precisely! But....

can you add an example? I'm not exactly sure how I would code this up.

@smithjw1 @alecgeatches I added an integration test in 4cfdd75. Thanks for the suggestion. It helped me realize that the JSONPath syntax for navigating the output schema of the parent query is pretty painful and not at all intuitive.

Would the child query mapping be against the parent query's output schema or the shape of the data returned to the parent query?

Currently, it's against the output schema, but as noted that results in some tough ergonomics.

In other words, can the child grab data returned in the parent query but not output?

The child query cannot access data that was "discarded" or not output by the parent query. This is somewhat intentional, since it is assumed that valuable extraction and formatting is being done by the parent query that would be repetitive to duplicate. I can definitely see the argument for the opposite approach, though!

Could the block registered to the child query be inserted outside the parent block?

Yes, because the child query is completely configured and can be executed independently. Blocks have no awareness of the embedded hierarchy of the query.

Would parent overrides filter down to the child?

No, because overrides are configured at the block level, and blocks have no awareness of the hierarchy.

This is very cool, but introduces a few new failure modes and complications that we want to document and try to prevent in the editor.

Indeed, I think this is probably half-baked. I think queries as data sources are a useful tool to reduce code duplication, but the ergnomics need work.

Separately and more importantly, there is a use case to render "sub collections" out of query results, which might be better solved with some type hinting and a "helper block".

Instead of parent / child queries and separately registered blocks, we could ask a single query to describe all sub collections in its output schema (example), then direct the container block to register "helper" blocks that target the sub collections, like this:

register_remote_data_block( [
	'title' => 'Toy',
	'render_query' => [
		'query' => $toy_query,
	],
	'child_blocks' => [
		[
			'name' => 'Toy Variant',
			'render_query_target' => '$.variants',
		],
	],
] );

In theory, this gives us all the information we need to understand the data structure of the sub collection and register "helper" blocks that can be placed as children of the container block. Blocks would then have awareness of the hierarchy which can address some of the issues raised by your questions. Most critically, there would be no need to "re-curate" the data in the helper blocks—they would inherit the selected data from container block and know how to select the sub collection.

I'll leave this for now and give it some more thought, but this might be a better approach.

[brookewp]

@chriszarate have you had a chance to look into this more? I may have run into some use cases which may be helpful in thinking about this.

The Chicago Art Institute has an endpoint to get artwork types , which could be a good way to use this and implement a filtering example.

Also, I wonder if this would help with filtering in Shopify. To get data without having to paginate through all the products (although we'd have to paginate through all the potential filters, but I imagine that would be less heavy than all products).

[chriszarate]

@brookewp Not a ton, I didn't want to get too far ahead of product exploration that might illuminate a better path. But maybe it's a good time to start those conversations!

[smithjw1]

I didn't want to get too far ahead of product exploration that might illuminate a better path.

My gut is your sub-collection definition at block registration is a better first step. Would a pattern be auto-generated based on the structure of the child?

If so, @alecgeatches, I think that's just the ticket for what we need.