Talk:Community Wishlist/W397

Received your wish on filtering offtopic subcategories in deepcategory view

We’ve added this wish to the list for internal review and will follow up after identifying the most relevant PM or team to evaluate it. It's great to see how your ideas continue to build on each other and push for more useful, intuitive workflows, especially for Commons power users.

We'll be back in touch once we’ve completed our initial review.

--ARamadan-WMF (talk) 18:04, 17 June 2025 (UTC)

Thanks. The deepcategory-related ideas are mostly pragmatic steps to get to what I proposed in this #2-most-supported proposal in the Commons Wishlist survey for a wall of images view of categories.

I wouldn't say this is necessarily mostly or especially for Commons power users – only the early version may/would be since so few people know about and extensively use the deepcategory search operator. However, the idea is to ultimately make this a very accessible view mode that one can easily switch to via some toggle button on the category page or similar so this would be used even by users doing their very first visit of the site, trying to find some useful image or just exploring (who then depending on the category would then not unlikely need such an easy method for filtering). Prototyperspective (talk) 16:09, 18 June 2025 (UTC)

"#2-most-supported proposal in the Commons Wishlist survey for a wall of images view of categories."

I guess you meant to link to this instead https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Technical_needs_survey/Wall_of_images_view_for_category_pages_including_images_in_subcats CParle (WMF) (talk) 13:38, 19 June 2025 (UTC)

Yes, thanks. Prototyperspective (talk) 20:25, 19 June 2025 (UTC)

Clarification

Hi @Prototyperspective

AIUI what you're asking here is a couple of modifications/enhancements to the functionality of the mediasearch results page if the user has searched using deepcategory - is that correct?

CParle (WMF) (talk) 12:39, 18 June 2025 (UTC)

Yes, that is correct. Manually typing in deepcategory followed by the cat name into the searchbar is required to switch to the wall of images view. If phab:T378756 gets solved, people can also use the category search box (for an example see c:Category:Our World in Data) and eventually there could be some button to switch the viewmode. So yes, if deepcategory is used in the search and it's the MediaSearch view (I think there is no point implementing this for legacy SpecialSearch), then I suggest such filter options are made available. Prototyperspective (talk) 16:15, 18 June 2025 (UTC)

Ok I see what you're getting at, but if https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Technical_needs_survey/Wall_of_images_view_for_category_pages_including_images_in_subcats is what you really want then I don't think this is the right technical path towards it

deepcat is really a beta feature, and the graph database that backs it isn't capable of dealing with intensive use in the way that a search engine is. If what you want is to be able find pictures in the "poodle" category when you search for "dog" (which deepcat kinda gives you) then - especially if we want to serve casual users as well as power users - we need to solve that problem in the search engine first.

FYI we have tried to do this in a few different ways in the past (we did a lot of work on image search a couple of years ago, see this blog post) but so far we haven't found anything that works well

I think maybe what we need here is a wish around regular search (rather than just deepcat) being able to traverse the category graph (and similar for the "instanceof" tree in structured data for images). Leave it with me for a bit, and I'll try and put together a wish, and then we can revisit this and see if we still need it CParle (WMF) (talk) 14:02, 19 June 2025 (UTC)

I thought of the wall-of-images view as a separate thing but realized that deepcat could be used for this and may the most straightforward way to achieve this where steps that build upon another to improve it probably are the most realistic pragmatic way to get the functionality.

the graph database that backs it isn't capable of dealing with intensive use why (roughly)? I think then fixing this issue seems like one of the needed steps. Deepcat itself works fairly well from the user side (loads quite quickly etc). in the way that a search engine is deepcat is a search operator of the search engine so I don't understand this contrasting.

If what you want is to be able find pictures in the "poodle" category when you search for "dog" in this case the proper way to go about it is to navigate the category tree until you're in the poodle category and then switch on the deepcat-viewmode so as to e.g. find a suitable picture (e.g. one you like or one of good general quality or showing something particular for which there is no subcategory) where you can just scroll through all the files.

we need to solve that problem in the search engine first are you referring to a problem of high serverload when using deepcat or that the search engine doesn't understand (sub)categories (e.g. when using query deepcat:dogs poodle) or what do you mean? If you're referring to the problem described in this wish, I don't understand this as it's kind of deepcat-specific to filter away various subcats in the deepcat-view.

[…] see this blog post) but so far we haven't found anything that works well reading it now; please see / glance over my Suggestions for improvement / issues for MediaSearch, especially point 10 and 11.

I think maybe what we need here is a wish around regular search (rather than just deepcat) being able to traverse the category graph Yes...probably point 10 of the issues above would be one of the most impactful but also one of the more difficult things to implement and one can currently either browse the category or in many cases of smaller cats use deepcat. It would be great if you could create a wish for that, thanks for looking into it. I still don't see how this would make this wish less relevant or important or impactful and think of these as kind of separate things – this wish here is about making the deepcat view more widely useful. Prototyperspective (talk) 23:45, 21 June 2025 (UTC)

"the graph database that backs it isn't capable of dealing with intensive use why (roughly)? I think then fixing this issue seems like one of the needed steps. Deepcat itself works fairly well from the user side (loads quite quickly etc). in the way that a search engine is deepcat is a search operator of the search engine so I don't understand this contrasting."

Sub-category data is not stored in the regular search engine, so deepcat works by first using a SPARQL query to get the first 1000 subcategories of what you're searching for, then uses those on a filter for the search results. It's the SPARQL query (running on BlazeGraph, which sits on different servers from the search engine) that's the problem - BlazeGraph gives us much more flexibility when querying, but at the cost of speed and scalability, and if it got the kind of traffic that regular search gets it'd either be incredibly slow or just fall over.

So what you're asking for here is tricky. deepcat is a limited tool that we don't really advertise because we can't scale it, and because we can't scale it improvements to the interface around it are always going to be low-impact. Actually storing the category data in the regular search engine is, as I've said, a large project, and attempts we've made to do it (or something similar to it) in the past have failed, but it's still less impossible than scaling deepcat CParle (WMF) (talk) 16:39, 23 June 2025 (UTC)

FYI I created this new wish https://meta.wikimedia.org/wiki/Community_Wishlist/Wishes/Traverse_the_category_graph_when_searching_for_media_on_Commons CParle (WMF) (talk) 11:53, 24 June 2025 (UTC)

Just checking back in on this @Prototyperspective ... I understand where you're coming from, but the current deepcat doesn't (and can't) scale so I think the approach you're outlining here is kind of a dead end, so I think we're going to have to decline (or rewrite) this wish.

I agree that a wall-of-images display for File pages in categories is a good idea on Commons fwiw CParle (WMF) (talk) 14:23, 31 July 2025 (UTC)

It's good to eliminate alternatives to make a given set or one option(s) more replicable and confidently right. However, you didn't explain why the current deepcat doesn't (and can't) scale so I don't know why that could be the case. I suggest this is kept since it would be very useful for people currently using deepcat and a separate wish could be 'forked' basically of this wish based on or involving a method other than deepcat. Here the question would be which other method would there be? Currently, there doesn't seem to be one and this wish is about pragmatically achieving roughly the envisioned feature with what is available/currently-possible even if that's a bit hacky and not an ideal bottom-up-built thing. Regarding a separate wish could you outline things a bit? Prototyperspective (talk) 11:28, 10 August 2025 (UTC)

I did explain! Just up there ☝️

Sub-category data is not stored in the regular search engine, so deepcat works by first using a SPARQL query to get the first 1000 subcategories of what you're searching for, then uses those on a filter for the search results. It's the SPARQL query (running on BlazeGraph, which sits on different servers from the search engine) that's the problem - BlazeGraph gives us much more flexibility when querying, but at the cost of speed and scalability, and if it got the kind of traffic that regular search gets it'd either be incredibly slow or just fall over.

Hmmm ... I'm not sure how tech-y you are though, so maybe it's not clear? I can try and clarify if there are bits that aren't clear? CParle (WMF) (talk) 12:50, 11 August 2025 (UTC)

Sorry, I only read your last comment beneath it and somehow didn't notice the one above it. Thanks for explaining. Btw, I could not reply to your comment now using the Reply tool which is another problem. One thing that remains open is what alternative option(s) for this could be – do you know of any that you could roughly outline? Maybe the sparql query could be run on a dedicated server and be made more performant and scalable. Assuming that can't be done what may be alternatives? --Prototyperspective (talk) 17:25, 11 August 2025 (UTC)

I think it depends on exactly what problem you're trying to solve.

If this was intended to be a step towards a proper wall of images for category pages then I think we should probably create a separate wish for that. I can do that if you like - I have an idea for how it might be done

If you're trying to find quirks in the category graph that's kinda different and tbh I'm not sure how to go about that CParle (WMF) (talk) 11:11, 12 August 2025 (UTC)

More or less it was intended to be that...basically it's a missing functionality in that already functional method/tool that can be used for what's described there despite being suboptimal and barely known and not having any proper UI to use it except for the nondefault DeepcatSearch Gadget.

However, it's also useful in a broader sense for uses of deepcategory in general. A few offtopic cats is often the issue with it and manually excluding them with -deepcategory and without any info on where the images are from (you need to glance over all subcategories and guess where they're from or open them and guess which of the files' cats is in the category the deepcategory is run on) is cumbersome. If there was a better way to achieve the wall of images cat view, then it wouldn't make this wish redundant, it would just greatly deprioritize it if that way gets implemented.

What's your idea how it might be done? Eager to hear, we could also first discuss it here first if you don't want to create the wish directly.

It's not about miscategorizations (well mostly not) – if you use deepcategory a few times you'll see what I mean and maybe I can bring some examples at later point...I thought the examples in the wish may suffice to communicate what is meant: for example one may run deepcategory:"Maps of the world" (along with other search params/terms) to find maps where the year shown is not yet specified in the categories or that relate to a specific topic area; then it may show many files like this and this (imagine this was every second image in the resulting wall of images) because the cat has a subcategory "c:Category:Maps of the world in art"; by seeing that cat at say spot 14 in the list of top cats by number of files provided to the deepcat search results (as shown in the wish's UI mockup image), the user could easily identify that cat as being unwanted and remove those with a click on × 14 Maps of the world in art. The subcat does belong into that cat but one may not want to see its files in the deepcat wall of images. Prototyperspective (talk) 19:42, 22 September 2025 (UTC)

Gah! Sorry @Prototyperspective I only saw this now!

What's your idea how it might be done? Eager to hear, we could also first discuss it here first if you don't want to create the wish directly.

So I was thinking of replacing the category page on Commons (possibly within the existing MediaSearch extension) so that it's similar to how the search page works - i.e. the contents of the page are retrieved via a search (incategory: search in this case) and then displayed as a wall of images with a scroll

... and then we could also fetch subcategories via an API call, and as the user scrolls get images from those subcategories again via incategory: search. This way we wouldn't need deepcat.

If you think that sounds good let me know and I can create a wish

(fwiw what I'm describing would be a pretty big chunk of work, and not necessarily easy to get prioritised, but it's do-able and I think would be a big improvement to category browsing on Commons)

CParle (WMF) (talk) 15:04, 21 October 2025 (UTC)

@Prototyperspective just chasing you up on this - does what I'm saying make sense? Do you think I should create a wish for what I'm describing above? CParle (WMF) (talk) 15:23, 17 November 2025 (UTC)

Having a wall of images and infinite scroll on the category page sounds like a good idea. However, the subcategories really still need to be shown at the top or in some sidebar as links. And also one shouldn't just show random files from random subcategories. I think a main value would come from ranking the results so that e.g. more-used files show higher up (all described in the techsurvey link). Btw, it would also be good to show the file's relation to the current cat as e.g. a cat chain on hover. Which subcategory/ies are excluded or possibly included often needs to be choosable/modifiable for the files to be relevant. It's also important to know that often a large category is overflowing with hundreds of low-quality files directly in the cat and has all the files that are actually useful and/or of highest quality in subcategories, possibly quite deeply nested in them. So just showing the files directly in it and then after one has reached the end files from random subcategories wouldn't be a good idea.

Another important thing is that subcategories can go very deep in many layers like for example Renewable energy->Renewable energy statistics->Renewable energy maps->Solar energy maps->Solar energy maps of the world (maybe not a good example and not nearly as deep or moving toward tangential relation to starting cat as is the case with countless other cats). The user may go to the renewable energy cat and in scenario 1 look for RE-related good quality photos of all sorts about any subtopic and scenario 2 for RE-related data graphics like but not only specifically an up-to-date used-on-Wikipedias solar energy world map. The functionality described would be most useful when user 1 goes to a photos-specific subcat and user 2 to the statistics subcat to then see deepcat results; however often such subcats don't exist or the user doesn't know about it. Then one could exclude subcats not interested in and benefit from the wall-of-images-with-ranked-subcat-files. Here user 1 would have to exclude the statistics & maps subcats with two clicks because there actually is no photos-subcat and the files dispersed throughout. They may not want to dig for long through all the subcats as they just want a good-quality photo of any type of RE that meets their needs and criteria.

The feature you're describing wouldn't make this wish redundant. Having a wall of images is just one half of the coin and it would also be useful if one, and including new visitors, could easily switch to a wall of images viewmode without having to know the incategory trick and all the typing. On that, also see W413: Larger thumbnails in category views . Your idea may be more related to that so far. The other half however is that files from subcategories are shown already at the first shown files (or maybe at least after the first scroll) with all files being ranked there to put files of likely high quality high up and vice versa; scrolling further several times could eventually get a new batch of files more deeply nested. Sorry that I didn't reply earlier...it was on my list to do right now for quite a while so I meant to reply at any time, there's just so many things. Prototyperspective (talk) 00:44, 18 November 2025 (UTC)

Also, could the wish please be accepted? Also pinging @MusikAnimal (WMF):. Prototyperspective (talk) 00:51, 18 November 2025 (UTC)

Just to quickly chime in: this has been accepted now and I think it would be helpful to have a spinoff wish as @CParle (WMF) suggested above. MikeZ-WMF (talk) 14:38, 4 December 2025 (UTC)

Added the spinoff wish https://meta.wikimedia.org/wiki/Community_Wishlist/W478

CParle (WMF) (talk) 15:49, 4 December 2025 (UTC)