Planned Updates to Module:ItemList

This is the next installment of the extremely sporadic series where I try and make the Wiki better using Semantic MediaWiki (SMW). This week's target is Module:ItemList, the cool piece of code behind most (all?) pages that list or compare items. Examples include the item listings on Category:Gloves and Shadowy Items, as well as the BiS annotations on equipment pages.

This module currently relies on a series of Lua tables which exhaustively list all items in Fallen London. This "database" is a maintenance burden because it is hand-maintained and duplicates the content already present in the item pages themselves. These properties make it well-suited for replacement by Semantic MediaWiki queries.

Goals for the Update

• Preserve the editor-facing interfaces that would be used on article pages, i.e. {{Item}}.
• The single source of truth for details about an item is that item's wiki page
• Reduce the server load caused when an item gets added or updated
• Try and get this all by next Thursday, so everything is in place when the new Rat Market item becomes available
• Non-goal: It is considered acceptable to change the template-facing interface of Module:ItemList itself.

Contemporary Status

Semantic MediaWiki can be queried in Lua Modules (via Semantic Scribunto). So the meat of the update is to replace the tables and the code that loads them with SMW queries that produce equivalent tables. The Lua code that queries those tables and produces those results will be modified only minimally.

A working version of this is present and rolled out on the dev wiki dev:Module:ItemList. Module:ItemList has received a few updates on the prod wiki since that code was forked, so merging the changes is slightly more complicated than a straight copy-paste, but the major problems are already solved.

The ItemList tables include the items themselves, their equipment slots, their effects (stat bonuses), and restrictions. The current Wiki already encodes the first two as SMW properties: Property:Equips in slot and Property:Has effect. All item restrictions currently used by Module:ItemList can be inferred from item categories, e.g. Category:Fate, Category:Ambition: Nemesis Items, or Category:Hallowmas Equipment. SMW can query page categories.

There is a bit of a chicken-and-egg problem, which is that currently some (many?) item pages set these categories via Module:ItemList. For example, belongs to Category:Election Equipment. The only place this is currently set is in Module:ItemList/Companions.

Implementation Plan

The first stage is finding a different way for item categories to get on the page that doesn't require a billion page edits. My current plan is to update {{Item}} to read the Access info parameter. (We are already using the Access parameter to set Fate and Retired categories.) In the example of the Anchoress above, the Access info is Election (historical). My impression is that this parameter has generally been deployed pretty broadly across the Wiki. There are some odd edge cases (e.g. four or five different ways to get items during Christmas), but I think a #switch statement should be able to handle that without getting too gnarly.

Stragglers can have categories set manually. Reading Access info is just trying to be a 95% solution.

In parallel I'll work to updates to Module:ItemList. As mentioned, there will be some template-facing interface updates. For the most part these updates are to push filter conditions into queries. In my testing, the main performance bottleneck is the amount of items returned by SMW to Lua. Code changes that filter results in SMW rather than Lua are preferred. So for example, the listing of gloves on Shadowy Items currently pulls a Lua table containing all gloves, and filters for Shadowy gloves in Lua; it's better for performance if SMW filters the gloves before handing off to Lua.

As far as I'm aware, updated templates are {{Item}}, {{ItemList}}, {{ItemListByQuality}}, {{ItemListByClass}}, and {{BestInSlot}}. To minimize disruption to the Wiki, I'll create new beta versions of Module:ItemList and those templates, then update the real templates to use the beta modules and new interface, then update the main Module:ItemList, and then update references in the real template to the updated Module:ItemList.

Then hopefully the new rat-item will come out and maybe everything will work!

The nice this about testing this code is that by their very definition, the item listings are exhaustive. So it's easy to compare two lists of every hat in the game and find discrepancies.

Future Work

The internal workings of Module:ItemList are still somewhat opaque to me. It's a complicated piece of code which does a lot of whizz-bang things. For my own sake I've tried to minimize changes to how it works, but there's possibility for future improvements or features when using SMW as a data source instead of hand-made tables.

For one thing, the list of access restrictions is a flat list. I presume this is for simplicity so that editors don't have to wrangle with complex data structures when adding restrictions. If they are generated by code, they are allowed to be more convoluted. This may aid in representing mutually-exclusive restrictions, such as Ambitions or Professions.

Current Status

We are live! \o/

List of outstanding differences between ItemList and SMW queries can be viewed at User:PSGarak/Sandbox. The majority of discrepancies are related to the Grand Clearing-Out: several items are listed as Election in ItemList but their access info says GCO; and several items that were Fate-locked or Legacy were available with Cthonic Scrip, including several in the 40-60 range due to an edge case. The second-largest category is various backer rewards. Neither of these seem huge, and the long tail can be whittled down over time, although the GCO stuff is likely to get updated again next year when there's a new recurring festival.

The Template {{Item}} adds categories based on {{{Access}}} and {{{Access info}}}. This may need occasional tweaks if new events or restriction types come out. The call to ItemList:categories has been pulled, categories are no longer populated from Module:ItemList tables.

The updates to Module:ItemList are in place, as are some minor updates to Module:Item. I did some clear-cutting of ItemList while I was in there. There were a variety of helper functions for filtering item lists, which are no longer needed when the fetched results are pre-filtered. I also elected to delete rather than update a handful of functions that weren't being used. Then I went through and deleted other stuff that wasn't being used, because I like seeing file sizes get smaller.

Changelog:

• New internal functions for returning lists of item effects & qualifiers.
  • smw_fetch_items (lower-level)
  • fetch_items_class_quality (higher-level)
• Functions updated under the hood to use SMW.
  • create_list
  • find_items
  • find_minmax
• Removed filtering functions that are no longer needed:
  • no_regular_qualities
  • quality_match
  • has_quality
  • find_item_in_class,
• Updated interface:
  • internal function item_match(name, item, quality restrictions) slimmed to item_match(item, restrictions)
  • exported function items_exist: "name" argument removed--this was only used to flag items with no ItemList data
  • export function is_retired is now retired; instead Module:Item checks the Access parameter to {{Item}} (this change is already live on the Wiki)
• Other deleted functions:
  • deal_with_menace
  • create_item_line_previous
  • list_size and list_size_body (already documented as obsolete)
  • verify_item (also documented as obsolete)
• Other notes
  • Special-casing is now handled differently. She is currently excluded from all ItemList queries. Let me know if you want her back.
  • No changes were needed for {{ItemList}} or derived templates. They were not affected by any interface updates.

Module:ItemList sub-modules could be locked when desired.