User:BPirkle (WMF)/Stuff/Local Dev/Vagrant

From Meta, a Wikimedia project coordination wiki

The following is a bit dated. I may update it someday. I'm leaving it as-is for reference.

I find myself frequently bouncing between various Phabricator tasks. Each may have its own particular set of extensions that I need loaded, or other local settings that I need set to particular values. I also sometimes find it convenient to execute arbitrary code during tested, so that debugging info is appended to output and/or so that I can step through it with xdebug. So my local development setup allows easily specifying these things on a per-task basis, and automatically switching between them when I switch tasks.

Also, after improperly using a database-related function that generated an unindexed query and slowed down live Wikipedia, I wanted an additional safety net. So I added automatic runtime detection and display of dubious queries to the debugging output.

There may be better ways, but for right now this works for me.

tl;dr: I combine some file in a git-excluded directory with a custom extension and a one-line change to LocalSetting.php. Then I name my local git branches after my tasks, one branch per task. I then create the following files, named after the branch, as needed (all three files are optional):

  • LocalSettingsT123456.php (local settings override specific to the task)
  • T123456.php (arbitrary code executed for web processes. Any output displays at the bottom of the page)
  • T123456Cli.php (arbitrary code executed for command line processes. Any output displays after the normal command line output)

These files happily persist in the git-excluded directory, don't step on each other, and automatically activate themselves depending on the active branch.

Here are the long instructions:

  • Install the MediaWiki Vagrant box per instructions at: mw:MediaWiki-Vagrant
  • Add the following to <git root>/.git/info/exclude:
/local
/extensions/LocalBag
  • Create a "local" directory at the root level of the MediaWiki code (same level as index.php)
  • Create a file within this directory named (somewhat redundantly) LocalSettingsLocal.php, with the following contents:
<?php

wfLoadExtension( "LocalBag" );

$wgAutoloadClasses = [
	'LocalBagSpi' => $IP . '/extensions/LocalBag/includes/LocalBagSpi.php',
	'LocalBagLogger' => $IP . '/extensions/LocalBag/includes/LocalBagLogger.php',
];

$wgMWLoggerDefaultSpi = [
	'class' => LocalBagSpi::class,
];

// If using the mediawiki vagrant box, the /vagrant/LocalSettings.php file may have already done
// this with the default value.  So redo it to use our desired override.
MediaWiki\Logger\LoggerFactory::registerProvider(
	ObjectFactory::getObjectFromSpec( $wgMWLoggerDefaultSpi )
);

// Custom global for identifying active task by git branch.  This is fragile for some
// branch names, but as it is for local dev only, the cost of failure is low
$wgLocalTask = trim(
	implode( '/', array_slice( explode( '/', file_get_contents( $IP . '/.git/HEAD' ) ), 2 ) ) );

if ( is_file( __DIR__ . '/LocalSettings' . $wgLocalTask . '.php' ) ) {
	include_once __DIR__ . '/LocalSettings' . $wgLocalTask . '.php';
}
  • Edit LocalSettings.php to add the following line at the bottom
require_once __DIR__ . '/local/LocalSettingsLocal.php';
  • Add the code for the LocalBag extension under the extensions directory (the name "LocalBag" was inspired by the "BagOStuff" class). This code is available here:

https://github.com/bpirkle/local-bag

And can be cloned by executing this from within your extensions directory: 

git clone https://github.com/bpirkle/local-bag.git LocalBag

Now, every time I start a new task, I do the following:

  • update master, then create a git branch named for the new task
git checkout -b T123456
  • create a local settings file specific to the task within the "local" directory. For example, here's my LocalSettingsT182748.php file. This task was related to external diff engines, so it was helpful to specify one:
<?php

$wgDebugLogFile = '/var/log/mediawiki/debug-T182748.log';
$wgExternalDiffEngine = '/usr/bin/diff';
  • create a php file specific to the task within the "local" directory. For example, here's my T165768.php file. This task was related to exception detail display, so it was helpful to throw an exception on every page load.
try {
	throw new Exception( 'foo' );
} catch ( Exception $e ) {
	print MWExceptionRenderer::reportOutageHTML( $e );
}

Because the per-task files are only brought into the codebase when their associated git branch is active, I didn't have to worry about my local environment throwing exceptions or using an odd diff engine when I was on an unrelated task. Those intrusive changes just automatically disappeared when they were not needed.

Also, the LocalSettingsLocal.php file is a nice place to put local settings that should apply across all your tasks. You can, of course, just put these in LocalSettings.php, which is already git ignored. But I like keeping as many of my customizations as possible in the "local" directory. FWIW, my LocalSettingsLocal.php currently contains the following additional settings: 

$wgShowExceptionDetails = true;
$wgDebugToolbar = true;
$wgShowDebug = true;
$wgDevelopmentWarnings = true;
$wgDebugDumpSql = true;
$wgDebugLogFile = '/var/log/mediawiki/debug-{$wgDBname}.log';
$wgDebugLogGroups = [
        'bdp' => '/var/log/mediawiki/bdp.log',
];

// Force this back to default (database), the version in /vagrant uses redis
$wgJobTypeConf = [
        'default' => [ 'class' => JobQueueDB::class, 'order' => 'random', 'claimTTL' => 3600 ],
];

// Disable automatic job run on normal page load
$wgJobRunRate = 0;

Final trick: the dubious query display can be very useful, but it can also be very noisy. I often find myself testing the same operation repeatedly during development, with the dubious query log displaying the same (sometimes long) list of results every time. The $wgLocalBagIgnoreNeedles global variable allows configuring substrings that, if found in a particular query, will cause that query to be ignored, so that it does not appear in the log. ("Needle" comes from the PHP needle/haystack parameter naming for the strpos function.)

Because I have per-task local settings files, I can easily configure these ignores per-task. I find it most helpful to set up my ignore needles at the beginning of a task, before I make any changes, to ignore existing queries that don't apply directly to my task. Then, if any new dubious queries appear in the log, I know they are worth special consideration. For example: 

$wgLocalBagIgnoreNeedles = [
	'SELECT  ipb_id,ipb_address  FROM `ipblocks`    WHERE ipb_address',
	'SELECT  page_id,page_len,page_is_redirect  FROM `page`    WHERE',
];
Retrieved from "https://meta.wikimedia.org/w/index.php?title=User:BPirkle_(WMF)/Stuff/Local_Dev/Vagrant&oldid=21921610"