Extension:JsonConfig

The JsonConfig extension allows other extensions to store their configuration data as a JSON blob in a wiki page.

Available features and usage patterns

File:JSON visualization by JsonConfig extension.png

Example JSON visualization rendered by the JsonConfig extension

• You may use JsonConfig to store data in a number of ways:
  • in a single configuration page, e.g. a number of settings for your extension in Config:MyExtSettings (Config is the default namespace associated with the JsonConfig extension);
  • in a set of pages with similar structure residing in their own namespace, e.g. IP addresses of known proxies or event log schemas named "Proxy:Opera", or schemas named Schema:AccountCreation
  • using it only for pages whose titles match a regex pattern, e.g. Config:Proxy:Opera or Config:Opera.proxy. This avoids filling a wiki with lots of namespaces, one for each content model.
• You may provide a content class to do data validation (beyond valid JSON) and normalization.
• You may provide a view class to customize HTML display.
• You can store data:
  • "one-per-wiki", "one-per-cluster", or even one per some "family" (different structure of the caching key for shared memcached);
  • in a public or private wiki and accessed with credentials;
  • in a separate cluster, and do remote notification when changed.

On-wiki usage

Current

• Commons tabular data sets
• Commons map data sets
• Dashiki dashboard configurations
• Others?

Past

• OODA WIKI Zero operator configurations (see Extension:ZeroBanner)

Installation

• <translate> [[<tvar name=2>Special:ExtensionDistributor/Extension:JsonConfig</tvar>|Download]] and place the file(s) in a directory called <tvar name=name>Extension:JsonConfig</tvar> in your <tvar name=ext>extensions/</tvar> folder.</translate>
• <translate> Add the following code at the bottom of your <tvar name=1>LocalSettings.php </tvar> file:</translate>

  wfLoadExtension( 'Extension:JsonConfig' );
  

• Configure as required.
• Y <translate> Done</translate> – <translate> Navigate to <tvar name=special>Special:Version</tvar> on your wiki to verify that the extension is successfully installed.</translate>

<translate> To users running MediaWiki <tvar name=1>1.29</tvar> or earlier: </translate>

<translate> The instructions above describe the new way of installing this extension using <tvar name=LoadExtension>wfLoadExtension()</tvar>.</translate> <translate> If you need to install this extension on these earlier versions (MediaWiki <tvar name=1>1.29</tvar> and earlier), instead of <tvar name=code>wfLoadExtension( 'Extension:JsonConfig' );</tvar>, you need to use:</translate>

require_once "$IP/extensions/Extension:JsonConfig/Extension:JsonConfig.php";

Configuration

$wgJsonConfigs

This variable defines profiles for each type of configuration pages. $wgJsonConfigs is an associative array of arrays, with each sub-array having zero or more of the following parameters. By default, JsonConfig uses the string key as the model ID that this profile represents, but in case you want to reuse the same model ID in more than one profile, you can override it with the model parameter.

parameter	type	default	description
model	string	(key in $wgJsonConfigs)	Model ID to use when creating a new page. If none is given, the array key of this profile becomes the model ID. If this value is set to null, the built-in module ID JsonConfig will be used. The model ID must also be listed in the $wgJsonConfigModels except for null or 'JsonConfig'.
namespace	int	NS_CONFIG (482)	Namespace in which these configs will reside, or 'Config:' by default
nsName	string/false		For non-default (not NS_CONFIG 482) namespace, assigns it a canonical (English) name. If more than one $wgJsonConfigs array uses the same namespace, only set the name once. If not found, a warning will be given, and the namespace will get automatic "ConfigNNN" name. In case this config is not stored locally, nsName will be used as remote namespace, without local declaration. If you want to share namespace with other, non JsonConfig content pages, set this value to false.
nsTalk	string	nsName+_talk	For non-default namespaces, assigns their talk pages a canonical (English) name
pattern	string	(all)	Regular expression to match the page title in the given namespace. If empty, the configure will apply to all the pages in the namespace. To match the legacy "subspace" behavior, for isSubspace=true, use ^/^name:./ pattern, and for isSubspace=false, use /^name$/ pattern.
isLocal	bool	true	If true, this configuration will not be shared across the cluster, but rather each wiki would have a local config. If false, this configuration is meant to be shared across multiple wikis, and in order to make this specific wiki store it locally, set the 'store' field. This flag affects the caching key, so a config page with the same name would be shared in memcached if this setting is false, but will be unique per wiki if true.
cacheExp	int	24*60*60	How long (in seconds) should the value be cached in memcached
cacheKey	string		If the value is set, add that string to the caching key. Use cacheKey to invalidate previous cached values - e.g. when making an incompatible change that would modify cached values, and you worry that you may need to roll back. This key can also be used to allow all language wikis in the same family to share the same cached value, while being stored in the same page but on different wikis. For example, OODA WIKI could use "wp" and Wikiquotes could use "wq".
flaggedRevs	bool	false	If true, will attempt to get the latest revision that has been verified via flagged revs extension, if null, flagged revision will be checked, and if available, used. If false, will always use the latest revision.
remote	array		An object with parameters describing how to access configuration from another wiki. Must be present if 'store' is not set, and isLocal is false.
‑ url	string		Non-default API endpoint to get config from remote wiki just for this configuration profile. If not specified, will use the value of $wgJsonConfigApiUri.
‑ username	string		If given, will be used to authenticate to the remote configuration storage.
‑ password	string		See username.
store	array		Set this value to true or an array to make the current wiki actually host this configuration data. This value will automatically populate if missing and isLocal is true.
‑ cacheNewValue	bool	true	When config changes, how should the cache change. If false, deletes cached value, forcing other wikis to re-request it. true sets the new value into cache. This is mostly useful if the wiki that stores and wikis that consume this configuration share the same cache. Beware that in case of false, if consumer wiki requests the new value too fast, it might get it from a possibly stale slave DB.
‑ notifyUrl	string		Optionally specify remote API URL that will be called when configuration changes.
‑ notifyUsername	string		Optional username to use with notifyUrl
‑ notifyPassword	string		Optional password to use with notifyUrl

$wgJsonConfigModels

This variable defines which custom content class will handle which Model ID. More than one Model ID may be handled by the same content class. All content classes must derive from the JCContent class; if the modelID maps to null, then the default JCContent class handles the model ID.

Example:

$wgJsonConfigModels['Config.MyExtSettings'] = 'MyExt\MyContentClass';

If you implement a separate class to do HTML rendering, you can specify your config model as an array with an extra view class:

$wgJsonConfigModels['Config.MyExtSettings'] = [
    'class' => 'MyExt\MyContentClass',
    'view'  => 'MyExt\MyContentViewClass', // easier to extend JCDefaultObjContentView or JCDefaultContentView
];

Examples

Before version 1.0.0, JsonConfig used name and isSubspace values to customize which pages belonged to the given content model. Both have been removed in version 1.0.0, and replaced with the pattern parameter that accepts a regular expression. Make sure to surround your patterns with ^ and $ symbols, and add slashes (PHP style)

Hello World

The simplest case is a single configuration page without any validation, stored locally on each wiki. Just add these settings to LocalSettings.php

// Content model is 'JsonConfig.MySettings'
// Model class is set to NULL to allow non-validated data
$wgJsonConfigModels['JsonConfig.MySettings'] = null;

$wgJsonConfigs['JsonConfig.MySettings'] = array(
  'pattern' => '/^MySettings$/', // Page name in Config namespace
);

The above enables namespace "Config" on the local wiki, but only allows the "Config:MySettings" page to be created, not any other page in that namespace. You can store well-formed JSON data in the page.

To read the MySettings data in PHP, use a TitleValue object to access the page, then request its content:

use JsonConfig\JCSingleton;
$tv = new TitleValue( NS_CONFIG, 'MySettings' ); // DB Key
$content = JCSingleton::GetContent( $tv );
if ( $content->isValid() ) {
    $data = $content->getData();
    ...
}

Supporting Wikimedia templates

Some templates (e.g. Template:TNT) from Wikimedia projects require JsonConfig to support the Data namespace on commons for mw.ext.data.get. You can use the following config to support that usecase.

wfLoadExtension( 'JsonConfig' );

$wgJsonConfigEnableLuaSupport = true; // required to use JsonConfig in Lua
$wgJsonConfigModels['Tabular.JsonConfig'] = 'JsonConfig\JCTabularContent';
$wgJsonConfigs['Tabular.JsonConfig'] = [ 
        'namespace' => 486, 
        'nsName' => 'Data',
        // page name must end in ".tab", and contain at least one symbol
        'pattern' => '/.\.tab$/',
        'license' => 'CC0-1.0',
        'isLocal' => false,
];

$wgJsonConfigModels['Map.JsonConfig'] = 'JsonConfig\JCMapDataContent';
$wgJsonConfigs['Map.JsonConfig'] = [ 
        'namespace' => 486,
        'nsName' => 'Data',
        // page name must end in ".map", and contain at least one symbol
        'pattern' => '/.\.map$/',
        'license' => 'CC0-1.0',
        'isLocal' => false,
];
$wgJsonConfigInterwikiPrefix = "commons";

$wgJsonConfigs['Tabular.JsonConfig']['remote'] = [ 
        'url' => 'https://commons.wikimedia.org/w/api.php'
];
$wgJsonConfigs['Map.JsonConfig']['remote'] = [
        'url' => 'https://commons.wikimedia.org/w/api.php'
];

Multiple configs shared in a cluster

Lets say we decide to store the trusted proxies' IPs as "Config:Proxy:SomeName" pages on Meta-Wiki and share this data with the cluster.

// All wikis must have this configuration:
$wgJsonConfigs['JsonConfig.Proxy'] = [
  'pattern' => '/^Proxy\:./', // require at least one letter after the ':'
  'isLocal' => false,
];

// The LocalSettings.php for all wikis except Meta-Wiki will set this URL to Meta-Wiki's API endpoint:
$wgJsonConfigs['JsonConfig.Proxy']['remote'] = 'http://meta.wikimedia.org/w/api.php';

// LocalSettings.php for Meta-Wiki will indicate that the data should be stored there
$wgJsonConfigs['JsonConfig.Proxy']['store'] = true;

If instead you want to dedicate a separate namespace to proxies, the parameters would change to:

$wgJsonConfigs['JsonConfig.Proxy'] = [
  'namespace' => NNN, // NNN is the number you would reserve for Proxy namespace
  'nsName' => 'Proxy', // Canonical namespace name
];

Validation

Most of the time one would also want a custom content class with its own validation. JSON pages are handled by the content classes which derive from JCContent. The content class is responsible for parsing and validating raw text. JCContent does not do any validation beyond JSON parsing, but you may choose to derive from it and override JCContent::validate(). Better yet, you may derive from the JCObjContent class which provides a number of validation primitives, and only override JCObjContent::validateContent().

// This should be done on all wikis, including Meta-Wiki
$wgJsonConfigModels['JsonConfig.Proxy'] = 'ProxyExt\ProxyContent';

For the sake of this documentation, let's presume that the proxy config page describing Opera Mini servers has this format:

{
    "enabled": true,
    "comment": "See http://... for updates",
    "ips": [
        '37.228.104.0/21',
        ...
    ]
}

Here is the content class to validate that data.

use JsonConfig\JCObjContent;
use JsonConfig\JCValidators;

class ProxyContent extends JCObjContent {

    /**
     * Derived classes must implement this method to perform custom validation
     * using the check(...) calls
     */
    public function validateContent() {
        // 'enabled' must be a boolean, true by default.
        // JCValidators::* already handle localized error messages
        $this->testOptional( 'enabled', true, JCValidators::isBool() );
        // an optional field 'comment' of type string
        $this->testOptional( 'comment', '', JCValidators::isString() );
        // 'ips' must be a list of valid CIDR ranges
        // field is not optional when default value would not pass validation
        $this->test( 'ips', self::getIpValidator() );
    }

    private static function getIpValidator() {
        // JCValue $value is a value of the field being checked wrapped with the status information.
        //              $v->getValue() actual value being examined
        //              $v->isMissing() if the value is not present in the data
        //              $v->defaultUsed() if the value did not exist and a default was supplied
        // array $path the location of this field in the hierarchy - each value is either string or an int
        // JCObjContent $self - this object, useful to get access to other fields via $self->getField()
        // You may modify the value stored inside, set additional flags, or report an error
        // using the $v->error( $key, $path, ... ) function
        return function ( JCValue $value, array $path, JCObjContent $self ) {
            $isErr = false;
            $v = $value->getValue();
            if ( is_string( $v ) ) {
                // user supplied a single string, treat as an non-assoc array
                $v = array( $v );
            } else {
                // ensure that $v is an non-assoc array, and all of its values are strings
                $isErr = !JCUtils::isList( $v ) || !JCUtils::allValuesAreStrings( $v );
            }
            if ( !$isErr ) {
                // @todo: do the rest of the IP validation and set $isErr to true on failure
            }
            // Error message might be in this form:
            // "Parameter \"$1\" must be an array of valid non-restricted (no private networks) CIDR IP blocks"
            if ( $isErr ) {
                $value->error( 'my-proxyconfig-err-ips', $path );
            } else {
                $value->setValue( $v );
            }
        };
    }
}

Customizing behavior on the storage wiki

You may also want to customize viewing and page creation on the storage wiki (Meta-Wiki in the above example). There are two ways to do this: inside your JCContent-derived class, or via a separate "view" class that derives from JCContentView. The second approach is preferable as it cleanly separates the architecture and the view class need only exist in the storage wiki (e.g. Meta-Wiki), not on all the wikis that use the data.

To override default value in JCContent-derived class, override the constructor and set the $text value to the new default if it came in as NULL before passing it to the parent constructor. To override the HTML generation, override JCContent::getHtml().

With the recommended way, create a view class that derives from JCContentView or a more feature-rich JCDefaultContentView. For JCContentView, you will need to implement valueToHtml() and getDefault(). By default, the view is implemented by the JCDefaultContentView class, which can also be used as a customizable base if you just need minor adjustments to the way it looks.

// Changing model definition from the above - should be done on all wikis, including META
$wgJsonConfigModels['JsonConfig.Proxy'] = [ 'class' => 'ProxyExt\ProxyContent' ];

// Add the view class - should only be done on the storage wiki (e.g. META)
$wgJsonConfigModels['JsonConfig.Proxy']['view'] = 'ProxyExt\ProxyView';

class ProxyView extends JsonConfig\JCDefaultObjContentView {
    public function getDefault( $modelId ) {
        return <<<JSON
{
    "comment": "See http://... for updates",
    "ips": [
        "N.N.N.N/NN",...
    ]
}
JSON;
    }
}

Best practices

• Extensions that use JsonConfig should add their configurations to the $wgJsonConfigs and $wgJsonConfigModels in the main extension file.
• If you share config data among multiple wikis, document the key name used in $wgJsonConfigs[], and initialize the 'store' / 'remote' section in LocalSettings.php. This is better than introducing a number of global variables that duplicate the config functionality. See for example Wikimedia's configuration of the Graph extension across multiple wikis (although that uses complex multi-wiki setup rather than simple config variables).

Status

Implemented features

• JSON parsing converts JSON text into either an array or an object
• Visualization shows JSON as an easy to view table rather than code, with some extra highlighting. For example if the value is not provided and a default is used, it is shown in gray, or when the value is the same as default, it shows as purple. For example, see this and this.
• Code Editor simplifies JSON editing
• Custom Validation performs complex checks such as checking that the value is in the proper format or that a user ID exists.
• MemCached caching stores json blobs in memcached under custom keys and expiration policies, and resets them on save.
• Flagged Revisions support allows configurations to be marked as "reviewed" before going into production
• Localization of most basic interface elements has been done in many languages, and it would reduce translation work if most common messages would be done just once in one place.

Unimplemented nice-to-haves

These features would be desirable to more than one type of configs:

• Schema validator - Validate against JSON Schema, as most extensions might not need complex validation rules, or might want to combine schema plus extra validation.
• Custom editor - Zero team has been thinking about implementing a more complex editor, possibly based on JSON Schema.
• API query support - Allow config pages to be returned as regular API results in all formats - json/xml/... instead of text blobs:

  api.php ? action=query & titles=Config:Proxy:Opera & prop=jsonconfig

• Localization - it would be good to be able to show localized descriptions for each configuration key

External access

The stored configuration data may frequently be needed by some external agent such as JavaScript, bot, or other programs. JavaScript could use either JSONP to access needed data by calling standard action=query&rvprop=content API, or we could develop a forwarding service if CORS is unavailable. Extension authors may choose to add their own API modules to provide domain-specific information. Lastly, the rvprop=jcddata Query API parameter would return JSON data as part of the API result, not as a text blob that rvprop=content would return.

// NOT IMPLEMENTED! Use regular action=query API until we decide how to do it right
var req = {
    format: 'json', action: 'query',
    titles: 'Proxy:Opera',
    prop: 'revisions', rvprop: 'jcdcontent', indexpageids: '',
};
$.ajax({
    url: '//meta.OODA WIKI.org/w/api.php',
    data: req, cache: true, dataType: 'jsonp',
    success: function(result) { /* handle errors and warnings, process content */ }
});

See also

• Help:Tabular Data