Community Tech/Wikitext editor syntax highlighting

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search

Other languages:
Deutsch • ‎English • ‎Esperanto • ‎español • ‎français • ‎Bahasa Indonesia • ‎italiano • ‎日本語 • ‎한국어 • ‎Nederlands • ‎português • ‎русский • ‎Tagalog • ‎Tiếng Việt • ‎中文
Syntax Highlighting is now live as a Beta feature on all LTR projects!

The Wikitext editor syntax highlighting project aims to make it easier for contributors to edit wikitext by using colors and shade to make it easier to visually separate article text from the code for links, references and templates. This will help editors working on paragraphs with many references, or complicated templates where it's easy to miss a curly brace and mess up the template. We are using Extension:CodeMirror for this.

Screenshot of Syntax Highlighting in English.
Screenshot of Syntax Highlighting in Hindi.
Screenshot of Syntax Highlighting with the popup for first-time users.

The syntax highlighting is available in both the classic wikitext editor as well as the New Wikitext editor (NWE), as a button (Codemirror-icon.png) that appears in the toolbar. The user can switch the highlighting on and off using that button. This is done via a hidden preference, so if you turn it off then it will stay off, until you turn it on again.

Currently, Wikitext syntax highlighting is available as a beta feature. We welcome your feedback on improving it and helping us into rolling it out further in the future.

The feature was primarily built by volunteer developer Pastakhov; Community Tech helped him to finish the feature, and get it deployed. On the Community Tech side, Niharika and Ryan did the bulk of the work, with help from the Editing team's Ed Sanders on integration with Visual Editor's wikitext mode.

Rationale[edit]

We have decided to use Extension:CodeMirror as the syntax highlighting tool.

Here are the advantages that CodeMirror offers:

  • Much faster performance, it's been heavily optimized
  • Uses "wysiwyg"-style effects, to make it easier to see the impact of the wikitext elements -- for example, bold words are rendered in bold, italics in italics.
  • Section headings are bigger, which helps the editor navigate on the screen.
  • Links are underlined and blue, which makes them more recognizable as links. When there's displaytext that's different from the link, the link is in blue, and the displaytext after the pipe is black, to indicate that's the part that will be seen as text.
  • Uses bold colored text for template parameters, light colored text for entries.
  • Uses light background highlighting on links, and not on other elements. Too much background highlighting can make the wikitext harder to scan.
  • Waits for the editor to finish typing before it changes highlighting colors, to allow the editor to finish typing the link or the bold/italics without making everything below that line flash as different colors.

The other tools we looked at before deciding on CodeMirror were wikEd by Cacycle, and Syntax highlighter by Remember the dot.

Status Updates[edit]

Sept 19, 2017[edit]

Happy to report that a couple important bugs are fixed! Ctrl-F browser search now works for both the regular edit window and the new wikitext mode, and word jumping using the option+arrow key now works on Mac browsers.

Fixing browser spellcheck is close to done; we'll be releasing that fix soon.

We still need to fix an issue with the new wikitext mode not respecting whether the beta feature is turned on or off -- you can turn it off on the Beta features page, but still see it on in the new wikitext mode. You can follow progress on this ticket: phab:T173480.

Sept 7, 2017[edit]

We're currently working on an important bug -- the Ctrl-F browser search only works if the text you're looking for is currently displayed in the edit window, or just below it. We've been investigating this for the last couple of weeks; here's an update on how it's going. You can follow progress on this ticket: phab:T174480.

We were looking at using the search add-on that GitHub uses in their version of CodeMirror, and it's not an acceptable solution for us. It only works when you've got the cursor in the edit window -- if you're not inside the edit window, ctrl-F still gives you the browser search. That's not very discoverable -- for some people, it would just be broken sometimes. The add-on also uses a combination of ctrl-F to start the search and ctrl-G to move from one highlighted word to another, which is also not very discoverable. So that is a dead end for us.

We're now investigating a couple other options, including loading the page outside the edit window in the background, after the text in the edit window is displayed. This might slow things down, so we're testing the performance.

We're still focused on solving this issue; I'm sorry it's not fixed yet.

Aug 23, 2017[edit]

Some feedback from Andy M. on Wikimedia-L to consider:

"My first impression is that it is lacking in contrast - it's far harder, now, to differentiate the various types of content. Indeed the colour pairs used (e.g. #8800CC vs. #AAAAB3) fail WCAG web accessibility guidelines for colour contrast. I realise that choice of styling colours is a "bikeshed" matter, but contrast ratio is a quantifiable and objective accessibility issue. Also, because the script does not load immediately, the larger headings cause the page to "dance" as the script kicks in."

Aug 2, 2017[edit]

Syntax Highlighting has now been released as a Beta feature on all LTR Wikimedia projects! We hope that people love using the new feature. Please leave feedback on the talk page!

We're still working on making it work for RTL languages; you can follow this progress on this ticket: T170001.

Temporarily turned off, see phabricator:T172458.

July 18, 2017[edit]

CodeMirror deployed to Simple English Wikipedia as a Beta Feature.

March 22, 2017[edit]

Team discussion, playing with CodeMirror on CommTechwiki. Performance is good, as advertised. We're generally happy with the highlighting choices. We found some bugs, and some elements that should be improved -- for example, ref tags aren't highlighted in the current version. Ryan took a lot of notes, and he'll be working on these issues -- either working with the developers, or making adjustments on our own version.

March 21, 2017[edit]

Ryan talked to Ed S on the Editing team -- they're interested in including the syntax highlighter in the new wikitext editor. This will require some investigation.

There's a current hacked version by Ed: https://gerrit.wikimedia.org/r/#/c/343878/

This version works by loading CodeMirror underneath the WikiText editor surface, making the text and background of the WikiText editor transparent, and syncing user input from the WikiText editor to the CodeMirror surface as it is entered. The problem is that it's sluggish and has syncing issues -- see phab:T161054 and phab:T161052 for more.

February 27, 2017[edit]

There's an existing extension -- Extension:CodeMirror -- that provides wikitext syntax highlighting on demand, adding a button at the top of the wikitext editor to turn highlighting on and off. (This uses a sticky preference so that an editor can turn it on once, and then see it on all future page loads.) The highlighting is acceptably fast, it takes less than a second even on a lengthy article like English WP's Barack Obama page.

There are two different use cases for syntax highlighting -- essentially, it could be a power editor feature, or a new editor training tool. Very active editors want syntax highlighting to help them edit; if this is the use case that we're focusing on, then we can make it a special preference, and let people decide if they want to use it. If we want to focus on helping new editors, then the feature would have to be on by default, and would require much stricter QA. The comments on the Wishlist proposal and votes indicate that people are requesting the power editor feature.

With that in mind -- we're planning to deploy CodeMirror as a special preference for the old wikitext editor. This will need some instruction on the preferences page to explain how to use the button & turn it on.

The VE team is planning to include this in the 2017 wikitext editor as well; we'll talk to Ed S. to see how we can work together. We'll have to make sure we agree on the color choices. In general, we'll be expected to explain the color choices, so we'll look at the existing CodeMirror colors and make sure we understand the rationale.

Another thing we have to figure out -- how will this work outside of en.wp?


Important links[edit]

Color and style customization[edit]

The colors and styles can be overridden by defining your own colors and styles for the classes in the table below on your common.css page (User:Example/common.css).

CSS Class(es) Element CodeMirror styling (CSS) wikEd's styling (See InitFrameCSS) Schnark's styling Rememberthedot's styling
.cm-mw-skipformatting leading space foo bar  foo bar  foo bar
.cm-mw-list asterisk or number sign * foo bar, # foo bar * foo bar # foo bar * foo bar # foo bar * foo bar # foo bar
.cm-mw-signature,
.cm-mw-hr
signature, horizontal rule ∼∼∼∼, ---- ~~~~, ---- ~~~~, ---- ~~~~, ----
.cm-mw-indenting colons ::: foo bar ::: foo bar ::: foo bar ::: foo bar
.cm-mw-mnemonic HTML entities & & & &
.cm-mw-comment HTML comment <-- comment --> <!-- comment --> <!-- comment --> <!-- comment -->
.cm-mw-apostrophes-bold,
.cm-mw-apostrophes-italic
bold and italic '''bold''' ''italic'' '''bold''' ''italic'' '''bold''' ''italic'' '''bold''' ''italic''
Headings
pre.cm-mw-section-1 heading (size not changed in NWE) = Heading = = Heading = = Heading = = Heading =
pre.cm-mw-section-2 heading (size not changed in NWE) == Heading == == Heading == == Heading == == Heading ==
pre.cm-mw-section-3 heading === Heading === === Heading === === Heading === === Heading ===
pre.cm-mw-section-4 heading ==== Heading ==== ==== Heading ==== ==== Heading ==== ==== Heading ====
pre.cm-mw-section-5 heading ===== Heading ===== ===== Heading ===== ===== Heading ===== ===== Heading =====
pre.cm-mw-section-6 heading ====== Heading ====== ====== Heading ====== ====== Heading ====== ====== Heading ======
.cm-mw-section-header equals signs ==
Templates
example {{cite book|title=foo bar}} {{cite book|title=foo bar}} {{cite book|title=foo bar}} {{cite book|title=foo bar}}
.cm-mw-template other template content foo bar
.cm-mw-template-name template name cite book cite book
.cm-mw-template-argument-name template parameters title= title=
.cm-mw-template-delimiter pipe | |
.cm-mw-template-bracket opening/closing brackets {{ }} {{ }}
.cm-mw-template-ground
.cm-mw-template2-ground
.cm-mw-template3-ground
whole template
(number is the level of nesting)
{{foo|{{cite book|title=foo bar}}}}
Variables
example {{{foo|bar}}} {{{foo|bar}}} {{{foo bar}}} {{{foo bar}}}
.cm-mw-templatevariable variable bar
.cm-mw-templatevariable-name triple-braced variable name foo
.cm-mw-templatevariable-bracket opening/closing brackets {{{ }}}
.cm-mw-templatevariable-delimiter pipe |
Parser functions
example {{#assessment: A | High }} {{PAGENAME}}
.cm-mw-parserfunction parser function parameters High
.cm-mw-parserfunction-name parser function name PAGENAME #invoke
.cm-mw-parserfunction-bracket opening/closing brackets {{ }}
.cm-mw-parserfunction-delimiter delimiter :
Extension tags
Note: The extensions can use own styles and rules for highlighting text inside own tags. For example the Cite extension highlights text inside <ref> tags as a wikitext.
example <poem style="padding: 1em;">string
whole line
string</poem>
<ref name="foo"> </ref> <tag>...</tag> <tag>...</tag>
pre.cm-mw-exttag whole line inside tags
whole line
.cm-mw-exttag text inside tags string
.cm-mw-exttag-name tag name poem
.cm-mw-exttag-bracket opening/closing brackets < >
.cm-mw-exttag-attribute reference attributes style="padding: 1em;"
HTML tags
example <span style="foo">foo bar</span> <span style="foo">foo bar</span> <tag style="foo">foo bar</tag> <tag style="foo">foo bar</tag>
.cm-mw-htmltag-name HTML tag span
.cm-mw-htmltag-bracket opening/closing brackets < >
.cm-mw-htmltag-attribute parameters and values style="foo"

superscript
subscript

foo<sup>bar</sup>
foo<sub>bar</sub>

Foo<sup>bar</sup>
Foo<sub>bar</sub>

Pre-formatted text
pre.cm-mw-tag-pre,
.cm-mw-tag-pre
pre-formatted text <pre>foo bar</pre> <pre>foo bar</pre> <tag>foo bar</tag> <tag>foo bar</tag>
pre.cm-mw-tag-nowiki,
.cm-mw-tag-nowiki
nowiki text <nowiki>foo bar</nowiki> <nowiki>foo bar</nowiki> <tag>foo bar</tag> <tag>foo bar</tag>
Internal links
example

[[foo bar]]
[[foo#section|bar]]

[[foo bar]]
[[foo#section|bar]]

[[foo bar]]
[[foo#section|bar]]

[[foo bar]]
[[foo#section|bar]]

.cm-mw-link-pagename internal link target foo bar
.cm-mw-link # character #section
.cm-mw-link-tosection characters after the # #section
.cm-mw-link-bracket opening/closing brackets [[ ]]
.cm-mw-link-text link label (no additional formatting)
.cm-mw-link-delimiter pipe |
.cm-mw-pagename internal link targets foo bar
External links
example

https://www.example.org
[https://www.example.org]
[https://www.example.org example.org]

https://www.example.org
[https://www.example.org]
[https://www.example.org example.org]

https://example.org
[https://example.org]
[https://example.org example.org]

https://example.org
[https://example.org]
[https://example.org example.org]

.cm-mw-extlink,
.cm-mw-free-extlink
external link target www.example.org
.cm-mw-extlink-protocol,
.cm-mw-free-extlink-protocol
protocol https://
.cm-mw-extlink-text link label (no additional formatting)
.cm-mw-extlink-bracket opening/closing brackets [ ]
Tables
example

{| class="wikitable"
|-
! Header text !! Header text
|-
| Example || Example
|}

{| class="wikitable"
|-
! Header text !! Header text 
|-
| Example || Example 
|}

{| class="wikitable"
|-
! Header text !! Header text 
|-
| Example || Example 
|}

{| class="wikitable"
|-
! Header text !! Header text
|-
| Example || Example
|}

.cm-mw-table-bracket opening/closing brackets and pipe {| |}
.cm-mw-table-delimiter row and column separators | |- !
.cm-mw-table-definition table classes and styles class="wikitable"
.cm-mw-table-caption header cell text !caption
Non-CodeMirror
magic words (phab:T170041) __TOC__ __TOC__
semi-colon (d.term) ; foo bar
(treated as extension tag) math <math>\sum_{i=0}^\infty 2^{-i}</math> <math>\sum_{i=0}^\infty 2^{-i}</math> <tag>foo bar</tag> <tag>foo bar</tag>
(treated as extension tag) syntaxhighlight (former score) <syntaxhighlight>...</syntaxhighlight> <syntaxhighlight>...</syntaxhighlight> <tag>foo bar</tag> <tag>foo bar</tag>
(treated as internal link) category [[category:foo bar]] [[category:foo bar]] [[category:foo bar]] [[category:foo bar]]
file preview (file preview at the right-hand edge)
character disambig
(nbsp thinsp ndash mdash minus etc.)
WikEd character disambiguation.png  (screenshot) -