Community Wishlist Survey 2022/Better diff handling of paragraph splits/Test strategy
Project documentation
[edit]- Project page: Community_Wishlist_Survey_2022/Better_diff_handling_of_paragraph_splits
- Phabricator tag: https://phabricator.wikimedia.org/tag/better-diffs/
Feature documentation
[edit]- General help: mw:Help:Diff and Help:Diff
- Wikidiff2, which is used on production and beta wikis: mw:Wikidiff2
Test documentation
[edit]Where to test it
[edit]This needs to be figured out. How will we test the latest version of wikidiff2 and compare its output to previous versions?
How to install locally
[edit]By default, local wikis will use an "internal" diff engine.
In order to test wikidiff2 locally, you have to install it on your server. Doc to be written.
Coverage criteria and factors
[edit]- Diff engines
- We may want to test the "internal" PHP diff engine in case UI changes cause problems
- Data
- Different types of wikitext (e.g. tables)
- Unicode, zero-width, non-printing
- Different content models
- JSON, Javascript, CSS, etc.
- Different languages
- Especially languages which have word tokenisation/segmentation different to English, such as Thai
- Different types of diffs and places where they appear
- See /Types of diffs
- Different combinations of Wikidiff2 and VisualEditor
- Wikidiff2 installed but not VE
- VE installed but not Wikidiff2
- Neither installed
- No JavaScript?
- Actions on diffs
Oracles
[edit]- Previous version of wikidiff2
- Other diff engines
- Hand-crafted "gold standards"
- UX mockups
- Self-verifying data?[1]
- Metamorphic oracles?
- "Explainability", "Desirability" and "Purpose" oracles[2]
Risks
[edit]- Vandals can craft their edits in order to avoid detection
Quality criteria
[edit]- Usability/understandability
- Accessibility
- Performance
- Other than the intended changes, diffs behave the same as before
Techniques
[edit]Nouns and Verbs
[edit]Generate sentences combining the nouns, verbs, adverbs and adjectives below to generate ideas for tests (please feel free to expand this list):[3]
- Verbs
- Add
- Remove
- Replace
- Modify
- Move
- Split
- Combine
- Rearrange
- Copy
- Nouns
- Characters
- Words
- Sentences
- Paragraphs
- Newline
- Whitespace
- Adverbs
- Repeat(edly)
- Sequentially
- In parallel
- From one <noun> to another
- Adjectives
- Large
- Small
- Multiple
- Similar
- Extra
For example, "move sentences from one paragraph to another" gave me the idea for this test case.
Or "combine multiple paragraphs" for this.
Test data
[edit]See Community_Wishlist_Survey_2022/Better_diff_handling_of_paragraph_splits/Test_cases.
and w:User:Cacycle/diff_test_cases.
Further examples could be generated:
- Taken from real-life examples (e.g. from production wikis), see #Random
- Hand-crafted (see #Techniques)
- Created programmatically and/or at random
Test tools and scripts
[edit]See https://gitlab.wikimedia.org/dwalden/betterdiffs-testing
Random
[edit]- Script which finds random diffs from a wiki, runs them through the current MediaWiki diff and the wikidiff2 demo and compares outputs: https://gitlab.wikimedia.org/dwalden/betterdiffs-testing/-/blob/main/random_diffs.py
Selected
[edit]- Script which takes diffs you have selected, runs them through the current MediaWiki diff and the wikidiff2 demo and compares outputs: https://gitlab.wikimedia.org/dwalden/betterdiffs-testing/-/blob/main/all_all_diffs.py
Logs and debugging
[edit]Current testing status
[edit]Test strategy
[edit]Testing done
[edit]Testing still to do
[edit]Testing which will not be done
[edit]How-to guides
[edit]Other documentation
[edit]Notes
[edit]- ↑ https://bbst.courses/materials/BBST%c2%aeFoundations2022.pdf pp. 117
- ↑ https://developsense.com/blog/2012/07/few-hiccupps
- ↑ This technique is taken from Hendrickson, Elisabeth (2013). Explore It! Reduce Risk and Increase Confidence with Exploratory Testing. The Pragmatic Bookshelf. Chapter 6.