Talk:General User Survey/Questionnaire

How to ensure the survey to be representative? If you just let everyone fill in as he or she wants it won't be. What's "the average wikimedia project contributer" we want to measure? -- Nichtich 07:18, 18 July 2005 (UTC)

My only suggestion to this is to advertise it widely and hope that participation will be high from across all projects. This tends to give greater balance - though it can always be slanted in the direction of, say, experienced contributors, for example. The only way that I can see to avoid this latter possibility is to actively invite newcomers. Cormaggio 10:03, 18 July 2005 (UTC)

For each person the first time they do an edit just pop it up. This gets a representative sample across people (assuming a not very large self selection bias which can be ignored if response rates are high). This allows different surveys to be run on different samples while assuring that they all have statistical validity.

This requires close integration with Mediawiki code, which I think is a bridge too far for now. The survey will need to prove its usefulness. Maybe in a later stage follow up surveys could use that approach. Erik Zachte 20:52, 13 September 2006 (UTC)

Dispute resolution / conflict

I think it would be good to probe a little further into this area - should it go into the community section or a new section of its own? Questions I think might be useful include:

• I have experienced a conflict in the last year Y/N
• I have found the process to resolve this conflict to be v. straightforward -- v. complicated
• (Compared to last year, this was much better -- much worse)
• I have found others to be v. helpful -- v. unhelpful in resolving this/these conflict(s).
• (Compared to last year, this was much better -- much worse)
• I am v. satisfied -- v. unsatisfied with the outcome of the conflict

We should run this by the Arbitration and Mediation Committees if we're going to go with this. Cormaggio 09:50, 18 July 2005 (UTC)

I'm just including them for now - edit mercilessly and all that.. Cormaggio 16:41, 17 August 2005 (UTC)

Usability questions

I have a problem with this question: "I would favour more editing power and flexibility even at the cost of a more extended syntax."

I think that more experienced than inexperienced users will take part in the survey. They might tend to think that they want to have more power at the cost of less newbie-friendly features, which is quite natural.

However, the outcome of such a survey could influence MediaWiki's development by giving the developers the illusion that the users (all users, even beginners and future users) want more features at the cost of usability.

I think that MediaWiki should become more user-friendly, and finding out that power-users want power-features will not help us on the way.

Best regards, --zeno 14:49, 20 July 2005 (UTC)

The survey also tries to establish the level of experience of the person filling in the questions. So answers can be correlated to that. Erik Zachte 12:07, July 21, 2005 (UTC)

Ordering

Getting down to details on the survey. I would suggest moving the demographic questions to the end. That is the preferred way of asking for this data. People will often balk at the idea of providing their personal information. The idea is that you ask questions they are more likely to respond to at the first, and save the demographics for last. K1v1n 12:12, 15 August 2005 (UTC)

Done, also for qustions about amount of involvement Erik Zachte 15:24, August 15, 2005 (UTC)

My comments

1. Technical note: we are using w:Likert scale. That's a good scale :)
2. I'd suggest differentiating between users (read-only) and editors, and rename all instances of 'user' in this survey to 'editor', up to and including the title

- Good point Erik Zachte 21:05, 9 August 2006 (UTC)

1. "I rate the wikimedia editing process:" easy is subjective. I'd suggest adding a question like 'It took me about _ hours to learn basic editing in wikimedia'

- "Basic editing" is somewhat vague as well, as there is so much to learn, when does basic editing stop and advanced editing start? Rating UI easy or difficult is subjective indeed, but it tells us something important: for a task was easy when it took me much less time to complete it than I was willing to spend, in other words than the benefit I gained. So I might rate opening a can difficult if it took me three minutes and learning a language easy if it took me half a year. Erik Zachte 21:05, 9 August 2006 (UTC)

1. I see little point in "I would favour more editing power and flexibility even at the cost of a more extended syntax": as users don't have to learn the syntax to keep editing in their current levels, who would object to it?

- More complex syntax makes editing more daunting for existing and new editors alike. For example the con of metatagging, which is very powerful, is more markup yet and less oversight, until we have wysiwig editor. Erik Zachte 21:05, 9 August 2006 (UTC)

1. "I rate the wikimedia facilities for navigation and information gathering:" - what is 'information gathering'? Navigation is enough, don't force the users two think about twho concepts, this makes it potentially 'double-barrelled' (some users will differentiate between navigation and information gathering, other's wont, and some will answer only half of the question).

Concur, Erik Zachte 21:05, 9 August 2006 (UTC)

1. "In general I rate the Wikimedia community:" - lose 'in general'. KISS.

Concur, Erik Zachte 21:05, 9 August 2006 (UTC)

1. There are some questions that can be added to the community section. I'd especially recommend the following one:
   1. 'Are you interested in Wikipedia community, like a regional noticeboard or Esperanza?' - it would be interesting how many users care more for content creation then interacting with others
2. The 'Project awareness' section is about MediaWiki software, yes? I'd suggest duplicating this section twice, and asking the same question about Wikimedia Fundation and Wikipedia
3. 'In general I rate the content of Wikimedia projects that I am involved in:' is confusing: are we talking about 'encyclopedic content' or the 'performance of the projects, like WikiProjects'? This needs to be clarified (I was confused by this question!). And indicate you are asking about trust and completness in the question, not the answer, having two same questions is confusing.
4. 'I have experienced a conflict in the last year' - add 'on Wikipedia'.
5. 'Involvement' - consider asking about number of hours per day (or week) user spends on Wikimedia project
6. 'Demographic data' - consider adding: 'Do you have any position which grants you special editable rights, like an administrator, bureaucrat, steward or developers?'
7. Regarding education levels, I suggest asking about highest level of education completed: primary school, secondary school, BA, MA, PhD, with an additional question if the user is still studying. Those levels are mostly universal.
8. Finally, before the survey is launched, consider adding an annoucement that users can submit their questions. We can parse over them and see if there are any interesting that could be asked in the appendix section.--Piotrus 16:34, 7 July 2006 (UTC)

One more thing comes to mind. Is survey is supposed to be anonimous? Should we give the users the option to not be anonimous?--Piotrus 16:09, 13 July 2006 (UTC)

Subst vs advanced html

Since finding an html programmer willing to help us is likely one of our bottlenecks, I rewrote the survey to be usable with substitution template, as described on main page. There are still some issues to resolve with some specific questions (as noted above) and formatting of answers (in text). I added the ANSWER=XY to each answer which should make it easier to design a bot which will farm the answers to present statistics.--Piotrus 02:05, 15 July 2006 (UTC)

Currently the subst version can be seen here, hopefully we will go with the advanced html version.--Piotrus 15:42, 10 August 2006 (UTC)

Simple version of questionaire

As much as I value Piotrus' comments and suggestions above (I'll comment more on those), I feel the proposal for a simple template based questionaire is impractial. Validation and encoding of input (especially important to sync multilanguage editions) would be hardly practical or impossible. Automated processing of the results would be a disaster. I replaced the proposal by a link to the final revision that contained it. Erik Zachte 21:22, 9 August 2006 (UTC)

I agree with most of your points; my 'subst version' was one of desperation - something to push forward if nobody with technical skills would help. Hopefully this has changed during Wikimania :) As for the encoding of input, I don't think it would be a problem - I think it would be easy to design a bot which would go over answrs and see that for 'Q7' the answer value is '8'. But as editing in a wiki mode is not as friendly as filling an html questionnaire, I am all for a new better version (perhaps some of the work done on it can be shared with the WYSIWYG editor project - just a thought).--Piotrus 15:38, 10 August 2006 (UTC)

Linkage with User Data

Might the survey invite respondents to share their usernames, so that answers could be connected with actual (logged-in) editing behavior from the history dumps? Jeremy Tobacman 22:18, 5 November 2006 (UTC)