Brazil Program/Education program/Learning/Brazil Pilot 1/Meeting goals

From Meta, a Wikimedia project coordination wiki

We had various Factors outlined for success of the program.

Overall notes[edit]

The two categories of goals we did not meet were:

  1. Impact on Portuguese Wikipedia, and
  2. The road ahead

For some of that, we recognize that we made intentional concessions during the course of the semester which resulted in lower editing counts. We decided to prioritize Goals 3 & 4: positive community reception and constructive classroom experiences. We succeeded in this!

That said, we can look at the categories we did less positively on and as we ponder why, decide (a) whether the goal itself was wrong, or (b) reconsider how we should approach the goal next semester.

Impact on Portuguese Wikipedia[edit]

Given the history of the Brazilian WP-PT communities, we wanted to take time to socialize the program and let people feel like they were being heard and trying things for themselves. As a result, we didn't push quite so hard or try to professionalize or over-hire: we wanted to develop ownership and buy-in, so that it wasn't just the "big-bad WMF" coming in to control. We have found in past work with Brazil that this is essential. Also, given the WP-PT community, we wanted to make sure that they have time to get acquainted with the program since any rejection of it on their part would result in extremely frustrated students and faculty. All these things are an ongoing process, and we will need to continue to work on them going forward. This past semester, though, we saw great success in these areas, as there has been positive reception from all Wikipedian/Wikimedian participants.

That said, it is clear that there is more room for being bold in pushing for editing on Wikipedia, and in order to make a bigger impact on the Portuguese Wikipedia, we need to provide more support in making this editing happen. Some thing we can do to help with this next semester:

  • More carefully working with professors in incorporating editing milestones into lesson plans
  • Requiring editing of Wikipedia in our supported courses
  • Increasing number of ambassadors involved
  • Providing more training to ambassadors to help them be more hands-on in helping students and incorporating content into the main namespace
  • (Potential) Hiring more support (e.g., fellowships, additional BR-WMF contractors)

The Road Ahead[edit]

It is really difficult to prioritize documentation when there is a lot of physical work that needs to get done. That said, clear documentation helps the team in San Francisco track with the team in Brazil and it also helps the community to understand on which projects the staff is spending time and resources. Some suggested ways to help support this:

  • Write all reports in Portuguese
  • Solicit volunteers to write blog posts throughout the semester (one from a student, one from a professor, one from an ambassador) to take the load off of the project coordinator

Goal achievement: 1st semester 2012[edit]

Key[edit]

Fully met the original goal Partially met goal / met modified goal Missed goal (reevaluate!) Not enough information to assess

Table[edit]

Areas of impact Goal Set, 1-2012 Actual result, 1-2012 Goal achievement
1. Impact on Portuguese Wikipedia At least 75% of the students participating in the pilot add 500 words or more to the Wikipedia article namespace Exact numbers TBD, but will not meet this goal. This is partially due to the class structure (lots of group editing) and the lack of direct editing to the main namespace
At least 75% of the content contributed by students "survives" (i.e., is not deleted from the Wikipedia namespace). 100% of the articles moved from sandboxes to the mainspace survived, but 67% of the articles with contributions (20 articles out of 30) were moved to the mainspace. All courses were oriented to start working on sandboxes and then move to the mainspace, but some professors have decided to work offline and then at the end of the course to start moving to Wikipedia.
[1]
At least 50 students in the Education Program part of the pilot start editing TBD.[2] This number, though may be hard to measure based on students contributions: we had a lot of students who worked on group projects and therefore did not edit directly (rather, contributed offline) TBD
At least 10 people being recruited through the general university outreach part of the pilot start editing Chose not to pursue alternate outreach given resource constraints.[3]
2. Structures for success The Brazil Pilot starts on-time (March 2012). Started!
Training sessions are conducted by volunteers for the professors, campus ambassadors, and online ambassadors Held training sessions, but they were poor. Did not host a comprehensive one for ambassadors, which was needed.[4]
Surveys are created and distributed to professors and students at the beginning and end of the semester Created and distributed[5]
Feedback from training session is gathered via a post-training survey Created and distributed[6]
Clear documentation is created showing different options for assignment (e.g., editing existing articles, creating new articles, translating articles) Created ultimately by the broader education team, and localized to Portuguese; it was not available for the professors at the beginning of this semester.[7]
Clear documentation is created showing different options for Ambassador models, and pros/cons Not documented: this is something we should do to help ambassadors understand where to plug in better.
We will have numbers that clearly show which model for in-classroom activities is more successful (in terms of both amount and quality contributed) TBD - we know that we have to have professors that are engaged in milestone projects, and we also saw that a dedicated professor makes the biggest difference. We are still waiting for final numbers, but they will not be best situated to give us the full story of what went on behind the scenes, given the number of students who edited offline.
3. Community impact and reaction Members of the Portuguese Wikipedia community are supportive of the pilot and volunteer to participate in supporting it, both online and offline. A few community members engaged in the discussion and strategy, but many more engaged in the training materials and support pages on the PT.WP.
We start general outreach activities on campus in the first semester in 2012 We de-prioritized this (see above) given resource constraints, but we did see one event occur on campus by CAs, and also we spoke at other campuses to spread the world
We fill all volunteer positions; 50% of the volunteers are Wikipedians Yes!More than 50% of the ambassadors that ended up participating in the project were already Wikipedians
More than 75% of the community volunteers involved in the pilot indicate that it was worth their time Didn't ask this question directly, but ~67% had "Good" or "Very good" experiences, and 100% want to help next semester[8]
Members of the Portuguese Wikipedia indicate support for continuing to build an education program in Brazi All those volunteers involved with the project agree it was and is useful, though many indicate needs for improvement if it will be successful long-term.[9]
4. Professor and student reaction We fill all slots for professors in 5 classes We worked with 5 professors in 6 courses this semester, though a few did not complete Wikipedia assignments[10]
Two (2) prominent universities participate in the program We worked with three of the top universities in the country: USP, UNESP, and UFRJ[11]
100% increase in the level of comfort in editing After orientation, all the professors indicated increases in their comfort on Wikipedia. For students, they moved from being mostly uncomfortable editing to planning to edit more in the future.[12]
75% of the participants indicate happiness with the outcome. Yes! 100% of students were "Very Likely," "Likely" or "Somewhat Likely" to recommend the program to a friend, and 93% of students "Agree" or "Strongly Agree" that the program was a beneficial experience.[13]
More than 50% of the students indicate that they would prefer a Wikipedia assignment type course to a traditional course in the future. 50% of students "Agree" or "Strongly Agree" that they would prefer a Wikipedia assignment toa term paper assignment.[14]
More than 50% of the teachers indicate that they would use Wikipedia as a teaching tool again 100% of the teachers surveyed indicated they would want to participate in the program again. Two, though, indicated they would want a less restrictive version of the program[15]
More than 50% of the teachers indicate that they would recommend using Wikipedia as a teaching tool to their colleagues/peers 100% of the professors surveyed indicated they would recommend this to friends or colleagues[16]
5. The road ahead Bi-weekly updates on the program are shared with the PT:WP community; we publish at least 3 interim reviews with stories of success and lessons learned on the WMF blog We were able to publish 3 blog posts.[17] We were not, though, as diligent in publishing comprehensive reports on where we were throughout the program.[18], and we got some feedback about not being quite transparent enough in the activities.
An outcome / lessons learnt / the road ahead document will be completed by July 2012. Late, but being worked on it![19]
We will identify which in-classroom activities are most successful (in terms of both amount and quality contributed) Should be incorporated into the above report

Notes[edit]

  1. Met goal for those articles that are currently on the mainspace, but not all articles have been moved onto the mainspace yet. Goal needs some clarification.
  2. See Category for potential way to estimate; also coud use the list on meta.
  3. Goal was originally adapted from the Cairo Pilot plan, which was able to incorporate outreach through the hiring of another individual
  4. See post-semester Ambassador feedback
  5. See the folder of feedback surveys
  6. See the folder of feedback surveys; for results see Brazil Program/Education program/Learning/Brazil Pilot 1
  7. Full case studies brochure on Outreach
  8. See Ambassador feedback
  9. See Ambassador feedback
  10. List of courses
  11. List of courses
  12. Note: we did not ask directly for students to compare their editing comfort following the course. We need to add this question for next semester. See post-semester survey results
  13. See post-semester student feedback
  14. See results from the courses
  15. See post-semester professor feedback
  16. See post-semester professor feedback
  17. See communications page for blog posts and external articles
  18. See Pilot Program reports
  19. See Everton's draft, which will encompass the survey analysis work