Grants talk:Programs/Wikimedia Community Fund/Research on effectiveness of Wikipedia in Education as a platform of improving the cognitive ability among students

Please write your comment, opinion and suggestions, if any, here.-Pavanaja (talk) 05:15, 2 December 2021 (UTC)

Feedback by the SAARC Regional Grants Committee

Thank you for the proposal. Upon initial review the committee has asked for further clarification on the following topics. Please note that a member of the Community Resources Team is posting this feedback on behalf of the SAARC Regional Grants Committee. The committee would be happy to meet with the applicant/s if they feel that a discussion will be more fruitful in responding to the feedback. Summary of these discussions will be posted on meta upon the consent of the committee members and applicant/s.

Research methodology and output: What kinds of tools will be used while measuring this work? As this project is related to educational purposes one psychologist and one education expert should be involved in the project to strengthen the learning capacity of students. The project proposal (in its present stage) focuses primarily on the training of the students, with little or no emphasis on the research aspects of the cognitive abilities. No description/abstract is provided as to which aspects of cognitive ability would be tested and what would be the methodology for this. Two papers have been listed in the proposal which are generally relevant to the topic, but provide no guidance to the aspects that will shape the research questions for the project.

The proposal is meant for research into educational and psychological aspects of the student editors as they learn, but there is no background information about the method, what thesis is to be tested, experimental/research design and very little detail of the research aspect. There is a need for professional academic input from the fields of cognitive psychology and educational pedagogy for the success of this project. Such a partnership or learning mechanism has not been presented in the proposal.

Engagement and Sustainability: There is no commitment from the Karavali User Group agreeing that they are part of the programme. The proposal also mentions that the project would be conducted at Dr G Shankar Government Women’s First Grade College & PG Study Centre, Ajjarkadu, Udupi, Karnataka, India. The letter of agreement between the College and Dr Pavanaja needs to be provided for information of all, as a supporting document for the grant application, especially as the College will have to make substantial commitments, such as:

• Allowing the 12 workshops to be conducted on their premises
• To permit a break in College studies for two days of the full-day workshops.
• To provide internet connections, and desktop/laptop access to the 25 students for the complete two days of the 12 visits.

Additional point to be considered:

• Faculty members of the participating institute do not appear to be co-opted in any manner, nor do they benefit from the training, and their involvement is zero, which is an indicator against sustainability.

Metrics and Evaluation: The proposal is planned for only 25 students. Considering the CoVID situation, there is a chance of dropout from the course/this proposal for various reasons, hence this number is likely to reduce. The impact of this large scale, multi-lakh rupee project, will therefore be small. More significantly, the sample size for this project is statistically insufficient, suggesting the need of a professional input of a statistician. In comparison, the study quoted in reference 1 [Meseguer-Artola et al. (2020)] has a sample size of 2,330 university students. There is a need for a greatly increased sample size.

The proposed research study is planning to conduct only one test at the beginning and one test at the end. The change in cognitive ability over the period of engagement cannot be determined by just two tests. The requirement is for continuous evaluation on different cognitive aspects throughout the period of engagement. What is the guarantee that the change in cognitive ability is only due to the wiki editing alone? The tests can be successful only when a similar set of people are identified for tests who do not participate in the wikipedia editing activities.

Resources and Budget: Project is over budgeted — Rs 20+ lakhs to teach editing and Wiki skills for 25 students and analyse their progress. Volunteer tasks should be maximised. Involvement of the user group should be maximised to reduce hiring costs. People should be hired for very specific tasks that cannot be done by volunteers, e.g. programmers, developers. For reference: Tamil wiki community has conducted “Tamil wiki internship programme” for two years in 2020 and 2021 for one month long with an almost daily programme (https://ta.wikipedia.org/s/9aqq). Each time more than 30 students have completed the programme successfully with the targets such as 20 Wikisource proofread pages, 100 Wikidata edits, 5 photo uploads, 5 new words addition in wiktionary, 5 article creation in wikipedia. After the month-long event the students who have completed were given certificates. The event was virtual. All the organisers and resource persons done the programme voluntarily with zero cost except the cost for certificate which was issued by CIS.

To inculcate the spirit of volunteerism the principal researcher may be reimbursed for travel, miscellaneous expenses, child care, food, etc when visiting the school as per current standards. If the project is so large that a full-time employee needs to be hired, the grant request should be moved to a movement partner, an ad hoc one-time, short-period, grant application is not a suitable model for such an engagement.

Lot of equipment is proposed to be purchased — a high end laptop for video editing (Materials: serial 2: laptop for video-editing: Rs 75,000 ), however, funds for commercial editing (remunerations: serial 6: Videography, Rs 125,000/-) are being asked. This is contradictory. The project should ask for either one of these. Budget item for a smartphone can also be reconsidered to favour a camera. All of these items should be seen as community assets and not as individual equipment.

On behalf of the SAARC Regional Grants Committee THasan (WMF) (talk) 15:02, 23 December 2021 (UTC)

Answers to Feedback by the SAARC Regional Grants Committee

Thanks for the feedback and queries. Here are my answers.

• The proposal was created using the Fluxx platform. For every textbox wherein we have to enter the answer to that particular question, there is character limit. Hence a lot of details could not be entered. I have created a little more detailed project proposal and uploaded it here. Please go through it.
• Research methodology and output: Details are given in the Google doc, link given above. A suitable resource person will be involved in designing and creating the tests. The remunerations towards this are covered in the budget. I was General Manager, Academics, at Manipal Global Education during 2007-09. I was Head, Learning and Development at Excelsoft, Mysore, during 2009-11. I have the experience of designing and administering tests using conventional methods as well as online methods. I am quite well versed with all aspects of this. Hence, I am confident in this area.
• Engagement and Sustainability: All the active members and office bearers of Karavali Wikimedians User Group have already endorsed the grant proposal. This grant proposal has been discussed internally during our telephonic conference call meetings. Ramachandra Adiga Gudimane, Head & Associate Professor, Department of Computer Science, Dr G Shankar GWFGC & PGS Centre, Ajjarkadu, Udupi, has also endorsed the grant proposal. He will be the key contact person from the college. He will be helping with all needed support from the college. Earlier he has helped in organising many editathons at that college. Once the grant is approved, there will be a formal MoU with the college before starting the program. There will be no break for students from their regular classes as the programs will be conducted on weekends as a certificate course. College is already conducting many other certificate courses on weekends.
• Metrics and Evaluation: Possibilities of someone dropping out of the program is very remote since this will be run as a certificate course. Nevertheless, we may take up to 30 students instead of 25 to account for any unforeseen dropout situation. The college has a PC lab with about 30 working PCs only. Hence the number of students is limited to that. It is also not possible to train a large number of participants in single batch. I believe this number will definitely give sufficiently reasonable research outcome. Even though it is mentioned about two tests, one at the beginning and one at the end, there will be smaller tests during the course. Study using a control group also may be thought of.
• Resources and Budget: This is a different kind of activity and can not be compared with “Tamil wiki internship programme” which is altogether different. Community will be involved wherever they can. It is mentioned in the grant proposal. The work to be done by the principal researcher is not just conducting 12 workshops. It involves much more. The entire certificate course has to be designed and administered. There will be regular interactions with the students apart from during the training programs during weekends. These interactions will happen through online, emails, Whatsapp group interactions and online meetings. They will also include checking of the students’ assignments, giving feedback, correcting their works where needed, giving more assignments, helping them wherever needed. If available, a LMS also will be used (Govt of Karnataka provides LMS to colleges. Not sure whether we will be given access to use that). Hence this is a full-time involvement for the principal researcher. I have removed the laptop from the budget. Smartphone requested has many functions, viz., for recording interviews, for photowalks, recording some sessions, and also for online meetings and sessions.-Pavanaja (talk) 08:58, 5 January 2022 (UTC)

Final recommendation by the SAARC Regional Grants Committee

Dear Dr. @Pavanaja:,

Thank you for your grant proposal and the clarifications you have provided us in response to feedback on it. After consideration of all of these, we would like to offer you a week’s time (from the publishing of this recommendation) to revise the proposal and provide an improved plan per the SAARC Regional Grants Committee recommendations. (The funding recommendation for the grant as it currently stands would otherwise be partial.) We believe that taking the time to revise will not only increase the robustness of the undertaken research in the face of the unforeseen, but may also placate concerns we raised such that we may ultimately recommend more than just partial funding.

Some thoughts to keep in mind for revisions are as follows:

• First, due to the resurgent pandemic situation, an altered proportion of in-person activities to virtual activities should be considered.
• Equally important are information about the cognitive tests themselves, which we believe should be elaborated on as far as their specific content goes. The testing methodology needs to be more continuous and progressive and test the hypothesis of the research with adequate academic rigour. Harmonizing the content of intermediate tests with those of the initial/final tests and possibly adjusting their frequency may be important in the absence otherwise of a sample size rivaling those of the resources you have cited.
• Related to this is the ambiguity regarding the venue for publishing this research’s output, which should be resolved.
• The revised project plan should provide details of collaborators and their roles in the project (a tentative list of names with designations is fine).

The information requested will help provide methodological clarity for the proposed research.

As a note, we are happy to recommend in any case that disruptions in activities due to college closures or other unforeseen circumstances not be counted for the purposes of the grant timeline.

For the Committee, Mahir256 (talk) 06:18, 17 January 2022 (UTC)

Answers to Final recommendation by the SAARC Regional Grants Committee

• Regarding virtual activities should be considered: The program will be conducted in a Women’s govt college. The students are from rural background mostly coming from lesser privileged strata. Many don’t have the robust infrastructure like laptop/PC, broadband Internet connection to participate in virtual classes. During the last pandemic, the college faculty tried their level best to conduct classes online but majority students could not attend due to lack of good connectivity. Then they tried sending the notes by Whatsapp. In that also some could not download huge files, mainly video files. Hence conducting the program fully online is not possible. Some follow-up activities through online are planned. This will be only complementary to the in-person programs. The ratio of online to in-person programs may be about 30:70.
• Regarding the venue for publishing this research’s output: Some of the SAARC Regional Grants committee members wanted to know where the output of the research will be published. They also wanted a confirmation that it should be an open access journal. I had not yet given a thought where to publish the output so much in advance. I did little research now and found out some open access research journals where this research output may be published. They are - International Journal of Cognitive Research in Science, Engineering and Education (IJCRSEE), Journal of Research in Innovative Teaching & Learning, International Journal of Educational Technology in Higher Education. The research methodology will be fine-tuned to match the structure of the journals. If anyone has any other suggestions, please feel free to provide.
• Collaborators and roles: Already one major collaborator’s name is provided in the earlier answer. Please scroll above. I have discussed with one expert on education psychology. He has agreed to be a collaborator. He is Dr.T.N. Lokesh, Professor at Department of Education, NMKRV College for Women, Jayanagar, Bengaluru. He has acquired M.Sc., M.Ed., M.Phil., P.G.DHE., UGC-SET. Ph.D. (Educational psychology), Ph.D. (Science Education) from reputed universities across India. He has more than two and half decades of teaching experience in the field of Teacher Education. He is teaching subjects like Educational Psychology, Educational Philosophy at B.Ed and M.Ed level. As a research guide, he guided many M.Ed students of Karnataka State Open University to complete their dissertation. He is a resource person in several educational projects of DSERT under the government of Karnataka. To his credit, he completed several educational research projects funded by DSERT, UGC and ICSSR. Identified as a resource person in Human Rights Education by National and State Human Rights Commission and SICHREM, Bengaluru. Being a good translator, he translated the D.Ed science source book prescribed by NCERT through TISS, Mumbai and DSERT, Govt.of Karnataka. He presented various thematic and research papers at national and international seminars, conferences and workshops. Invited by the Ministry of Bangladesh to deliver a talk on “EMPHASIS ON CITIZEN SAFETY AND MEASURES DURING COVID-19” on 15th June 2020 and represented India at World Leadership Conference. A recipient of the National Youth Educationist Award for the year 2020.
• Regarding cognitive tests: A lot about cognitive skills and tests have been already given in the grant application as well as in the bit more detailed project proposal in the Google doc. Learning objectives have been defined based on Bloom's taxonomy and its derivatives. Bloom's taxonomy is a set of three hierarchical models used to classify educational learning objectives into levels of complexity and specificity. The three lists cover the learning objectives in cognitive, affective and sensory domains. The cognitive domain list has been the primary focus of most traditional education and is frequently used to structure curriculum learning objectives, assessments and activities (Wikipedia). The cognitive domain is broken into the six levels of objectives listed below-

Bloom's Taxonomy

Bloom’s taxonomy provides the basis for assessing the outcome of learning programs. The cognitive domain consists of levels to be evaluated. Cognitive domain deals with how a student acquires processes and ulitises the knowledge. Cognitive domain is the core of the learning domain. Cognitive domain is well-suited for computer-based assessment or in other words the frequently used multiple-choice answers type of examinations. The levels in cognitive domains can be measured by making the students involved in -classrooms discussions, tutorials, demonstrations, participations, quizzes (multiple choice questions), projects, assignments, problem-based learning, seminars, viva-voce, etc. All these need not be used in the present scenario or all may not fit very well. But nevertheless, the evaluations will follow this framework. Questions will be designed involving the different levels. A template will be devised to set the questions. Based on the values, the achievements of cognitive levels of each student can be measured. The improvement in the course can be done based on the feedback. This can be done in a continuous evaluation model. Hence the entire process is not just - test in the beginning and test again at the end - kind of model.-Pavanaja (talk) 05:15, 24 January 2022 (UTC)

Funding recommendation by the SAARC Regional Grants Committee

Dear Dr. Pavanaja,

We are happy to recommend that your grant be approved with partial funding (but see below), 11,552 USD (8,66,034 INR) for the calendar year having begun on 1 January 2022 and ending 31 December 2022.

The activities you have described are sure to provide a lasting impact for the Kannada-, Tulu-, and Konkani-language projects, and your prior expertise in research and development within the education space is sure to yield impactful insights for future growth experiments in other communities. We truly appreciate the commitment and sincerity with which you as a long-time Wikimedian have designed, and approached us with, your proposal.

These being said, however, we recommend that in the interest of making the most effective use of these funds, you perform the following actions:

• You should further elaborate for us aspects of the methodology which may not have been described completely or sufficiently before—be these the specific types of questions you plan to supply for evaluations, the specific connections between these questions and the proposed activities, or indeed any other aspects as they directly pertain to the students and those activities. While we are grateful for the contextual information you have provided in responses to our feedback, we believe that there is still some clarity that could be achieved with respect to their description and their relevance to the specific activities the students will take part in, especially given our perception of such admissions to this effect in some of your responses. We continue to trust your capabilities in providing for us full closure in this regard.
• In about six months time from this recommendation, late July or early August, you should submit a report detailing the progress that has been made with respect to the activities funded by this partial grant, in terms of the metrics you have elaborated for us previously and (qualitatively) other direct tangible and intangible impacts of these activities on the students and their aforementioned-language project communities. We recognize that there may be challenges in providing a completely faithful assessment of your research at this stage in the process, but we believe that the indicators provided in this report will help describe the net effects of these activities and hint at future outcomes from their continuation. If there are clear indications of progress, both quantitative and qualitative, throughout this report, we will recommend that the remaining 17,327 USD (12,99,049 INR) be provided at that time.

We want to emphasize that we would like to see the success of this research, and that this alone is the rationale for the conditions stipulated above.

We again thank you for your proposal, and we wish the best this year for you, the students who will take part in its activities, and the other facilitators of those activities.

On behalf of the SAARC Committee, THasan (WMF) (talk) 11:31, 31 January 2022 (UTC)

Note on feedback and review

Thanks to the SAARC Regional Committee for the decision. In conjunction with this decision, I wanted to remind all participants here that the Community Resources team has specific expectations regarding discussion about proposals in this space.

In the Community Resources team's behavioral expectations for proposals and grant discussions, anyone with concerns about a proposal is welcome to express them in a constructive and supportive manner. However, to the extent that feedback is excessive, contains personalized or disparaging remarks about the applicant or their organization, or if the concerns are expressed in an hostile or punitive manner, they may be removed from the discussion page partially or entirely. The goal of discussion is to build shared understanding and work together with applicants to improve their proposal. It is not to engage in discussion that is confrontational or aggressive, even when there are genuine concerns. Relatedly, participants should follow the Universal Code of Conduct, which contains the minimum level standards for communications and behavior on Wikimedia projects generally. With thanks, THasan (WMF) (talk) 11:00, 14 March 2022 (UTC)