- 1 Change status to 'proposed' to submit for 8/2/16 Project Grants deadline
- 2 Staff pay
- 3 Relationship between grantee and ContentMine
- 4 Other funding and in-kind contributions
- 5 ContentMine and commercial interests
- 6 Precedent from similar projects
- 7 Usefulness for Medical Wikipedians
- 8 Relevant wikidata communities
- 9 Some thoughts
- 10 Primary sources tool?
- 11 Some comments
- 12 Example Facts
- 13 How about Claims ?
- 14 Dictionaries
- 15 Eligibility confirmed, round 1 2016
- 16 Broken Link
- 17 WikiFactMine and StrepHit
- 18 Aggregated feedback from the committee for Project/WikiFactMine
- 19 Round 1 2016 decision
- 20 Outputs
Change status to 'proposed' to submit for 8/2/16 Project Grants deadline
Please note that if you intend to submit this proposal for the August 2 deadline of the current round of Project Grants, you must change the status from draft to proposed. If you have any questions, our final proposal clinic is from 1600-1700 UTC on August 2.
I note that in the budget section the proposed pay for the software developer is USD 40,000/year and the proposed pay for the Wikipedian is 38,000 for the year.
One report says that the average salary in Cambridge is GBP 46,000, or USD 61,000. The work proposed in this project seems technical. Can you explain why and how you plan to recruit a technical position in this city while offering much less than average pay for the area? Blue Rasberry (talk) 14:02, 2 August 2016 (UTC)
The average pay is across all sectors and various levels of seniority, we will be recruiting a junior to mid-level technical role in a non-profit and the salary offered is in line with technical roles at this level at the University of Cambridge and with our existing rates. Based on our experience with previous recruitment activities in Cambridge, we do not foresee problems in attracting suitable candidates. --Jcmolloy (talk) 23:14, 20 September 2016 (UTC)
Relationship between grantee and ContentMine
The grant is proposed to go to Peter. Can you clarify how much of the grant work will be managed by Peter as an individual, and how much will be a part of ContentMine? It seems like ContentMine will be involved in everything, correct? Blue Rasberry (talk) 14:10, 2 August 2016 (UTC)
- petermr has been funded by the Shuttleworth Foundation to start and run ContentMine - this pays for staff and expenses. petermr himself runs CM on a voluntary basis. The project would be managed by petermr and staff as appropriate. ContentMine will provide in-kind support for the WikiFactMine project throughout its funded lifetime. —The preceding unsigned comment was added by Petermr (talk) 17:01, 2 August 2016
Other funding and in-kind contributions
Can you describe what kind of support that ContentMine or the University of Cambridge might contribute to this project? State if other funding support is likely, and also mention any in-kind contributions. Blue Rasberry (talk) 14:15, 2 August 2016 (UTC)
- The servers on which the code will be run have been provided through university funds which petermr held. We are working with the other departments to see whether additional funds can be provided.—The preceding unsigned comment was added by Petermr (talk) 17:01, 2 August 2016
ContentMine and commercial interests
I see that ContentMine embraces open content and licensing, including CC-by licensing for text, CC0 data licensing, and that all the software uses MIT's open source license.
Is ContentMine a nonprofit organization, or a company? Either way, how likely is it that ContentMine could put any part of the output of this project into its own closed product? Blue Rasberry (talk) 14:10, 2 August 2016 (UTC)
- ContentMine is a UK company limited by guarantee with a specific OpenLock in its articles that prevents acquisition by predatory closed companies (e.g. commercial publishers). CM deliberately does not create closed products and all material is made available in Open repositories such as GitHub.—The preceding unsigned comment was added by Petermr (talk) 17:01, 2 August 2016
Precedent from similar projects
Briefly, and with links to further reading if possible, can you point to any project similar to WikiFactMine which has been done before? Briefly, can you say what WikiFactMine might do that has not been done before? Blue Rasberry (talk) 14:12, 2 August 2016 (UTC)
Usefulness for Medical Wikipedians
This "informing... Wikipedia editors of relevant citation-supported facts" is not really the problem we have. We are lacking people interested in slowing going through high quality review articles and other secondary sources and paraphrasing them in simple language.
We actively discourage the use of primary sources which appears to be what this project will be mostly working on. The example given is for the type of primary source that we as Wikipedians should never be using for anything medical. It is a study of yeast in cell culture and therefore has no current application to humans. The summary left out the most important word potential before "use of lemon essential oil-based products as natural remedies against candidiasis". This is bench top research of the earliest form.
- Thanks. These comments are useful. The example given had a more chemical/bioscience emphasis and wasn't meant to emphasize medicine. A more typical medical result would be to alert to mentions of clinical trisls - we are currently part of the Open Trials group.
- We are more interested in systematic reviews than single trials. While we are supportive of single trials being open and the work of OpenTrials and AllTrials generally single trials just are not good sources for WP. We want experts to review all the avaliable research and come to an overall conclusion. Doc James (talk · contribs · email) 23:08, 7 August 2016 (UTC)
Just to reinforce what Doc James has written above, Wikipedia prefers secondary sources, not only for medical content (see MEDRS), but for all content (see see SCIRS and secondary). Also while the example sources you have given are from en:Europe PubMed Central and hence not hidden behind a pay wall, it is more important the sources are secondary rather than open. Fortunately this is easy to do in the advanced search in PubMed by restricting the publication type to "review".
The stated aim of this project is To make Wikidata the primary resource for identifying objects in bioscience. As already stated by others elsewhere, the biggest challenge is not finding high quality sources (en:PubMed already does an excellent job of this), but paraphrasing these sources to produce Wikipedia content. Boghog (talk) 06:24, 27 August 2016 (UTC)
This is exactly the type of feedback that will be useful for developing appropriate information feeds that are useful to editors, so thanks. We could filter for reviews. As more of these are closed access relative to primary articles we may uniquely be able to offer a feed of snippets and facts from them, particularly those that don't exist as full-text in Europe PubMedCentral. The project will not help find more editors but we hope it will make it easier for existing volunteers by extracting papers and parts of papers that are likely to be relevant, thereby reducing the amount of time they need to spend searching for relevant information to paraphrase. --Jcmolloy (talk) 23:28, 20 September 2016 (UTC)
- As far as Wikidata goes, the primary sources tool manages to get actual factual claims like Katō Kiyomasa is male and is from Japan. It provides not only the sources where the claims are made but it provides also the claims themselves. For WikiFactMine to be useful for Wikidata it would not only have to say "paper X speaks about protein Y" but it would also have to parse the claims into the structure of Wikidata. ChristianKl (talk) 15:09, 21 September 2016 (UTC)
Relevant wikidata communities
- have already done so.
Sounds great, and these comments are intended to provoke discussion, not criticise:
- Please notify the Wikispecies community
- I'd like to see more detail of what you think a Wikimedian in Residence would contribute to the project, and the reasons for offering that person a salary below the upper end of the range suggested by WMUK (disclosure: I have worked as a WiR (not least in the Royal Society of Chemistry role mentioned), hope to do so again, and am currently pro-bono WiR at ORCID)
- You say "[a] paper would be automatically offered to editors." Please explain what this "offer" would entail. For instance, a button to "create Wikidata item"?
- "W3C annotation standards" - which standard? Please add a link.
Primary sources tool?
Do you plan to feed the data into the Primary Sources Tool? Together with the work done as part of StrepHit and upcoming improvements this might just be what is needed to push it over the edge and make it the great import tool it should be. --Lydia Pintscher (WMDE) (talk) 12:53, 5 August 2016 (UTC)
- @Lydia Pintscher (WMDE): completely agree. I believe the planned T6 and T7 technical activities can be leveraged for direct collaboration with the StrepHit team. The shared goal would thus target the primary sources tool usability. Since ContentMine and StrepHit have already started joint work at WikiCite 2016, it would be really useful to follow up on this. Specifically, T6 would match the back-end part of StrepHit T1 task, while T7 would match the frond-end part. --Hjfocs (talk) 13:32, 28 September 2016 (UTC)
- The first section does not clearly state the goal(s) of this project.
- It is not clear to whom will the grant be made? In one place it is stated that "The grant would be to ContentMine Limited". On the other hand in the infobox User:petermr is listed as the grantee.
Peter Murray-Rust is a Director of ContentMine Ltd which is a non-profit company limited by guarantee and the company would administer the grant. The application form infobox is set up in such a way that it links directly to user pages so it made most sense to put Peter as the grantee as he is the primary applicant. --Jcmolloy (talk) 23:46, 20 September 2016 (UTC)
- As I understand the project is going to use the existing software, which has been already developed by ContentMine, plus 4-8 wikidata based custom dictionaries to mine the literature on daily basis to produce ~10k of "facts" per day and then deposit them in Zenodo? If this is true, what kind of development work the proposed full time developer will be tasked with?
ContentMine has core software that we have already developed to scrape and extract facts, but there are enhancements that need to be made for it to function in a useful way for this project. You can see the technical tasks and time allocation []. The facts currently go to Zenodo but the intention is to create automated feeds. --Jcmolloy (talk) 23:46, 20 September 2016 (UTC)
- I think you should better define what you mean by the "fact" in context of this project.
- The project lists three participants but it is not clear what their responsibilities are. It is also not clear if the full time software developer and full time Wikipedian-in-Residence will be selected from these three participants or not? Who will be responsible for dictionary creation, data mining, server operations, data publication?
- In sustainability section is stated that it is a pilot project. I think that this should be stated at the beginning of the project (in the solution or goals section). In addition, if it is a pilot project, you should probably state criteria (in Measures of success section) when the project will be judged sufficiently successful for continuation.
- A more general question: How the proposed article data mining compares with existing search tools like Google Scholar? I can already type a combination of words (a "fact") and find the references that are relevant to it.
- It's not entirely clear how these "dictionaries" will be available going forward. So the proposal is to fund ContentMine to develop software to build themselves the dictionaries from Wikidata? --Jura1 (talk) 17:10, 14 August 2016 (UTC)
- The data in that link doesn't seem to be directly useful to Wikidata or Wikipedia users. In contrast to the Primary Source tool where a Wikidata user just has to press "approve", in this case the Wikidata user still has to write the whole statement himself. ChristianKl (talk) 20:00, 29 September 2016 (UTC)
How about Claims ?
As said above, I think the use of the word "fact" here is pretty confusing. Perhaps this is related to legal terminology? I think it would be useful to add some content to the proposal about what wikidata claims will be created by the various workflows you describe. The most obvious one, I think, is the claim that links an item about a journal article to items that describe the article through a property like 'primary topic'. Apart from that, I guess that you would hope to feed systems that resemble the wikidata game with the extracted snippets and allow the users to define new claims? --I9606 (talk) 23:24, 23 September 2016 (UTC)
- @I9606: I am of the same opinion. It would be nice to explicitly clarify WikiFactMine input (i.e., free text) and output (i.e., Wikidata claims), maybe by expanding the two running examples. --Hjfocs (talk) 13:55, 28 September 2016 (UTC)
In the biomedical domain there are a number of sophisticated systems for doing NER. For some of the types you describe (especially genes), non-dictionary (e.g. machine learning) based systems like BANNER have much higher accuracy (though of course are much slower). I think the fundamental concept of building tools that identify unique concepts in the scientific literature and anchoring these to Wikidata items is fantastic, but I am wondering whether it would make some sense to explore taggers that have been evolving specifically in this space for some time as you develop your pipeline. Some good open examples are at PubTator http://www.ncbi.nlm.nih.gov/CBBresearch/Lu/Demo/tmTools/#RESTfulIntroduction --I9606 (talk) 23:24, 23 September 2016 (UTC)
- @I9606: thanks for the pointers to domain-specific NER tools. I would like to add that StrepHit has already implemented a domain-agnostic NER facility  (actually more powerful than NER, since it also identifies abstract concepts, besides real-world entities). Hence, I believe WikiFactMine could definitely benefit from StrepHit NLP technology, which is already stated in the proposal by the way: Grants:Project/WikiFactMine#Technical_Activities. --Hjfocs (talk) 14:09, 28 September 2016 (UTC)
Eligibility confirmed, round 1 2016
This Project Grants proposal is under review!
We've confirmed your proposal is eligible for round 1 2016 review. Please feel free to ask questions and make changes to this proposal as discussions continue during this community comments period.
The committee's formal review for round 1 2016 begins on 24 August 2016, and grants will be announced in October. See the schedule for more details.
Just a minor comment: the link in Grants:Project/WikiFactMine#What_is_your_solution.3F, second paragraph, appears to be broken. I think it should point to wikidata:Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements. --Hjfocs (talk) 13:45, 28 September 2016 (UTC)
WikiFactMine and StrepHit
Here is a list of key points that can lead WikiFactMine to tighten the collaboration with StrepHit:
- T6 and T7 (Grants:Project/WikiFactMine#Technical_Activities) could focus on the primary sources tool, as pointed out in #Primary_sources_tool.3F;
- C2 (Grants:Project/WikiFactMine#Summary_of_Community_Engagement_Activities) is definitely a shared task, meant to rethink the whole primary sources tool workflow;
- the StrepHit team is happy to actively contribute to the Wikimania 2017 workshop, as per C5 and Grants:Project/WikiFactMine#Wikimania_2017_Workshop;
- WikiFactMine would benefit from StrepHit entity linking facility, as discussed in #Dictionaries;
- the integration of Hypothes.is (mentioned in Grants:Project/WikiFactMine#Carvone) is also foreseen by StrepHit for the primary sources tool, cf. wikidata:Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements#Highlight_text_snippet_used_to_extract_the_claim.
Aggregated feedback from the committee for Project/WikiFactMine
|(A) Impact potential
|(B) Community engagement
|(C) Ability to execute
|(D) Measures of success
|Additional comments from the Committee:
This proposal has been recommended for due diligence review.
The Project Grants Committee has conducted a preliminary assessment of your proposal and recommended it for due diligence review. This means that a majority of the committee reviewers favorably assessed this proposal and have requested further investigation by Wikimedia Foundation staff.
- Aggregated committee comments from the committee are posted above. Note that these comments may vary, or even contradict each other, since they reflect the conclusions of multiple individual committee members who independently reviewed this proposal. We recommend that you review all the feedback and post any responses, clarifications or questions on this talk page.
- Following due diligence review, a final funding decision will be announced on March 1st, 2019.
petermrj, our interview with you was part of the due diligence process (I'm getting these comments posted late). You are still welcome to record any response you have to committee comments on your talkpage to make your feedback publicly accessible, but I think we gathered the information we needed for the committee during our talk with you. Best regards, Marti (WMF) (talk) 18:46, 2 October 2016 (UTC)
Round 1 2016 decision
Congratulations! Your proposal has been selected for a Project Grant.
WMF has approved partial funding for this project, in accordance with the committee's recommendation. This project is funded with $63,700
Comments regarding this decision:
The committee is pleased to support your project with partial funding to automatically mine scientific literature for data to add to Wikidata. We value the opportunity to bring the innovation of ContentMine to the Wikidata community. We appreciate your sensitivity and responsivity to concerns raised by some members of the Wikipedia community of editors, and we’re looking forward to working with you to make sure the project design sufficiently addresses these concerns before moving forward. We believe that can be done, in part, by designing sound volunteer curation processes that can then be coordinated by the Wikipedian in Residence. Because of the need to establish these processes, however, we expect this position to have a gradual start, so funding has consequently been reduced to halftime. Along with many others, we are excited to see what new possibilities WikiFactMine will open up for Wikidata!
- You will be contacted to sign a grant agreement and setup a monthly check-in schedule.
- Review the information for grantees.
- Use the new buttons on your original proposal to create your project pages.
- Start work on your project!
I've read Grants:Project/ContentMine/WikiFactMine/Final and clicked a number of links but I didn't manage to find a concrete example of something that was produced. I found a list of SPARQL query links, some kind of viewer which might be akin to resonator and an Elasticsearch dump on Zenodo, but no use case for them.
If not, it would really be useful to add at least an example or story of at least one user using one of your products to actually do something. Thanks, Nemo 08:48, 19 May 2018 (UTC)