Maybe this is an idea already discussed in "wiki space" already. But I have "only" been a contributor to wikipedia (english) for a bit over a year. I also happen to be a university professor.
The concept of peer review  is a foundational one in scholarly journals -- in which contributions are "reviewed" by external referees prior to publication. There are a few electronic journals that have been established which use a formal peer review process prior to "publication" in electronic media.
Let me analogize (I apologize in advance, I am an engineering professor not a computer scientist). My understanding of the concept of open source software, and also to some extent of the wikipedia and other wiki foundation projects is that they rely on the concept that many "eyeballs" improve the product (the converse of "too many cooks spoil the broth"). Suppose a WikiJournals project were organized in which scholarly articles (with appropriate footnoting, etc) were downloaded and subject to the "review" of the wiki-universe. I am impressed at the relative lack of errors in the wikipedia articles that are in areas I am highly knowledable about.
Peer reviews in scholarly journals are conducted by selected reviewers known to an editor. Could it not be that a WikiJournal would attract a community of knowledgeable contributors and reviewers (and unlike traditional journals -- either electronic or print -- the nature of Wiki permits intensive dialoging (and very import to me -- it seems -- archiving and potential for redaction of changes). My observation of the dynamics in wikipedia is that except for politically/culturally charged topics, after a while the edits do die off.
Of course such a project would require some "early adopter" volunteers amongst the scholarly community.
I look forward to comments.
--Professor Water 22:48, 5 Jul 2004 (UTC)
Personal opinion: I think there will be problems with the "ownership" of articles. Editing an encyclopedia article which is supposed to be general is different to editing an article where someone else put his personal style and research results in it. How do you think a scholarly journal could benefit from using a wiki? Do you know http://plos.org/ ? A german wikipedian had the idea of creating a wiki for abstracts (and translation of abstracts) of scholarly articles, I gave him a pointer to this page here. Maybe this could be also discussed in the context of the proposed Wikiversity and Wikiresearch. --Elian 17:52, 6 Jul 2004 (UTC)
- In review articles there is less personal style. A scholarly journal with review articles and/or abstracts could benefit a lot from using a wiki but I doubt that you can list your wiki edits in your publication list :-( on the other hand Wikipedia does have a huge w:impact factor ;-) By the way I really think of funding a journal of our own within the next years (some kind of WikiReader WikiResearch) -- Nichtich 21:08, 6 Jul 2004 (UTC)
See http://www.livingreviews.org/. These journals contain peer reviewed, open access and regularly updated review-articles. It's only a little step further to do it with a Wiki. Are you thinking of using a Wiki for the internal process of peer review or publically available? -- Nichtich 21:08, 6 Jul 2004 (UTC)
WikiAbstracts could also be started with a couple of motivated (PhD) students. WikiReviews is nothing but excellent Wikipedia articles with a full bibliography treating research topics. -- Nichtich 21:27, 6 Jul 2004 (UTC)
Thanks for the above comments. A few reactions -- 1) http://plos.org/ is a "classical" electronic journal (see also for example British Medical Journal -- http://bmj.bmjjournals.com/ ). In this model, papers are still submitted, reviewers are selected by an editor, and a decision is rendered. This relies in the editor being able to select knowledgeable reviewers and also in the editor him/her self being unbiased. A WikiScholarlyJournal would open up review to the wikiuniverse. Would this work -- the empirical evidence from w:Wikipedia seems to be that "on average" open reviewing leads to a good product. Thanks for the link to http://www.livingreviews.org/, however it is a bit similar to plos, although the articles appear to be envisioned as somewhat more dynamic, although the review process is still a static one.
2) It might be that a WikiReviews would be the way to test the concept out -- open, NPOV reviewed articles. The "History" function of the wiki seems to be one that is amenable to citation for the primary author and contributors to review processes of articles. As an academic myself, this is not something that I would ever advise a junior (nontenured) faculty member to engage in, however it might lead to a fundamental creation of a new vehicle for dissemination and evaluation of knowledge.
- Sorry for the 10+ years delay for me to find this page. There is now a project in Wikiversity which works as a open-access journal where works undergo peer review: Wikijournal. Mikael Häggström (talk) 11:33, 16 August 2016 (UTC)
Wikiresearch << >> Wiki Scholarly Journals
I think this is sort of what I meant with Wikiresearch :) Let's merge, okay? See also this thingy about Wikiresearch. Only, to think of it is as a journal is kind of redundant, I think, wikis are more or less timeless. Akagu 20:58, 8 Jul 2004 (UTC)
Akagu 20:58, 8 Jul 2004 (UTC)
It seems to me that Internet Think Tank should also be merged into the same project as well. Why limit WikiResearch or Wiki Scholarly Journals to scientific research, when it can include research about politics and anything else? - TalkHard, 6 Aug 2004
- This seems to be a hard science vs. soft science question -- Nichtich 09:49, 7 Aug 2004 (UTC)
All of the above have a lot of similarity with an idea that I've been toying with over the past year. I figure it doesn't hurt to expand on the idea here, since it can always be edited out later if it's of no use.
The focus of the site I had in mind was to present specialist information (both original, and historical) which I define as that which requires significant understanding of prior knowledge. For instance, it is not expected that you could understand non-linear control systems, without a thorough understanding of calculus, or linear control systems for example. This differs from the wikipedia, where each article can generally be read and understood by itself.
Because the information on the site is specialised, there are two main uses I could think of. There is the open-learning area, which I think is probably adequately covered by a number of other sites, and the research aid which has relatively little competition. As a researcher myself, I find that there are numerous problems with existing research tools which could be overcome by this site. These include
Finding research in related areas: Currently, this involves finding the correct paper with a guess the keyword type search. Then once a paper is found, you chase up its citations and things that cite it. Many of these citations are only in related areas, and not of specific interest. Also, important papers can easily be missed if they involve the same sort of problem, but are in a different application, as there is not a lot of cross-disciplinary citation(even between very similar areas such as statistics and machine learning).
Understanding new papers: Papers on similar problems in different subject areas, or by people with different technical backgrounds are often difficult to understand because of different assumed knowledge, or notation, missing lines or unexplained working, or generally poor writing. Also, there is a lot of repetition of ideas within papers, and often a lot of these need to be read fully to work out what (if anything) is new to the paper.
Paper publication: Traditional paper publication is a slow process, requiring fairly extensive surveys of the literature (which are very time consuming), writing an introduction to the problem (which has often already been described better in other papers), repeating information from other papers (necessary for the paper to have a chance of being understood by readers), and then finally the main text of the paper. Also, writing the papers usually require style files which are different for each journal. Then the paper needs to run the gauntlet of reviewers, and the wait can be quite long.
The above problems could (eventually) be solved using a wiki, with main articles which can have edits suggested by everyone, and a tree like structure consisting of directed links. For instance, an article would have a special backwards link to other articles which need to be understood before the current article makes sense (the wiki could check that there aren't any loops). The articles could then be arranged in topics (not necessarily related to the nodes of the tree), and ranked by "popularity" to make it easy to find articles directly related to the area in which you are interested. Original research could be submitted either as a new article, or a "brain storming" article where people pool ideas until the problem is (hopefully) solved.
One of the more difficult problems faced by this type of wiki will be getting people to contribute. This is because most people who would contribute original research have jobs which reward the recognition provided by journal publications. This is, in fact, probably the only reason people publish in a journal, as opposed to web publishing. If this wiki somehow provides an equal level of recognition (I propose a karma like score which I will elaborate on later), then I don't see anybody having reason to publish anywhere else (although it might take quite a while for sufficient numbers of people to start using it).
Detailed description: This is a sort of page by page description of how I would see the site working. It is incomplete in some areas since certain problems/features only occurred to me as I was writing this, and I haven't had time to come up with a solution.
FRONT PAGE: This would contain links to the main SUBJECT AREAS (such as Mathematics, Chemistry, Philosophy, Geology, Physics, etc.) It would also allow someone to LOGIN to their homepage.
SUBJECT AREAS: The name of the subject area would be at the top of the page, with a link to the REVIEW article for that area. Separate lists for subtopic SUBJECT AREAS, normal ARTICLES and BRAINSTORM articles should be present, and ranked by popularity (number of people subscribed).
If the user is logged on, the user should only be shown articles that he has not ignored, and be given the option to ignore articles for next time, or to turn off the ignore option and see all of the articles. The ignore option will allow user to easily see new articles without having to wade through a list. Articles which the user has subscribed to should be made bold.
Also, if the user is logged on, they should have the option of selecting three articles which belong in a new sub-topic, or to move an article to another subject (if the number of articles left is two or less, then the current sub-topic is folded back into its original subject area), or to make a symbolic link from one area into another subtopic, etc.
Finally, the user should have the option of adding a NEW DOCUMENT for review.
LOGIN: This page would ask for certain details about the user (name, e-mail would be compulsory. Affiliation, country and state might also be useful. After entering this information, an account is created, and they are taken to the OPTIONS page.
HOMEPAGE: This contains a prominent link to the OPTIONS page, as well as separate lists for alerts (significant changes to subscribed articles and new articles in subscribed topics), as well as subscribed articles and subscribed topics.
OPTIONS: Contains various options for configuring the site. I haven't thought too much about what this might contain yet.
REVIEW: The review article for a subject area should briefly describe the subject (or problem) and any sub-topics (with links to their SUBJECT AREAS). If the subject is at the top level (so no sub-topics.. something like corner detection in machine vision might be an example), it should give a brief review of all of the articles. A list of all uncited articles in the subject area should appear automatically at the bottom of the review, to indicate that they should be mentioned in a later update. The review might also have links to the users and organisations with the largest "karma" in this area (although perhaps this will be more useful on the SUBJECT AREA page).
ARTICLES: These are the informational pages of the site, discussing completed research.
Each article would start with a list of documents with backwards links (just normal links, but are specified differently in the wiki source, to imply that the documents are required to be understood before the current document can be). The links would only be to the highest level links, so for example if a "dynamics" article required "calculus" and "arithmetic", and "calculus" already required "arithmetic", then only a link to "calculus" would be included in the back links. Back links to articles that the user has not claimed they "understood" (see end of the ARTICLES description) should be displayed in bold case.
Afterwards, there would be a link to the REVIEW page for the article, which would take the place of an introduction and/or problem definition in a printed journal. Following this would be the main text of the article. Links to similar articles may be included using normal links. Proofs and other arguments should have symbols next to them (sort of like zooming in and out) which allows more or less proof detail to be presented, since if you are just skimming the article, seeing all of the detail may not be helpful. There could also be links to alternative proofs of the same theorem.
Finally, at the end of the article, there should be a list of articles directly depending on this article (all articles with backwards links to this article). There should also be buttons which allow people to ask a NEW QUESTION about the article (sort of like a software bug report) or to suggest an EDIT to the paper (which can only be done after they have reviewed the existing edits, unless it is a minor correction). They could also ANSWER QUESTIONS, VOTE on suggested changes, or click on a text-box to indicate that they fully "understand" the article.
EDIT: Allows the user to suggest an edit to an ARTICLE. Because the subject areas covered by the wiki are quite technical, it can be very easy for errors to be introduced which can invalidate the entire article (especially in mathematics). For this reason, while anyone can suggest changes, some sort of peer review is necessary before they can be implemented. This is accomplished by requiring a VOTE on new changes (unless they are of a minor nature, such as spelling, or rewording sentences).
The first step is to ask the user whether the article is a spelling/rewording change, or a more major change. The first would not need peer review, but would get little KARMA.
The next step is the actual editing. I'm only slightly familiar with the current wikipedia editing facility, so the existing editing tools may already be better than that suggested here. First, the existing text would be placed in a text-box, and allow the user to add editing marks surrounding the sections that need to be edited. When this is finished, the contiguous sections are displayed one at a time, with text-boxes underneath each to contain the suggested change as input by the user. Finally, the change is either implemented, or an edit page is created where it can be VOTEd on.
VOTE: Voting on suggested changes to an article has two purposes. Firstly, it ensures that the changes have been vetted by an external source, so the changes are less likely to be flawed. Secondly, it allows feedback on the quality of the change to determine how much karma the author should be given if the change is implemented.
The vote page would 1) show the contiguous regions of the original text, followed by the suggested change. 2) show a list of comments related to the proposed change, and give the user the option of ADDing TO DISCUSSION. 3) A voting ballot box, where the user can select whether making the change is a step in the right direction, and to rate how important the change is to the entire article. After a certain time and/or percentage of subscribers have voted, the change is implemented according to the results of the vote.
ADD TO DISCUSSION: Prompts for the text of the discussion.
NEW QUESTION: Prompts for the text of the question to be asked.
ANSWER QUESTIONS: If a change has been made to the document (or if the answer appears in a previous document the inquisitor should have read but didn't), then this information should be communicated to the person who asked. When going to the answer questions page, the user should either select a link to the EDIT page containing the answer to the question, or type in the number of the article containing the information. An alert is then sent to the questioner stating that the question has been answered. Clicking on this alert would send the user to the question page, along with the answer, and a checkbox should appear asking if the question had been adequately answered. If the user clicks no, then a textbox appears where the questioner should explain what is wrong with the proposed answer. Otherwise, the question is removed.
BRAINSTORM: In a brainstorm, a problem of some sort is defined, and people suggest ways for the problem to be solved. Sometimes people are only able to solve part of a problem, and so this work would be less likely to be published. This type of article makes it easier to publicly collaborate, and get recognition (or karma) for partial solutions.
The brainstorm page would actually be a collection of linked pages, with threads. The first page would be the parent page. This would, at the top, have a notice if the brainstorm had been turned into an article (see BRAINSTORM TO ARTICLE) and be followed by a list of prior articles (articles with backward links) and description of the problem and possibly a partial solution (written by the user who started the brainstorm). Directly beneath this would be a list of threads, with brief descriptions and links to the appropriate thread page (each of which would be very similar in format to the main page, but fewer buttons... for instance, the BRAINSTORM TO ARTICLE link might only be on the parent BRAINSTORM page). Each thread would correspond to a different approach to solving the problem. Here, there should also be two links. An ADD TO BRAINSTORM button, which allows the user to add a new thread, and a BRAINSTORM TO ARTICLE link, which gives the user the option of selecting nodes from the BRAINSTORM pages, which will be modified to turn into an article once the problem has been solved. After this, follows a set of short notes and comments. At the bottom, a button should be provided to add a comment, and possibly checkboxes could be supplied next to each comment to indicate the usefulness of each comment (this would only be useful if the number of comments is very high, so this might not be necessary).
ADD TO BRAINSTORM: Allows you to add a thread to the BRAINSTORM. This thread should be more substantial than just a comment, or a citation of some reference. When this page is selected, it should prompt for a short title, a short description, and then the main text of the article. At the bottom there should be two buttons. One just saves the thread, but the other saves the thread and automatically goes to the BRAINSTORM TO ARTICLE PAGE (in the case where the thread added completes the solution to the brainstorm problem).
BRAINSTORM TO ARTICLE: Once the BRAINSTORM problem has been solved, the solution (which may span multiple threads) needs to be consolidated into the easily readable form of an ARTICLE. The first step is to allow the user to choose the threads from the BRAINSTORM which make up the final solution. These are then collated into one document, which the user then edits to produce the final text. An article is then created, and the user is taken to the REVIEW BRAINSTORM page.
REVIEW BRAINSTORM: Once a BRAINSTORM has been turned into an article, it needs to be reviewed to ensure that the article adequately answers the BRAINSTORM problem, and that contributions from the BRAINSTORM have been correctly and completely cited. For the first, the user needs to be presented with a checkbox to indicate whether they think the BRAINSTORM problem has been adequately solved. For the second, people who subscribed or contributed to the BRAINSTORM should be sent an alert asking them to check that the article has been adequately cited. I haven't quite worked out how to handle adding an attribution to a thread if one (or more) is missing. There should probably also be a check that the article to which a citation is suggested is older than the article itself. There also needs to be some way of portioning the credit for the article, also probably by vote.
NEW DOCUMENT: Prompts for the title of the document, whether it is a new article, new summary of existing work, or a brainstorm. It should also give the option of producing a "secret" article, which would allow people to work on an article (along with any nominated collaborators) amongst themselves, prior to eventually making the article public. If the article is secret, a URL for the article is displayed so the author can access it later. It then prompts for the text of the document, and creates it.
KARMA: I don't have a complete idea of how "karma" should be handled, and this will probably need significant tweaking. There will probably also be strategies found to increase karma in ways not intended. The general philosophy behind karma is that it measures expertise in a particular area. It should increase with the number of articles added (with greater emphasis on higher quality and original articles) as well as contribution to discussion and reviewing of papers.
The total "karma" of a person in a particular subject area would be made up of several components. The first, and largest, would be contributing new articles and significant improvements made to existing articles. The score for this would be proportional to the number of subscribers (although perhaps some other monotonically increasing function with decreasing slope. There might also need to be a minimum number of subscribers before any contribution starts, to prevent people adding many stub articles to gain karma).
The second largest component of karma would be due to adding review articles. Each article should generate a fixed level of karma (perhaps the equivalent of 1/10 of an average original article) once the number of subscribers has reached a certain minimum. This should not be proportional to the number of subscribers because this does not necessarily reflect the quality of the review, but the popularity of the subject that was chosen to review.
The next largest karma component would either be for contributing to a BRAINSTORM or adding a discussion to an EDIT page of an ARTICLE. Both of these would be measured as a total number over the previous six months, and either capped at some level, or allow discussion points to be weighted (allow people to have a checkbox next to each item to choose whether it was a useful, indifferent or unhelpful suggestion), so that inane comments could generate negative karma.
Karma should also be generated for spelling/wording corrections and questions, but since these are not moderated in any way, they should definitely be capped, so that the total amount generated this way does not exceed a certain amount.
Finally, the number of articles subscribed to or "understood" should not affect the karma at all, since it is trivial to subscribe to all articles, without contributing anything, or demonstrating any knowledge of those subject areas.
HELP: I would like to see help pages appearing automatically when a logged on user enters a new area of the site. For instance, if someone is adding an ARTICLE for the first time, then some information describing what they are expected to do would appear. The pages would also appear as an index off a HELP link somewhere on each page. The individual HELP pages could be edited in the same way as an ARTICLE, except that even spelling and wording changes would need to be VOTEd on.
Another feature which the site might eventually contain is an employer/ment finder. Here universities or industry might submit an employment offer with a specified minimum "karma" (or subject knowledge score or whatever measure it's called) in a particular combination of subject areas, along with salary information, etc. The site could then compare all of the user's qualifications and job requirements against those of the job, and issue an alert to the user when a match is found. This might be good as a paid service for the employers placing the ad, to raise revenue for the site (if this service were free, it seems to me it could easily be abused by spammers). I don't expect this feature would be used much until after the site has taken off though.
I think the best way forward from this might be to use the above points as a starting point for discussions, and get some suggestions for improvements (on which I'm sure there will be many), and try to agree on some detailed structure for the web site. Then once this is done, we just need to trick people into implementing it in actual code.
Tristrom Cooke: August 20
I don't know if this is the correct place for such a comment, if it is not my apologies. I have been working with a project 'Old Tibetan Documents Online' where we use a wiki that has to be logged in to, and the results are published as a book and webcite. In general the wiki component has been extremely effective. I think this has great potential for other research projects, and that it would be advantageous to not have to rely on the small amount of economic support from a university that the last project required. I have an active fantasy for a similar Old Burmese project. While I find a fair amount of discussion on the development of wiki-journals, and there is of course wiki-books for books. I cannot find any discussion of this sort of wiki research project, where for starters it makes sense to have users of the project use a different interface than editors, even if these two groups have the same members. I would welcome any thoughts. You can find me on the English language wikipedia as 'Tibetologist'.