Grants:Project/Iamamz3/Prototype A Scalable WikiData

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search


statusdraft
Prototype A Scalable WikiData
summaryPrototype a software that will allow to scale wikidata
targetmostly wikidata and all project relying on wikidata
amount61 000 EUR
nonprofitno
granteeIamamz3
contact• amirouche@hyper.dev
this project needs...
contact
advisor
organization
grantee
volunteer
join
endorse
created on01:18, 17 June 2019 (UTC)


Project idea[edit]

What is the problem you're trying to solve?[edit]

What problem are you trying to solve by doing this project? This problem should be small enough that you expect it to be completely or mostly resolved by the end of this project. Remember to review the tutorial for tips on how to answer this question.


When we take the wikidata project in its entirety, it is reported to be difficult to scale. Not only the WDQS service but also the whole infrastructure. It happens that it is also difficult to scale in terms of human skills as different part of the project requires expert knowledge that is vastly different with the foolowing tools JavaScript / PHP / MySQL / ElasticSearch / maybe REDIS and Python.

What is your solution to this problem?[edit]

For the problem you identified in the previous section, briefly describe your how you would like to address this problem. We recognize that there are many ways to solve a problem. We’d like to understand why you chose this particular solution, and why you think it is worth pursuing. Remember to review the tutorial for tips on how to answer this question.


I intent to demonstrate that is possible to scale wikidata both in term infrastructure, development and features using a single language stack (Scheme) and single database FoundationDB. With the bonus that it will be easy to setup on a single node machine relying on WiredTiger.

Project goals[edit]

What are your goals for this project? Your goals should describe the top two or three benefits that will come out of your project. These should be benefits to the Wikimedia projects or Wikimedia communities. They should not be benefits to you individually. Remember to review the tutorial for tips on how to answer this question.


  • Scale wikidata
  • A simpler programming stack
  • Clear the sky for future developments of Wikimedia like Abstract Wikipedia

Project impact[edit]

How will you know if you have met your goals?[edit]

For each of your goals, we’d like you to answer the following questions:

  1. During your project, what will you do to achieve this goal? (These are your outputs.)
  2. Once your project is over, how will it continue to positively impact the Wikimedia community or projects? (These are your outcomes.)

For each of your answers, think about how you will capture this information. Will you capture it with a survey? With a story? Will you measure it with a number? Remember, if you plan to measure a number, you will need to set a numeric target in your proposal (i.e. 45 people, 10 articles, 100 scanned documents). Remember to review the tutorial for tips on how to answer this question.


  • All the queries from the survey from 2017/2018 must be runnable
  • Performance will be measured but the conclusion must take into account the conditions of the benchmark (mainly the hardware used to do the benchmark)
  • The performance of the solution could be measured in terms future of development times and operational costs

Do you have any goals around participation or content?[edit]

Are any of your goals related to increasing participation within the Wikimedia movement, or increasing/improving the content on Wikimedia projects? If so, we ask that you look through these three metrics, and include any that are relevant to your project. Please set a numeric target against the metrics, if applicable.


Project plan[edit]

Activities[edit]

Tell us how you'll carry out your project. What will you and other organizers spend your time doing? What will you have done at the end of your project? How will you follow-up with people that are involved with your project?

  • I will be mostly coding and doing regular updates
  • I hope to join Wikimedia foundation
  • I will continue to take part in wikidata and related projects
Quarter Title Tasks Release
1 SPARQL on FoundationDB
  • Backend: Adapt existing FoundationDB bindings to work with Chez Scheme
  • Backend: Implement a minimal pre-fork HTTP server
  • Backend: Support SPARQL (support geo and labels)
  • API: SPARQL endpoint that can be queried
2 Frontend
  • Frontend: Adapt Chibi Scheme wasm build to allow asynchronous requests
  • Frontend: Port wikibase main interface to edit concepts and lexemes
  • GUI: edit concepts and lexemes that is directly accessible in SPARQL
3 Change request
  • Backend: Make sure the change request mechanic works correctly and time travelling queries are possible
  • Backend: Expose a read only API that will allow time travelling queries and history metadata using a custom syntax (not SPARQL)
  • Frontend: Expose the change request mechanic in the graphical user interface
  • Frontend: Display history of concepts and lexemes as tuples add / delete actions
  • API: history and time travelling queries
  • GUI: make change request
4 bugfixes
  • bugfixes
  • working prototype

Budget[edit]

How you will use the funds you are requesting? List bullet points for each expense. (You can create a table later if needed.) Don’t forget to include a total amount, and update this amount in the Probox at the top of your page too!

Community engagement[edit]

How will you let others in your community know about your project? Why are you targeting a specific audience? How will you engage the community you’re aiming to serve at various points during your project? Community input and participation helps make projects successful.

  • I will continue to blog on hyper.dev
  • I will update the community with weekly (or monthly) summaries and request for feedback
  • I will continue to hang on freenode. My freenode handle is amz3
  • Receive feedback and bug reports via phabricator

Get involved[edit]

Participants[edit]

Please use this section to tell us more about who is working on this project. For each member of the team, please describe any project-related skills, experience, or other background you have that might help contribute to making this idea a success.

I am software engineer with 8 years of professional experience. I have done too many projects to cite only a few. Here are my repositories bitbucket, github, source hut. The most interesting stuff is summarized in my blog. I might not be senior, I still confident I can achieve this project. I have one cat and one dog.

Community notification[edit]

You are responsible for notifying relevant communities of your proposal, so that they can help you! Depending on your project, notification may be most appropriate on a Village Pump, talk page, mailing list, etc.--> Please paste links below to where relevant communities have been notified of your proposal, and to any other relevant community discussions. Need notification tips?

Endorsements[edit]

Do you think this project should be selected for a Project Grant? Please add your name and rationale for endorsing this project below! (Other constructive feedback is welcome on the discussion page).