Grants talk:Project/Maxlath/WikidataJS

Scheduling Project Grant interview[edit]

Hello Maxlath, I will be reaching out shortly to set up your project grant interview. Can you please send your email address to me at lmiranda wikimedia · org at your earliest convenience? Thank you!LMiranda (WMF) (talk) 22:48, 4 April 2018 (UTC)[reply]

done -- Maxlath (talk) 10:45, 5 April 2018 (UTC)[reply]

A seed for a Wikidata UI toolkit[edit]

@Maxlath: I'd like to share a thought which is beyond the scope of this proposal, but still related to JavaScript. One thing that is currently missing in Wikidata is a UI toolkit to natively access UI objects: right now, we have to make massive use of DOM manipulation via jQuery. What do you think?

Cheers,
--Hjfocs (talk) 09:16, 30 January 2018 (UTC)[reply]

Hello @Hjfocs:! There is definitely a need, I remember someone from the Wikidata team mentioning that they had several requests for a Wikidata autocomplete widget for instance. But I'm not fond of UI toolkits as they force you to couple a component with a given UI technology (often React lately, but it's hard to tell what will be all the rage in 2 years from now), and while it makes it easy to start, it's, in my experience, a pain to customize and maintain. I would thus prefer that we collectively invest in good low level tools (like wikidata-sdk) providing all we need to build the logic behind those UI elements, but give some elaborated implementation examples with those different view frameworks. That said, as mentioned by @Atomotic:, in the future, we could have some beautiful standard Web Components, and that could make such an investment worth it! -- Maxlath (talk) 11:49, 30 January 2018 (UTC)[reply]

Important: Change your proposal status from "draft" to "proposed" to submit by deadline[edit]

User:Maxlath,

Please note that you must change your proposal status from "draft" to "proposed" to submit by your proposal for review in the current round. You have until end of day today to make the deadline.

Warm regards,

--Marti (WMF) (talk) 21:46, 31 January 2018 (UTC)[reply]

@Mjohnson (WMF): Thank you for notifying me! I missed that piece of information, and changed the status some 12 minutes late if the reference time is UTC, but a good few hours early if it's the US West Coast time: do you know which one applies? -- Maxlath (talk) 00:30, 1 February 2018 (UTC)[reply]

Comments from Ruslik0[edit]

Thanks for this proposal, which may really make a difference though I have a few comments:

Please, specify duration of the project.
Another problem is information about the grantee. The proposal itself contains nothing about User:Maxlath, not even a wikilink to the account! Please, provide more information about you in the text of proposal.
It is not clear in which form these tools will be made available to users? Will there be some gadgets or some user scripts? Or, since you mentioned node.js, will they be in the form of web based apps?
I often complain about walls of text that are difficult to read. But in this case the proposal contains too little information about possible practical uses of the proposed scripts.

Ruslik (talk) 18:24, 14 February 2018 (UTC)[reply]

Hi, thank you for your comments, I tried to answer them point by point:

Please let me know if some points still need more details -- Maxlath (talk) 13:07, 15 February 2018 (UTC)[reply]

Comments of Glrx[edit]

I would decline this proposal. I expected to see more functionality in the library.

The primary function of the library appears to be a syntactic mapping from a function name to a Wikidata API URL that can be used in a Wikidata query. The mappings do not perform the query, but leave it to the user to perform the HTTPS query request. Once the user performs the request, the user gets whatever the Wikidata API returns. There is not much distance between the function and the URL. In addition, the user is still responsible for throttling server requests. One example in the wdk documentation is

const url = wdk.searchEntities({
  search: 'Ingmar Bergman',
  format: 'xml',
  language: 'sv'
})

(WikidataJS also allows a more condensed wdk.searchEntities(search, language, limit, format).)

The result of this invocation is a Wikidata API wbsearchentities URL[1] that would look like

https://wikidata.org/w/api.php?action=wbsearchentities&search=Ingmar%20Bergman&format=xml&language=sv

The encoding gain seems small especially when JavaScript (and other language) libraries already include methods to build search strings that include URI encoding the values. See jQuery.param(object).[2] The general URL Living Spec has been extended to include URLSearchParams.[3] Many JavaScript implementations include URLSearchParams; IIRC, Edge recently added it. Even NodeJS has URLSearchParams.[4]. URLSearchParams can be used in a POST.[5]

It's also not hard to generate search strings from an object with a map and join. I've even done Wikidata queries by building a URL with string operators:

var url = urlWikidata + "?action=wbsearchentities" + "&search=" + encodeURI("Ingmar Bergman") + "&format=json";

The WikidataJS library has a programmer using the method wdk.searchEntities instead of using a more general URL search parameter method and including the key-value pair action: "wbsearchentites".

var searchParams = new URLSearchParams({
  action: 'wbsearchentities',
  search: 'Ingmar Bergman',
  format: 'xml',
  language: 'sv'
});

Even with WikidataJS, the user must still refer to the Wikidata API section on wbsearchentities to interpret the returned result, so hiding the action=wbsearchentities key-value is not a big feature.

I do not see a value-added avantage to the added abstraction layer. I think it would be clearer to use general search string methods that stay close to the Wikidata API.

I also have doubts about the command line interface (CLI). Yes, one can do work with it, but will it be a common way of doing that work? For queries, I get characters written to stdout. If a person wants to do the query, then why not use an existing GUI? If a machine is going to read stdout, then there's an awkward parsing step involved. It seems better to execute the query in JavaScript and look at the JSON rather than a text string. For writing to Wikidata, it may be appropriate, but I still wonder if the program that generates the command line text could almost as easily use HTTPS to Wikidata. The CLI may also have a problem with throttling. Adding a million items to Wikidata needs to be spread over time.

I'm also not sure about the Grants:Project value of the library. The initial use of the library was not improving the WMF Wikidata project but rather a private Wikidata server about book holdings. I get this uneasy feeling that WikidataJS (even if it were not an unneeded abstraction layer) is not about improving Wikidata but rather improving outside projects that want to exploit Wikidata. The proposal does not claim to have a huge store of facts that can be added to Wikidata; it is only hoping that some other users will use the library to submit its one million facts. The proposal does not give the sense that the current library has been widely used.

Glrx (talk) 21:45, 15 February 2018 (UTC)[reply]

Trying to answer on some of the points raised:

Most of wikidata-sdk functions "don't do much" indeed: I have sometime presented this lib as a working code documentation of the Wikidata API, saving new comers the hours of struggle I experienced to get data out of Wikidata.
Most of wikidata-sdk functions "don't do much", but having to do those little things over and over among projects is boring: wikidata-sdk is inspired by the NodeJS small modules/small and expressive functions culture, I prefer to write
```
wdk.isEntityId(someString)
```
rather than
```
/^(Q|P)[0-9]+$/.test(someString)
```
It doesn't do the requests itself because

depending on the environment where you use it (either in NodeJS or in the browser), the underlying HTTP lib wouldn't be the same
there are several way to handle async in JS, and I didn't want to force my way on users. This point was especially true when I first published the lib in 2015, when the JS community was in full callback vs promises war mode. I still think that letting the user decide how they handle request is a sane design choice.

Most of wikidata-sdk functions "don't do much" but some do: wdk.simplify.claims especially, while keeping things simple for the new comer, does do quite a lot (keeping only truthy statements by default, supporting multiple standard date formats, etc.)
Meanwhile, wikidata-edit does fully handle its HTTP requests (because it's designed for NodeJS only, not the browser), including auth via the underlying wikidata-token lib, which was easily the single most painful point to address in getting to edit Wikidata from an external program
About wikidata-cli will it be a common way of doing that work?: I don't know, I just know that that's a tool I needed, and I know other people enjoy using it (see Grant supports). We won't know what it could become until we give people the opportunity to use it :)
Still on wikidata-cli: if you need to parse stdout, I recommend requesting the output to be in JSON, a format than many program can work with, by passing a
```
--json
```
flag (If it doesn't work for the command you are working with, please open an issue)
On throttling issues, this could be added to the list of issues to address, but I see it more as a confirmation of the usefulness of the tool
I still wonder if the program that generates the command line text could almost as easily use HTTPS to Wikidata: I'm curious to see what you have in mind. Any edit command depends on at least 2 requests just to get an auth token, I hardly see that happen on the long run without some tool doing it for you.
I get this uneasy feeling that WikidataJS is not about improving Wikidata but rather improving outside projects that want to exploit Wikidata: that's half the motivation yes (the other half being improving Wikidata itself). As Lydia puts it, More use of Wikidata's data both inside and outside Wikimedia is crucial in order to really achieve what we built Wikidata for: giving more people more access to more knowledge
The proposal does not claim to have a huge store of facts that can be added to Wikidata; it is only hoping that some other users will use the library to submit its one million facts: I alone did 250,000 edits using those tools, I do believe we can reach the million in the coming year (and our server about book holdings does have quite a lot more CC0-licensed facts in reserve that could eventually be imported with those tools - see our raw dump or the more verbose version-, but that's another topic)

As a conclusion, a last point to bear in mind: the necessity of an abstraction is very relative to one's coding style and culture. WikidataJS is largely the result of my desire to see more of the goodies I can see in the NodeJS community come to the heavily PHP/Python-influenced Wikidata community, and I had numerous feedback at WikidataCon that this effort was indeed fruitful and welcome. Best regards, Maxlath (talk) 16:02, 17 February 2018 (UTC)[reply]

Eligibility confirmed, round 1 2018[edit]

This Project Grants proposal is under review!

We've confirmed your proposal is eligible for round 1 2018 review. Please feel free to ask questions and make changes to this proposal as discussions continue during the community comments period, through March 12, 2018.

The committee's formal review for round 1 2018 will occur March 13-March 26, 2018. New grants will be announced April 27, 2018. See the schedule for more details.

Questions? Contact us.

--Marti (WMF) (talk) 01:52, 17 February 2018 (UTC)[reply]

Aggregated feedback from the committee for WikidataJS[edit]

Scoring rubric	Score
(A) Impact potential Does it have the potential to increase gender diversity in Wikimedia projects, either in terms of content, contributors, or both? Does it have the potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	7.0
(B) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support?	6.5
(C) Ability to execute Can the scope be accomplished in the proposed timeframe? Is the budget realistic/efficient ? Do the participants have the necessary skills/experience?	7.8
(D) Measures of success Are there both quantitative and qualitative measures of success? Are they realistic? Can they be measured?	8.7
Additional comments from the Committee: The base repositories has been implemented and it has fully functional works. The project could have a great impact in the most technical users who wants a wrapper to work with wikidata. The project fits with Wikimedia's strategic priorities because promoting the use of Wikidata is within such priorities. The potential for online impact is high. The improved SDKs can be scaled, sustained and adapted outside the Wikimedia community. this proposal is certainly not one of the top global priorities, but it has a great significance in the technical dimension. This is one of those projects that is written so that it needs some knowledge in understanding the problem described and the proposed solution. I would support anyone who plans to apply in the future to devote a little more attention to the proposal and description itself so that it would be more understandable to a larger number of people Not the most important thing but good enough. I am not a big expert in JS tools for Wikidata but it sounds like these tools are quite widely used and helpful for a significant number of people. As Wikidata development and technical improvements are a part of our priorities this is a good fit. The metrics are clear, but it can't be clear auditable by an external tool. The roadmap isn't clear and the next killer-features are not explicit in the proposal. The approach is iterative. The risks appear to be low as the project is aimed at improving the existing code base. The success can be measured. the project has clearly written measures of success, but until the actual realization of the project, because of its specificity, we can not be sure whether they are achievable or not. Looks like a good iteration to bring the existing and already used project to the new level. Not the next Big Thing nor the biggest problem but delivering a tool that would be used for 1 million automated edits sounds like a good thing to do. The only minor concern is that it should be qualified over which times these goals are supposed to be reached (+1 million edits over this 12 months? X months after the end of the period? I assume it's the first option) The project haven't ambitious goals to be accomplished in the development time. The budget seem reasonable and the grantee has a large experience on own codebase. Applicant has relevant experience and is very responsive to community feedback. I believe that the participant is able to execute this project as he has a wide experience with these SDKs. The budget is ok but more details will make it better.The project can be executed within 12 months. 50€/hr has long been a business and not a hobby or desire to help global wiki mission :) in each case, if everything is as planned, both sides will be happy and satisfied Clearly feasible in 12 months and applicant sounds qualified to do it. We can go into discussion on whether 50 EUR / hour is a good or a bad rate but I assume it is a reasonable and acceptable rate (I don't have enough details on applicant's professional level to judge). The project has a large base of support and success implementations across different open source software. Lots of endorsements from Wikidata users, as well as from Lydia Pintscher, Wikidata product manager. The community support is impressive. the proposal has clear support from the target group The community engagement is 10/10 for me here: there is a very clear target community (Wikidata users, outside users of Wikidata, tool developers etc.), clear engagement and significant support from this target community. The project don't show a clear roadmap to develop in the next year. The metrics aren't easy to be taken by an external tool and the achieving the number depends on external factors (users, tool usage, datasets available, etc.) The proposal currently lacks a detailed scope and specific goals. I'm assuming that's because this is a request for work that has been ongoing since 2015 and will continue for the foreseeable future. Still, I would urge the applicant to better communicate what will be achieved through this grant and provide expected outcomes as well as link those to the budget request and estimated hours of work The cost of personel is realistic and the impact too. The project should be funded, in my opinion, although the budget should have more details. Full funding provided salary rate is approved (I don't have enough information to judge). This tool seems to be very valuable and widely used, and this proposal is likely to make a significant positive effect.

This proposal has been recommended for due diligence review.

The Project Grants Committee has conducted a preliminary assessment of your proposal and recommended it for due diligence review. This means that a majority of the committee reviewers favorably assessed this proposal and have requested further investigation by Wikimedia Foundation staff.

Next steps:

Aggregated committee comments from the committee are posted above. Note that these comments may vary, or even contradict each other, since they reflect the conclusions of multiple individual committee members who independently reviewed this proposal. We recommend that you review all the feedback and post any responses, clarifications or questions on this talk page.
Following due diligence review, a final funding decision will be announced on Thursday, May 27, 2021.

Questions? Contact us at projectgrants wikimedia · org.

following those comments, I made revisions to this proposal (diff):

problem/solution: elaborated on the identified problem and the proposed solution
goals: added details to the grant goals
budget: broke down the budget into more detailed entries, reduced the hourly rate
participants: gave more context on the participants, that is, myself, and my involvement in Wikidata, to address the expressed concerns that this might not be rooted in a volunteer contribution.

I hope those revisions address the concerns expressed by the Committee, and stay at your disposal for further improvements of this grant proposal. -- Maxlath (talk) 17:59, 18 April 2018 (UTC)[reply]

Round 1 2018 decision[edit]

Congratulations! Your proposal has been selected for a Project Grant.

The committee has recommended this proposal and WMF has approved funding for the full amount of your request, 8500 €

Comments regarding this decision:
The committee is glad to support better tools for pairing Javascript with Wikidata.

New grantees are invited to participate in a Storytelling Workshop on June 5 and a publicly streamed Project Showcase on June 14. You can learn more and sign up to participate here: Telling your story.

Next steps:

You will be contacted to sign a grant agreement and setup a monthly check-in schedule.
Review the information for grantees.
Use the new buttons on your original proposal to create your project pages.
Start work on your project!

Upcoming changes to Wikimedia Foundation Grants

Over the last year, the Wikimedia Foundation has been undergoing a community consultation process to launch a new grants strategy. Our proposed programs are posted on Meta here: Grants Strategy Relaunch 2020-2021. If you have suggestions about how we can improve our programs in the future, you can find information about how to give feedback here: Get involved. We are also currently seeking candidates to serve on regional grants committees and we'd appreciate it if you could help us spread the word to strong candidates--you can find out more here. We will launch our new programs in July 2021. If you are interested in submitting future proposals for funding, stay tuned to learn more about our future programs.

Typescript Support[edit]

First of all thanks for the great library @Maxlath:. I'm using it for many of my wikidata based projects. However the major problem is that lack of TypeScript support. TypeScript is basically the way to write Javascript now, and it's can make coding so much better. The edit functions for example accept a wide range of inputs, it's really not clear and there is no feedback on the code editor. Typescript would fix that. I'm willing to contribute if wanted. Germartin1 (talk) 14:50, 11 June 2022 (UTC)[reply]

Hi @Germartin1:, thanks for the feedback, it's always good to hear that one's work is useful :D I'm quite busy working on inventaire.io (Q32193244) this days, but should have more time to work on WikidataJS tools after 2022-10-01. In the meantime, feel welcome to open an issue, and I'll get back to you then :) -- Maxlath (talk) 15:17, 11 June 2022 (UTC)[reply]