Discussion:
[Tools-discuss] Reviewing with Phabricator (and Datatracker integration)
Eric Rescorla
2018-05-31 14:04:00 UTC
Permalink
I wanted to let the tools team know about the way I'm using
Phabricator (https://en.wikipedia.org/wiki/Phabricator) and the new
Datatracker APIs to do my
AD and IESG. Phabricator's "Differential" tool is a code review
system that allows for:

- Line-by-line reviews
- Status and comment tracking
- Reviewer assignment
- Differences between any two versions of a document, complete
with in-context display of comments

The basic model I have is that I treat each draft as it if were a
submitted patch to the repository of all drafts, and then review it. I
find this lets me go through a draft relatively quickly, note issues,
then create a review, and then when a new version comes out, see how
the authors resolved them. In some cases, the authors make their own
Phabricator accounts and respond in comments (this is a better
experience for everyone) but even without that, it's been a pretty big
improvement in experience for me.


Here's an example of what this looks like on Phabricator:

https://mozphab-ietf.devsvcdev.mozaws.net/D5132

And the corresponding (auto-generated) ballot:

https://datatracker.ietf.org/doc/draft-ietf-tram-stunbis/ballot/

Sorry, you'll need to search, no anchors on this page.


I've developed a suite of tools to automate most of the work here:

https://github.com/ekr/iphab


Roughly speaking, here's what they do:

- Watch the draft repository and keep the list of revisions
(Phabricator's term for change lists).

- Parse the telechat agenda and assign revisions to be reviewed
based on the documents on the agenda (Protocol Actions get
marked as "Blocking" and Document Actions as non-blocking).

- When a review is finished, post it as a ballot in the datatracker
(using whether the revision was "accepted" or not to determine
how to ballot). It also automatically sorts comments in to
DISCUSS and COMMENT based on whether they are marked "IMPORTANT".

- Alternatively, download reviews into a text file so you can
edit them, email, etc.


Right now, this is still a bit manual (I run the tools on my machine
before each telechat and you have to individually tell them to submit
the ballot) and only sort of works for > 1 person but if other people
were interested in trying Phabricator out, it would be pretty easy to
make it work for more than one person, and at least automate the
updating piece of it (you'd probably still have to manually tell it to
ballot for each draft for a little while). I've avoided doing that
because it *is* work, but would be happy to if there was interest.

Big thanks to Henrik and the tools team for making the Datatracker
APIs (and the API key facility) that makes this possible.

-Ekr
Chris Morrow
2018-05-31 16:48:37 UTC
Permalink
On Thu, 31 May 2018 10:04:00 -0400,
[1 <multipart/alternative (7bit)>]
[1.1 <text/plain; UTF-8 (7bit)>]
I wanted to let the tools team know about the way I'm using
Phabricator (https://en.wikipedia.org/wiki/Phabricator) and the new
Datatracker APIs to do my
AD and IESG. Phabricator's "Differential" tool is a code review
- Line-by-line reviews
- Status and comment tracking
- Reviewer assignment
- Differences between any two versions of a document, complete
with in-context display of comments
oh neat, this looks a lot like:
gerrit.codereview.com
codereview.appspot.com

(codereview was the originator of those two) we use this sort of flow
at $WORK, it's super useful and makes making progress straightforward
and simple.
The basic model I have is that I treat each draft as it if were a
submitted patch to the repository of all drafts, and then review it. I
find this lets me go through a draft relatively quickly, note issues,
then create a review, and then when a new version comes out, see how
the authors resolved them. In some cases, the authors make their own
Phabricator accounts and respond in comments (this is a better
experience for everyone) but even without that, it's been a pretty big
improvement in experience for me.
https://mozphab-ietf.devsvcdev.mozaws.net/D5132
https://datatracker.ietf.org/doc/draft-ietf-tram-stunbis/ballot/
Sorry, you'll need to search, no anchors on this page.
https://github.com/ekr/iphab
- Watch the draft repository and keep the list of revisions
(Phabricator's term for change lists).
- Parse the telechat agenda and assign revisions to be reviewed
based on the documents on the agenda (Protocol Actions get
marked as "Blocking" and Document Actions as non-blocking).
- When a review is finished, post it as a ballot in the datatracker
(using whether the revision was "accepted" or not to determine
how to ballot). It also automatically sorts comments in to
DISCUSS and COMMENT based on whether they are marked "IMPORTANT".
- Alternatively, download reviews into a text file so you can
edit them, email, etc.
Right now, this is still a bit manual (I run the tools on my machine
before each telechat and you have to individually tell them to submit
the ballot) and only sort of works for > 1 person but if other people
were interested in trying Phabricator out, it would be pretty easy to
make it work for more than one person, and at least automate the
updating piece of it (you'd probably still have to manually tell it to
ballot for each draft for a little while). I've avoided doing that
because it *is* work, but would be happy to if there was interest.
Big thanks to Henrik and the tools team for making the Datatracker
APIs (and the API key facility) that makes this possible.
-Ekr
[1.2 <text/html; UTF-8 (quoted-printable)>]
[2 <text/plain; us-ascii (7bit)>]
___________________________________________________________
Tools-discuss mailing list
https://www.ietf.org/mailman/listinfo/tools-discuss
Please report datatracker.ietf.org and mailarchive.ietf.org
bugs at http://tools.ietf.org/tools/ietfdb
Please report tools.ietf.org bugs at
http://tools.ietf.org/tools/issues
___________________________________________________________
Tools-discuss mailing list
Tools-***@ietf.org
https://www.ietf.org/mailman/listinfo/tools-discuss

Please report datatracker.ietf.org and mailarchive.ietf.org
bugs at http://tools.ietf.org/tools/ietfdb
or send email to datatracker-***@ietf.org

Please report tools.ietf.org bugs at
http://tools.ietf.org/tools/issues
or send email to ***@tools.ietf.org
Russ Housley
2018-05-31 17:30:54 UTC
Permalink
I would be interested in the experiences of other ADs.

I would like to know how easy it would be for Directorates and Review Teams to leverage your work.

Russ
Post by Eric Rescorla
I wanted to let the tools team know about the way I'm using
Phabricator (https://en.wikipedia.org/wiki/Phabricator <https://en.wikipedia.org/wiki/Phabricator>) and the new Datatracker APIs to do my
AD and IESG. Phabricator's "Differential" tool is a code review
- Line-by-line reviews
- Status and comment tracking
- Reviewer assignment
- Differences between any two versions of a document, complete
with in-context display of comments
The basic model I have is that I treat each draft as it if were a
submitted patch to the repository of all drafts, and then review it. I
find this lets me go through a draft relatively quickly, note issues,
then create a review, and then when a new version comes out, see how
the authors resolved them. In some cases, the authors make their own
Phabricator accounts and respond in comments (this is a better
experience for everyone) but even without that, it's been a pretty big
improvement in experience for me.
https://mozphab-ietf.devsvcdev.mozaws.net/D5132 <https://mozphab-ietf.devsvcdev.mozaws.net/D5132>
https://datatracker.ietf.org/doc/draft-ietf-tram-stunbis/ballot/ <https://datatracker.ietf.org/doc/draft-ietf-tram-stunbis/ballot/>
Sorry, you'll need to search, no anchors on this page.
https://github.com/ekr/iphab <https://github.com/ekr/iphab>
- Watch the draft repository and keep the list of revisions
(Phabricator's term for change lists).
- Parse the telechat agenda and assign revisions to be reviewed
based on the documents on the agenda (Protocol Actions get
marked as "Blocking" and Document Actions as non-blocking).
- When a review is finished, post it as a ballot in the datatracker
(using whether the revision was "accepted" or not to determine
how to ballot). It also automatically sorts comments in to
DISCUSS and COMMENT based on whether they are marked "IMPORTANT".
- Alternatively, download reviews into a text file so you can
edit them, email, etc.
Right now, this is still a bit manual (I run the tools on my machine
before each telechat and you have to individually tell them to submit
the ballot) and only sort of works for > 1 person but if other people
were interested in trying Phabricator out, it would be pretty easy to
make it work for more than one person, and at least automate the
updating piece of it (you'd probably still have to manually tell it to
ballot for each draft for a little while). I've avoided doing that
because it *is* work, but would be happy to if there was interest.
Big thanks to Henrik and the tools team for making the Datatracker
APIs (and the API key facility) that makes this possible.
-Ekr
___________________________________________________________
Tools-discuss mailing list
https://www.ietf.org/mailman/listinfo/tools-discuss
Please report datatracker.ietf.org and mailarchive.ietf.org
bugs at http://tools.ietf.org/tools/ietfdb
Please report tools.ietf.org bugs at
http://tools.ietf.org/tools/issues
Eric Rescorla
2018-05-31 20:27:36 UTC
Permalink
Post by Russ Housley
I would be interested in the experiences of other ADs.
I think Warren has spent a little time on it.


I would like to know how easy it would be for Directorates and Review Teams
Post by Russ Housley
to leverage your work.
Understanding that would, I think, require understanding how those tools
work. I don't have a real sense of how those tools work. I know Tero has
some secdir stuff. Does everyone have their own thing?

-Ekr

Russ
Post by Russ Housley
I wanted to let the tools team know about the way I'm using
Phabricator (https://en.wikipedia.org/wiki/Phabricator) and the new
Datatracker APIs to do my
AD and IESG. Phabricator's "Differential" tool is a code review
- Line-by-line reviews
- Status and comment tracking
- Reviewer assignment
- Differences between any two versions of a document, complete
with in-context display of comments
The basic model I have is that I treat each draft as it if were a
submitted patch to the repository of all drafts, and then review it. I
find this lets me go through a draft relatively quickly, note issues,
then create a review, and then when a new version comes out, see how
the authors resolved them. In some cases, the authors make their own
Phabricator accounts and respond in comments (this is a better
experience for everyone) but even without that, it's been a pretty big
improvement in experience for me.
https://mozphab-ietf.devsvcdev.mozaws.net/D5132
https://datatracker.ietf.org/doc/draft-ietf-tram-stunbis/ballot/
Sorry, you'll need to search, no anchors on this page.
https://github.com/ekr/iphab
- Watch the draft repository and keep the list of revisions
(Phabricator's term for change lists).
- Parse the telechat agenda and assign revisions to be reviewed
based on the documents on the agenda (Protocol Actions get
marked as "Blocking" and Document Actions as non-blocking).
- When a review is finished, post it as a ballot in the datatracker
(using whether the revision was "accepted" or not to determine
how to ballot). It also automatically sorts comments in to
DISCUSS and COMMENT based on whether they are marked "IMPORTANT".
- Alternatively, download reviews into a text file so you can
edit them, email, etc.
Right now, this is still a bit manual (I run the tools on my machine
before each telechat and you have to individually tell them to submit
the ballot) and only sort of works for > 1 person but if other people
were interested in trying Phabricator out, it would be pretty easy to
make it work for more than one person, and at least automate the
updating piece of it (you'd probably still have to manually tell it to
ballot for each draft for a little while). I've avoided doing that
because it *is* work, but would be happy to if there was interest.
Big thanks to Henrik and the tools team for making the Datatracker
APIs (and the API key facility) that makes this possible.
-Ekr
___________________________________________________________
Tools-discuss mailing list
https://www.ietf.org/mailman/listinfo/tools-discuss
Please report datatracker.ietf.org and mailarchive.ietf.org
bugs at http://tools.ietf.org/tools/ietfdb
Please report tools.ietf.org bugs at
http://tools.ietf.org/tools/issues
Russ Housley
2018-05-31 20:42:57 UTC
Permalink
Post by Russ Housley
I would be interested in the experiences of other ADs.
I think Warren has spent a little time on it.
I would like to know how easy it would be for Directorates and Review Teams to leverage your work.
Understanding that would, I think, require understanding how those tools work. I don't have a real sense of how those tools work. I know Tero has some secdir stuff. Does everyone have their own thing?
No. The datatracker supports all of the reviews in a manner similar to ballots. The reviewer two options. First, the reviewer puts text into a frame on a datatracker page, and then the datatracker turns it into email. This is very similar to the way IESG ballots are handled. Second, the reviewer sends his own email and give the datatracker the URL into the mail archive.

Your description makes me wonder if this tooling could work with an API related to the first approach.

Russ
Eric Rescorla
2018-05-31 20:59:27 UTC
Permalink
Post by Eric Rescorla
Post by Russ Housley
I would be interested in the experiences of other ADs.
I think Warren has spent a little time on it.
I would like to know how easy it would be for Directorates and Review
Post by Russ Housley
Teams to leverage your work.
Understanding that would, I think, require understanding how those tools
work. I don't have a real sense of how those tools work. I know Tero has
some secdir stuff. Does everyone have their own thing?
No. The datatracker supports all of the reviews in a manner similar to
ballots. The reviewer two options. First, the reviewer puts text into a
frame on a datatracker page, and then the datatracker turns it into email.
This is very similar to the way IESG ballots are handled. Second, the
reviewer sends his own email and give the datatracker the URL into the mail
archive.
Your description makes me wonder if this tooling could work with an API
related to the first approach.
I think it would, yes.

-Ekr
Post by Eric Rescorla
Russ
Warren Kumari
2018-06-13 22:33:12 UTC
Permalink
Post by Eric Rescorla
Post by Russ Housley
I would be interested in the experiences of other ADs.
I think Warren has spent a little time on it.
​Yup, Warren has spent some time on it, and there are some bits which he
loves, and some bits which are less than great.

The bits which he doesn't like (much of this may be user error):
1: I'd like there to be an easier way to ​find what drafts I need to review
- currently I go to the agenda, find a draft name, and then search for that
in Phabricator.

2: I do my reviews on an iPad, reading and annotating (using GoodReader)
the PDFs. This is partly because I like the layout / ease of annotation,
but also because I do a lot of this on planes and similar, where I have no
Internets. I'd like some magic way for this to integrate (note the "magic"
in the previous sentence).

3: I complete the review, and then copy and paste the review bits into the
Datatracker manually. Basically I am using Phabricator simply as a
stand-alone code review tool, without any of the integrations, etc.

None of these are Phabricator's fault, rather a: my not having figured out
a work flow yet, and b: the fact that it isn't really integrated with
Datatracker.

I'm also somewhat confused about how multiple people's reviews would
interact with each others -- it almost feels like each AD having their own
separate instance would make more sense (but this might be because, until
now, we've largely worked separately.

So, with the "bits that I don't like" out of the way:
I really like how easy it is to indicate where in a document the issue /
nits are.

As an example:
" Section: Abstract

This document discusses most common scenarios of connecting an

O: discusses most common
P: discusses the most common"


Copying and pasting the above is a huge waste of time (probably 25
seconds per), being able to scroll in Phabricator, click a line and
say "add 'the'" is much faster and more efficient.

There is also the issue at the end of the process -- if I've submitted
15 comments scattered throughout the document, it is a massive waste
of time to confirm that all have been addressed -- Phabricator makes
this much better.


So, some more / better tooling around this to:

1: provide links from the telechat agenda (or a clone of the telechat
page with links to Phabricator)

2: some way to autogenerate the ballot for users other than EKR.

3: encouraging people to use this instead of the current only emil system

all seem like they would be great.

W
Post by Eric Rescorla
I would like to know how easy it would be for Directorates and Review
Post by Russ Housley
Teams to leverage your work.
Understanding that would, I think, require understanding how those tools
work. I don't have a real sense of how those tools work. I know Tero has
some secdir stuff. Does everyone have their own thing?
No. The datatracker supports all of the reviews in a manner similar to
ballots. The reviewer two options. First, the reviewer puts text into a
frame on a datatracker page, and then the datatracker turns it into email..
This is very similar to the way IESG ballots are handled. Second, the
reviewer sends his own email and give the datatracker the URL into the mail
archive.
Your description makes me wonder if this tooling could work with an API
related to the first approach.
Russ
___________________________________________________________
Tools-discuss mailing list
https://www.ietf.org/mailman/listinfo/tools-discuss
Please report datatracker.ietf.org and mailarchive.ietf.org
bugs at http://tools.ietf.org/tools/ietfdb
Please report tools.ietf.org bugs at
http://tools.ietf.org/tools/issues
--
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
---maf
Eric Rescorla
2018-06-15 16:13:41 UTC
Permalink
Post by Warren Kumari
Post by Eric Rescorla
Post by Russ Housley
I would be interested in the experiences of other ADs.
I think Warren has spent a little time on it.
​Yup, Warren has spent some time on it, and there are some bits which he
loves, and some bits which are less than great.
1: I'd like there to be an easier way to ​find what drafts I need to
review - currently I go to the agenda, find a draft name, and then search
for that in Phabricator.
This I actually do have some tooling for: the system will automatically
scrape the agenda and assign things for you to review. It doesn't work
perfectly though, because it currently just adds them to your list and it
doesn't run them automatically.



2: I do my reviews on an iPad, reading and annotating (using GoodReader)
Post by Warren Kumari
the PDFs. This is partly because I like the layout / ease of annotation,
but also because I do a lot of this on planes and similar, where I have no
Internets. I'd like some magic way for this to integrate (note the "magic"
in the previous sentence).
This I don't know how to fix.


3: I complete the review, and then copy and paste the review bits into the
Post by Warren Kumari
Datatracker manually. Basically I am using Phabricator simply as a
stand-alone code review tool, without any of the integrations, etc.
This should actually just work, if you run my tools locally with iphab.py
ballot <whatever>. Have you tried?



None of these are Phabricator's fault, rather a: my not having figured out
Post by Warren Kumari
a work flow yet, and b: the fact that it isn't really integrated with
Datatracker.
I'm also somewhat confused about how multiple people's reviews would
interact with each others -- it almost feels like each AD having their own
separate instance would make more sense (but this might be because, until
now, we've largely worked separately.
Yeah, basically you just end up with a zillion comments from different
people. I wonder if that's good?



2: some way to autogenerate the ballot for users other than EKR.
As above, this should work now. Can you try?

-Ekr

3: encouraging people to use this instead of the current only emil system
Post by Warren Kumari
all seem like they would be great.
W
Post by Eric Rescorla
I would like to know how easy it would be for Directorates and Review
Post by Russ Housley
Teams to leverage your work.
Understanding that would, I think, require understanding how those tools
work. I don't have a real sense of how those tools work. I know Tero has
some secdir stuff. Does everyone have their own thing?
No. The datatracker supports all of the reviews in a manner similar to
ballots. The reviewer two options. First, the reviewer puts text into a
frame on a datatracker page, and then the datatracker turns it into email.
This is very similar to the way IESG ballots are handled. Second, the
reviewer sends his own email and give the datatracker the URL into the mail
archive.
Your description makes me wonder if this tooling could work with an API
related to the first approach.
Russ
___________________________________________________________
Tools-discuss mailing list
https://www.ietf.org/mailman/listinfo/tools-discuss
Please report datatracker.ietf.org and mailarchive.ietf.org
bugs at http://tools.ietf.org/tools/ietfdb
Please report tools.ietf.org bugs at
http://tools.ietf.org/tools/issues
--
I don't think the execution is relevant when it was obviously a bad idea
in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
---maf
Russ Housley
2018-06-15 17:20:07 UTC
Permalink
Post by Russ Housley
I would be interested in the experiences of other ADs.
I think Warren has spent a little time on it.
​Yup, Warren has spent some time on it, and there are some bits which he loves, and some bits which are less than great.
1: I'd like there to be an easier way to ​find what drafts I need to review - currently I go to the agenda, find a draft name, and then search for that in Phabricator.
This I actually do have some tooling for: the system will automatically scrape the agenda and assign things for you to review. It doesn't work perfectly though, because it currently just adds them to your list and it doesn't run them automatically.
The use of the word "scrape" makes me wonder if you are using the JSON form of the telechat agenda: http://datatracker.ietf.org/iesg/agenda/agenda.json <http://datatracker.ietf.org/iesg/agenda/agenda.json>

Russ
Eric Rescorla
2018-06-15 17:21:48 UTC
Permalink
Post by Eric Rescorla
Post by Warren Kumari
Post by Eric Rescorla
Post by Russ Housley
I would be interested in the experiences of other ADs.
I think Warren has spent a little time on it.
​Yup, Warren has spent some time on it, and there are some bits which he
loves, and some bits which are less than great.
1: I'd like there to be an easier way to ​find what drafts I need to
review - currently I go to the agenda, find a draft name, and then search
for that in Phabricator.
This I actually do have some tooling for: the system will automatically
scrape the agenda and assign things for you to review. It doesn't work
perfectly though, because it currently just adds them to your list and it
doesn't run them automatically.
The use of the word "scrape" makes me wonder if you are using the JSON
form of the telechat agenda: http://datatracker.
ietf.org/iesg/agenda/agenda.json
Yes, I use the JSON form.

-Ekr
Post by Eric Rescorla
Russ
Warren Kumari
2018-06-15 17:56:18 UTC
Permalink
Post by Eric Rescorla
Post by Warren Kumari
Post by Eric Rescorla
Post by Russ Housley
I would be interested in the experiences of other ADs.
I think Warren has spent a little time on it.
​Yup, Warren has spent some time on it, and there are some bits which he
loves, and some bits which are less than great.
1: I'd like there to be an easier way to ​find what drafts I need to
review - currently I go to the agenda, find a draft name, and then search
for that in Phabricator.
This I actually do have some tooling for: the system will automatically
scrape the agenda and assign things for you to review. It doesn't work
perfectly though, because it currently just adds them to your list and it
doesn't run them automatically.
2: I do my reviews on an iPad, reading and annotating (using GoodReader)
Post by Warren Kumari
the PDFs. This is partly because I like the layout / ease of annotation,
but also because I do a lot of this on planes and similar, where I have no
Internets. I'd like some magic way for this to integrate (note the "magic"
in the previous sentence).
This I don't know how to fix.
3: I complete the review, and then copy and paste the review bits into the
Post by Warren Kumari
Datatracker manually. Basically I am using Phabricator simply as a
stand-alone code review tool, without any of the integrations, etc.
This should actually just work, if you run my tools locally with iphab.py
ballot <whatever>. Have you tried?
​Nope. So far all I've been doing is using the
https://mozphab-ietf.devsvcdev.mozaws.net/
<https://mozphab-ietf.devsvcdev.mozaws.net/D5132> instance, and have not
yet tried any of your tools -- they sound like they accomplish much of what
I'm looking for.



W
Post by Eric Rescorla
None of these are Phabricator's fault, rather a: my not having figured
Post by Warren Kumari
out a work flow yet, and b: the fact that it isn't really integrated with
Datatracker.
I'm also somewhat confused about how multiple people's reviews would
interact with each others -- it almost feels like each AD having their own
separate instance would make more sense (but this might be because, until
now, we've largely worked separately.
Yeah, basically you just end up with a zillion comments from different
people. I wonder if that's good?
2: some way to autogenerate the ballot for users other than EKR.
As above, this should work now. Can you try?
-Ekr
3: encouraging people to use this instead of the current only emil system
Post by Warren Kumari
all seem like they would be great.
W
Post by Eric Rescorla
I would like to know how easy it would be for Directorates and Review
Post by Russ Housley
Teams to leverage your work.
Understanding that would, I think, require understanding how those tools
work. I don't have a real sense of how those tools work. I know Tero has
some secdir stuff. Does everyone have their own thing?
No. The datatracker supports all of the reviews in a manner similar to
ballots. The reviewer two options. First, the reviewer puts text into a
frame on a datatracker page, and then the datatracker turns it into email.
This is very similar to the way IESG ballots are handled. Second, the
reviewer sends his own email and give the datatracker the URL into the mail
archive.
Your description makes me wonder if this tooling could work with an API
related to the first approach.
Russ
___________________________________________________________
Tools-discuss mailing list
https://www.ietf.org/mailman/listinfo/tools-discuss
Please report datatracker.ietf.org and mailarchive.ietf.org
bugs at http://tools.ietf.org/tools/ietfdb
Please report tools.ietf.org bugs at
http://tools.ietf.org/tools/issues
--
I don't think the execution is relevant when it was obviously a bad idea
in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
---maf
--
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
---maf
Loading...