Skip to content

Latest commit

 

History

History
155 lines (115 loc) · 9.82 KB

README.md

File metadata and controls

155 lines (115 loc) · 9.82 KB

Mapping evaluation workflow

The flowchart below focuses on the evaluation part of our process in detail. See mapping-evaluation-workflow.md for a more condensed flowchart.

{% hint style="info" %} Video overview: Two minute overview of our 6-step evaluation process. {% embed url="https://www.youtube.com/watch?v=ZCSeAmzMB50" %} {% endembed %}

Interactive presentation: For a visual walk-through of our evaluation process, see our step-by-step slide presentation (use arrow keys to navigate). {% endhint %}

graph TD
  S["Author (A) submits work (W)"] --> MP[M and FS prioritize work]
  NBER[Work enters 'prestige' archive ] --> MP 
  MP --> IA
MP --> WE[Work to evaluate]
  WE ..-  |M/FS may add| BEN["'Bespoke Evaluation Notes' (BEN)"]


MS["Managers (M) and field specialists (FS) <br/> select & prioritize work, contact authors"] --> 
AP[Author: OK]
AP --> WE
  MS --> AN[Author: NO] --> DE[Don't evaluate]
  MS --> ER[A: Embargo please ]
  ER ..- SD[See discussion]
  WE -->|Managers select| EM(("Evaluation <br/> Manager (EM)"))
  EM .-> |EM may add tips to| BEN 

IA[Inform A's, <br/>asks engagement q's]
  IA .-> AE[A's may engage <br/> feedback of interest <br/> link updated version] 

EM -->|EM selects & <br/> contacts| UJEV(UJ evaluators) 
UJEV --> |Accepts eval. assignment| EA((EV = Evaluator who <br/> accepts))
BEN ..-|incorporated into| UJT[Template, <br/> guidelines for EV]
AE ..-|may feed into| UJT 
EA --> |EM schedules with EV| EVC[All EV's <br/> complete evals] --> EC[\Eval. content\]
EVC --> EVR[\Eval. ratings\] 
UJT --> |EM shares with| EA 


EC -->|EM| EVPA[Passed <br/> anonymously <br/> to A]
EVR --> |EM| EVPA[Passed <br/> anonymously <br/> to A]
EVPA .-> ARE[\Authors' response\]
EVPA .-> TW[2 weeks <br/> w/o response]
EVPA .-> TB[A: 'too busy <br/> to respond']

TW ..-|skip to| EMS
TB ..-|skip to| EMS

EVC .->|EV chooses|EVA[Anonymity]
EVC .->|EV chooses|EVS[to sign eval.]

ARE .->|EM considers in| EMS[\EM's summary\] 
EVC ..- |EM considers in| EMS

EMS --> UJO[/Unjournal <br/> published output w/ DOIs/] 
EC --> UJO
ARE --> UJO 
EVR --> |Prominently <br/> reported in| UJO
EVS .-> |EV name| UJO

EVR .->|After UJ output|UJRD[/Ratings/predictions <br/> database <br/>/]

UJO -->|EM|UJOI[Inform authors and Evaluators]
UJO --> UJPUB[Publicize, follow availability, <br/> bibliometrics.]
Loading

Describing key steps in the flowchart

(Section updated 1 August 2023)

  1. Submission/selection (multiple routes)
    1. Author (A) submits work (W), creates new submission (submits a URL and DOI), through our platform or informally.
      • Author (or someone on their behalf) can complete a submission form; this includes a potential "request for embargo" or other special treatment.
    2. Managers and field specialists select work (or the project is submitted independently of authors) and the management team agrees to prioritize it.
      • For either of these cases (1 or 2), authors are asked for permission.
    3. Alternate Direct Evaluation track: "Work enters prestige archive" (NBER, CEPR, and some other cases).
      • Managers inform and consult the authors but permission is not needed1. (Particularly relevant: we confirm with author that we have the latest updated version of the research.)
  2. Prioritization
    • Following author submission ...
    • Following direct evaluation selection...
    • If requested (in either case), M decides whether to grant embargo or other special treatment, notes this, and informs authors.
  3. M assigns3 an Evaluation Manager (EM – typically part of our management team or advisory board) to selected project.
  4. EM invites evaluators (aka "reviewers") and shares the paper to be evaluated along with (optionally) a brief summary of why The Unjournal thinks it's relevant, and what we are asking.
    • Potential evaluators are given full access to (almost) all information submitted by the author and M, and notified of any embargo or special treatment granted.
    • EM may make special requests to the evaluator as part of a management policy (e.g., "signed/unsigned evaluation only," short deadlines, extra incentives as part of an agreed policy, etc.).
    • EM (also4, optionally) may add "evaluation suggestions" to share with the evaluators.
  5. Evaluator accepts or declines the invitation to review, and if the former, agrees on a deadline (or asks for an extension).
    • If the evaluator accepts, the EM shares full guidelines/evaluation template and specific suggestions with the evaluator.
  6. Evaluator completes an evaluation form5.
  7. Evaluator submits evaluation including numeric ratings and predictions, plus "CI's" for these.
    • Possible addition (future plan): Reviewer asks for minor revisions and corrections; see "How revisions might be folded in..." in the fold below.
  8. EM collates all evaluations/reviews, shares these with Author(s).
    • Evaluator must be very careful not to share evaluators' identities at this point.
      • This includes caution to avoid accidentally-identifying information, especially where evaluators chose anonymity6.
      • Even if evaluators chose to "sign their evaluation," their identity should not be disclosed to authors at this point. However, evaluators are told they can reach out to the authors if they desire.7
    • Evaluations are shared with the authors as a separate doc, set of docs, file, or space; which the evaluators do not have automatic access to8. (Going forward, this will be automated.)
    • It is made clear to authors that their responses will be published (and given a DOI, when possible).
  9. Author(s) read(s) evaluations, given two working weeks to submit responses.
    • If there is an embargo, there is more time to do this, of course.
  10. EM creates evaluation summary and "EM comments."
  11. EM or UJ team publishes each element on our PubPub space as a separate "pub" with a DOI for each (unless embargoed):
    1. Summary and EM comments
      • With a prominent section for the "ratings data tables"
    2. Each evaluation, with summarized ratings at the top
    3. The author response
      • All of the above are linked in a particular way, with particular settings; see notes
  12. Authors and evaluators are informed once elements are on PubPub; next steps include promotion, checking bibliometrics, etc.
  13. ("Ratings and predictions data" to enter an additional public database.)

Note that we intend to automate and integrate many of the process into an editorial-management-like system in PubPub.

Consideration for the future: enabling "minor revisions"

In our current (8 Feb 2023 pilot) phase, we have the evaluators consider the paper "as is," frozen at a certain date, with no room for revisions. The authors can, of course, revise the paper on their own and even pursue an updated Unjournal review; we would like to include links to the "permanently updated version" in the Unjournal evaluation space.

After the pilot, we may consider making minor revisions part of the evaluation process. This may add substantial value to the papers and process, especially where evaluators identify straightforward and easily-implementable improvements.

How revisions might be folded into the above flow

If "minor revisions" are requested:

  • ... the author has four (4) weeks (strict) to make revisions if they want to, submit a new linked manuscript, and also submit their response to the evaluation.
  • Optional: Reviewers can comment on any minor revisions and adjust their rating.

Why would we (potentially) consider only minor revisions?

We don't want to replicate the slow and inefficient processes of the traditional system. Essentially, we want evaluators to give a report and rating as the paper stands.

We also want to encourage papers as permanent-beta projects. The authors can improve it, if they like, and resubmit it for a new evaluation.

Footnotes

  1. Not at NBER and mainly not at CEPR, but see discussion. ↩

  2. This might occur through the same "submission form" that authors complete. ↩

  3. Here, this should be someone on the management team. A field specialist does not need to be the evaluation manager for the paper they recommend, but if not, they should ask someone on The Unjournal Management Team to help find an evaluation manager. ↩

  4. Perhaps in addition to any added by the Manager. ↩

  5. At present this is an out-link to a Googe Doc; we aim to embed this in an editorial-management-like system in PubPub. ↩

  6. Where evaluators chose anonymity, none of their evaluation content should be linked to their real names or identity by design. Still, this should be double-checked. ↩

  7. The evaluators can also ask us to disclose their identity and contact info to the authors at this point, to save them the hassle ↩

  8. Make this clear to authors. Authors can reach out to evaluators and share any of this if they wish, but they are not required to do so. ↩