Skip to main content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Stony Brook University

Systematic Reviews: Weeding/Selecting Articles

A guide to conducting systematic reviews.

Reviewing/Weeding Abstracts

At least 2 independent reviewers must read and separately make a decision on each abstract. In some cases, if a larger team is available, the results can be divided up between pairs of reviewers. Using a software/website specific for this purpose is helpful. Some are freely available online, like Rayyan. Others are paid software, like Covidence. Covidence does give each individual one free review, so researchers should consider this option.

Each reviewer must individually, without influence of another, decide if each abstract should be included (Yes), excluded (NO), or needs further examination of full-text or discussion (Maybe).

The decisions of the paired reviewers should be compared when completed.

  • All double "No" articles are excluded at this point. (COUNT THESE! If possible, record the reasons for exclusions, i.e., absent outcomes, wrong population, etc.)
  • All double "Yes" articles are put aside in an inclusion category (COUNT THESE!)
  • All double "Maybe" articles are carried through to the full-text round.
  • All disagreements are discussion and decisions made to place the articles in one of the preceeding three categories.
  • Inter-rater reliability should be recorded.

Full-Text Review

Pull the Full-Text of all "Yes" and "Maybe" category articles.

Again, two independent reviewers should read them and make separate inclusion/exclusion decisions. Reiteratively, those decisions are recorded, compared, and disagreements resolved.

At this point, there should be agreement on the full sets of included and excluded articles. In some cases, an additional expert or team member may be needed to resolve any serious disagreements.

Counts and measures of inter-rater reliability should be kept for both stages.