Technical Papers: Reviewers Instructions & Ethics of Review

TECHNICAL PAPERS: REVIEWER INSTRUCTIONS

Thank you for agreeing to review a paper for SIGGRAPH. Reviews have a direct and important impact on the quality of the most important conference in computer graphics. Reviews also help the computer graphics community as a whole to improve the quality of its research. To access review materials and the web review forms, please login to the online submission system using your standard submission account.

Journal Papers and Conference Papers
The Technical Papers program has two integrated paper tracks: Journal (ACM Transactions on Graphics) and Conference.

Journal Papers are expected to satisfy the ACM TOG criteria of excellence established over the years, and contain novel, well validated, and comprehensively described ideas which advance the state of the art. Papers accepted to the Journal track will be published in the SIGGRAPH 2023 issue of ACM Transactions on Graphics (TOG).

Conference Papers are expected to contain novel ideas and similarly advance the state of the art but can be less complete or comprehensive (e.g., have less extensive validation or formal verification). As such, the length of papers accepted to this track is capped as outlined in the call for papers. Papers accepted to this track will be published in SIGGRAPH 2023 Conference Proceedings.

In particular, when assessing a submission’s suitability for the Conference track, reviewers should be open to works that introduce new interesting problems, novel solutions to existing problems that are likely to improve on the state of the art, or original methodologies, even if the proposed methods are not fully validated, lack comprehensive ablations, and are only partially compared against alternative options.

Journal-only submissions can only be accepted to the Journal track and need to satisfy this track’s criteria to be accepted. Dual-track submissions can be accepted either as Conference or Journal Papers. We recommend that reviewers first decide if the paper’s overall contribution (in terms of novelty and advancement of the state of the art) merits acceptance to the SIGGRAPH Technical Papers program and then, if the answer is positive (be it weakly or strongly), evaluate the paper’s completeness and degree of comprehensiveness to determine the track the paper should be accepted to.

What to Look For
Be fair, promote fresh scientific ideas, and look for what is good or stimulating about the paper. In the words of former chair Tony DeRose:

“An important factor in evaluating a paper is the degree to which it will inspire follow-on research. After all, we are looking for papers that will propel the field forward as rapidly and vibrantly as possible, while at the same time keeping us well-grounded academically. Finding the right balance obviously isn’t easy, but doing so will ensure that SIGGRAPH remains a breeding ground for significant new areas of research.”

Minor flaws can be corrected and should not be a reason to reject a paper. Each accepted paper should, however, be technically sound and make a substantial contribution. And remember that one of the reasons we introduced the Conference track is to allow for less polished but novel ideas to be published and published fast.

Ethics and Professionalism
Please read the Ethics of Review section below. It is extremely important that we uphold our reputation for treating ideas confidently and professionally. By accepting a paper for review, you are committing to review all materials submitted in the approved formats: PDF for documents; for videos, we strongly encourage MP4 and PNG or JPG for images. If you are not willing to make this guarantee, please recuse yourself from reviewing. You also are expected to make a reasonable effort to review materials in non-approved formats, but you are not under the same absolute obligation to do so.

Anonymity
We use double-blind reviewing. Authors are expected to strictly adhere to the anonymity policy. Authors were asked to take all reasonable efforts to hide their identities in the submitted materials, including not listing their names or affiliations and omitting acknowledgements. This information will, of course, be included in the published version should the paper be accepted.

One area where anonymity can affect your evaluation of the submitted paper is if the paper builds on ideas that were previously available in some prepublication form, for instance as a SIGGRAPH Talk, or as a technical report, a thesis, or an arXiv publication. The authors were instructed to avoid plagiarism and to cite their sources if they used ideas from someone else’s work. Authors do not cite their own prepublications of largely overlapping works to avoid revealing their identity (see Submission and Anonymity Policies). Reviewers should not deliberately try to discover the identity of the authors. Immediately following the submission deadline, plagiarism detection software will be run on all submissions, and the sorting team will further ascertain the originality of submissions during the sort when needed. If you as a reviewer have a substantial concern regarding the originality of the submission, please contact the Technical Papers admin [link to: papersadmin@siggraph.org] prior to submitting your review. These concerns will be dealt with by the chair/advisory board, who will determine if the concern is valid and how to address it. They will inform you about the outcome.

Due to the number of new works posted on arXiv and other non-peer-reviewed websites on a daily basis, it is increasingly likely that you might find online reports that are highly relevant to the submitted work and that the authors were unaware of. The existence of these non-peer-reviewed material should not negatively affect your review of the submission. In particular, note that authors are allowed, but not required, to cite them as concurrent work without the burden of having to detail how their work compares or differs from these prepublications. If such preprints are not cited, authors of conditionally accepted papers can be made aware of these publications and asked to cite them in their final revision.

Be Specific
Please be specific and detailed in your reviews. In the discussion of related work and references, simply saying “this is well known” or “this has been common practice in the industry for years” is not sufficient nor acceptable. Please cite specific publications or public disclosures of techniques, and if these do not exist or you cannot find them, entertain the possibility that the contribution is indeed novel. The explanation section is one of the most important parts of your review. Your discussion, sometimes more than your score, will help the Technical Papers Committee decide which papers to accept, so please be thorough. Your reviews will be returned to the authors, so you should include any specific feedback on ways the authors can improve their papers.

Online Review
To access electronically submitted papers and supplemental material, log into the submission portal the same way you would to make a submission to SIGGRAPH, using your existing online submission account. Once you have logged in, access the “Submissions & Reviews” portion of the site at the top of the screen, and use the links in your “To-Do List.” If you have any questions or problems with the online review system, use the “Contact Support” link at the bottom of the page.

ACM and Eurographics Digital Libraries
The Eurographics library is now open access, so you can access it freely during the review process.

ACM has provided full access to their digital library (ACM Digital Library) for SIGGRAPH 2023 paper reviewer usage, effective 2 February–3 April 2023. You are encouraged to make full use of these resources.

You must log in to access the full text of an article. A username and password will be made available via our electronic review system to each reviewer.

Timely Reviews
The deadline for completed reviews is 1 March 2023. Adhering to this deadline is extremely important. In particular, the author rebuttal process starts immediately after the review deadline, where authors must be able to see the complete set of reviews.

When You Are Done
In previous years, these guidelines said, “After the review process, destroy all copies of papers and videos that are not returned to the senior reviewer and erase any implementations you have written to evaluate the ideas in the papers, as well as any results of those implementations.”

However, in 2012, SIGGRAPH introduced a new process for revised papers that were rejected from a previous SIGGRAPH conference, where the authors can choose to release the previous reviewers’ names so that the same reviewers can be reassigned. Therefore, there is a chance that you will be asked in the future to review such a resubmission and may need your notes, marked manuscripts, or implementations. You may keep them if necessary, but please be careful to insulate the ideas you learned from the review from your own research and from your colleagues and students. Also, please be aware that your reviews may be perused by future SIGGRAPH reviewers.

TECHNICAL PAPERS: ETHICS OF REVIEW

Protect Ideas
As a reviewer for SIGGRAPH, you have the responsibility to protect the confidentiality of the ideas represented in the papers you review. SIGGRAPH submissions are by their nature not published documents. The work is considered new and proprietary by the authors; otherwise, they would not have submitted it.

Of course, authors ultimately intend to publish their work; however, many of the submitted papers will end up being rejected from this year’s conference. Thus, it is likely that the paper you have in your hands will be refined further and submitted to another journal or conference or even to SIGGRAPH next year. Oftentimes, the work is considered confidential by the author’s employers. These organizations do not consider sending a paper to SIGGRAPH for review to constitute a public disclosure. Consequently, you must abide by a few simple rules to protect the ideas in the submissions you receive:

• Do not show the paper to anyone else, including colleagues or students, unless you have asked them to help with your review. See the Review Process section of the Technical Papers FAQ for more details on how to properly include a colleague or student in the review process.
• Do not show videos or other materials to non-reviewers.
• Do not use ideas from papers you review to develop new ones.
• Due to the possibility of paper resubmission with reviewer continuity, you may want to keep your notes, marked manuscripts, videos, or implementations. Please keep those strictly confidential.

Avoid Conflict of Interest
As a reviewer of a SIGGRAPH paper, you have power over the reviewing process. It is important for you to stay clear of any conflict of interest. There should be absolutely no question about the impartiality of reviews. Thus, if you are assigned a paper for which your review would create a possible conflict of interest, you should return the paper immediately and not submit a review. If you discover a conflict of interest after starting the review, you must recuse yourself from the assignment as soon as you discover it. Conflicts of interest include (but are not limited to) situations in which:

• You work at the same institution as one of the authors.
• You have been directly involved in the work and will be receiving credit in some way. For instance, if you are a member of the author’s thesis committee, and the paper is about their thesis work, then you were involved.
• You suspect that others might see a conflict of interest in your involvement. For example, even though Microsoft Research in Seattle and Beijing are in some ways more distant than Berkeley and MIT, there is likely to be a perception that they are “both Microsoft,” so folks from one should not review papers from the other.
• You have collaborated with one of the authors in the past three years. Collaboration is usually defined as having written a paper or grant proposal together, although you should use your judgment. For instance, being co-presenters in a course, co-authors of a survey paper, or co-chairs in a recent conference does not in itself lead to a conflict of interest.
• You were the M.S./Ph.D. advisor or advisee of one of the authors. This represents a lifetime conflict of interest.
• You have unpublished or unreleased work that would get scooped by the current submission because it tackles the same problem using a similar approach. If asked to review a paper that can create such a cross-reviewing conflict, please turn down the request and immediately inform the appropriate Technical Papers Committee member or the papers admin.

The submission review process strives to prevent all identifiable conflicts (co-authorship, same affiliation, etc.) but cannot address situations such as scooping or others which are not testable automatically based on public knowledge. If you recognize the work or the author(s) and feel it could present or be seen as presenting a conflict of interest, notify the senior reviewer as soon as possible so they can find someone else to review it.

Be Serious
The paper publishing business at SIGGRAPH is serious — careers and reputations, as well as academic tenure decisions, often hinge on these publications, and thus on your review. As an example of the perception of the quality of our review process, patent infringement cases have discussed whether something was considered novel enough to be published at SIGGRAPH. You are responsible for upholding this reputation.

This does not mean that we cannot have any fun in the paper sessions. But it does mean that we have a responsibility to be serious in the reviewing process. You should make an effort to do a solid and constructive review. This is obvious, but one of the complaints we have heard about the SIGGRAPH review process is that some reviews can be so sketchy that it looks like the reviewer did not even seem to take the time to read the paper carefully. A casual or flippant review of a paper on which the author has spent considerable time is not appropriate and certainly not professional. In the long run, casual reviewing is a very damaging attack on the SIGGRAPH conference. There is no dishonor in being too busy to do a good review, to realize that your competencies are not a good fit for a paper, or to realize that you have overcommitted yourself and cannot review all the papers you agreed to review. But it is a big mistake to take on too much and then not back out early enough to allow recovery. If you cannot write a high-quality review, give the paper back and say so. But please, do it early so that a suitable replacement can be found.

Be Professional
Belittling or sarcastic comments have no place in the reviewing process. The most valuable comments in a review are those that help the authors understand the shortcomings of their work and how they might improve it. Be respectful and carefully explain why you like or dislike a submission so the authors can learn from your expertise.

Remain Anonymous
All reviewers are expected to maintain anonymity forever. In particular, it is never appropriate for reviewers to reveal themselves to the authors of an accepted paper, as this could be perceived as an attempt to curry favor. Requesting citations primarily to one’s own work may thwart anonymity, so it should be carefully considered.

In Summary
Adherence to ethics makes the whole reviewing process indubitably more complicated and sometimes less efficient. But convenience, efficiency, and expediency are not good reasons to contravene ethics. It is precisely at those times when it would be easier or more efficient to bend the rules that it is most important to do the right thing. Ultimately, spending that energy and time is an investment in the long-term health of the Technical Papers sessions, the conference, and the entire community of computer graphics researchers.