Specialist Role Descriptions

Each student in a team is assigned a role to play to critique a paper. Each role is of equal importance. Details and suggested preparations for each role are detailed below. Students are encouraged to read through each role and request clarification if needed. To foster individuality and diverse student backgrounds, students are offered to give their preferences to play each role. However, the ultimate decision of assigning roles belongs to the instructor. It is plausible that students are not assigned their desired roles. If this is the case, students are encouraged to take this opportunity to step outside of their comfort zone and discover their potential. It is noted that not all roles would be assigned for all papers. Roles are assigned to a paper based on its suitability to the context and content of the paper. It is also noted that depending on the number of students in a course session (sessions A and B), one student might present multiple times. If this is the case, the student would be assigned to a different paper, team, and role each time she or he is presenting.

Presenter

The role of the presenter is to clearly communicate the main ideas of the research paper. Think of this as a visual version of your paper. You would need to create the main presentation, describing the motivation, problem definition, method, and experimental findings of this paper. Keep your presentation between 10-15 mins.

Divide your presentation into the following sections (Can use the following as a guide to creating effective presentations):

1. Introduction/Overview/Hook: Focus on the motivation behind the research. Why is it necessary to solve the problem addressed in the research? Relevance of the problem with respect to the time in which the paper was written. How does the problem fit into the larger picture? What are its applications? What makes the problem nontrivial?
2. Theoretical Framework/Research Question: Here, you should focus on defining your problem statement. Make sure you do not state the problem as is from your research paper. Simplify the problem statement as per your understanding. No matter how difficult and technical the problem is; it can usually be described succinctly and accurately in less than five minutes. If the audience doesn’t understand the problem being attacked, then they won’t understand the rest of your talk.
3. Background/Literature Review: Discuss the previous work done in the field and explain how your paper is different from all the research done in the past.
4. Methodology/Case Selection: Here, you talk about any kind of experimentation conducted to get some results. Try using visuals or flowcharts if possible to make the methodology easier to explain. Avoid adding too many complex tables. If your experiment involves a physical setup, you can try to include digital photos to understand the setup better.
5. Discussion of Data/Results/Analysis: Start by discussing key results. Try to avoid too much jargon. Focus on any unexpected results and explain the reasoning or significance behind them. If possible, try to include conclusions at the bottom of each result slide to give a better understanding of why the result is important.
6. Conclusion: Use key takeaways from the previous slides and combine them to form a coherent synopsis. Be honest and open about any uncertainties or areas of improvement in the research. This may also help avoid any antagonistic questions later.

Basic tips for effective presentations:

1. Don’t just read your presentation. Engage with the audience as much as possible.
2. Avoid long full sentences or making your slides too wordy.
3. Use more images to explain your processes better.
4. Use minimum jargon.
5. You can use humor/memes to keep the audience’s attention but do not make humor the major focus of your presentation.
6. Maintain Eye contact with your audience.
7. Speak clearly and with sufficient volume.
8. Make sure you use large fonts and clear contrasts to make your slides readable from afar.

Reviewer

Complete a full critical but not necessarily a negative review of the paper. Follow the guidelines for NeurIPS reviewers (under “Review Content”), taking note of the example reviews included therein. In particular, please answer questions 1 to 10 under “Review Content,” including assigning an overall score. Read the Neurips reviewer guidelines for detailed instructions.

Archaeologist

The aim of this role is to show the originality and relevance of your research with respect to past methodology. Determine where this paper sits in the context of previous and subsequent work.

Find and report on one prior paper that substantially influenced the current paper and one newer paper that cites this current paper.

General guidance to creating a good literature review:

1. Generate a list of references:
1. Make a preliminary list of statistical literature that is relevant to your research topic. Discuss within the team to identify the older paper that influences the current paper and a newer paper that cites your current paper. You can also keep multiple papers handy in order to identify the best 2 papers that fit the above criteria.
2. Now you have to ‘procure’ (get copies of) the literature on the list. How much time this takes can vary greatly. Some books and journals may be available from your library or from the internet. You can get in touch with the professors if you cannot find the journal you’re looking for.
2. Reading the literature:
1. Once you have a preliminary list of references, you now have to read this material. This process is ‘time-consuming’ (takes a lot of time) because you may have to read a large amount of statistical research depending on how many papers you select.
2. You must realize that not every reference will contain material that is relevant to your research problem. This is a necessary part of the process: Keep what is relevant and ignore what is irrelevant. It is better to read something that is not directly related to your dissertation than miss a reference that is important and relevant to the dissertation.
3. While you are reading, keep notes about the assumptions made and the important results. Good notes help when you begin to write the literature review.
4. Try to determine the methodology used by the author. This will be helpful when you want to describe how your research is related to prior research. For example, is your research an extension or modification of this author’s research?
5. Remember that you are not just summarizing. For each reference you read, think critically about the content. That is, ask the following questions (You can outline your summary across the following questions):
   1. What is the author trying to say?
   2. Is it relevant to my research? If yes, then why is it relevant?
   3. What is original about the methodology used by the author?

Find out as much as you can about one of the paper’s authors. This should help you determine how qualified they are to write the paper and if their work is credible enough to be considered.

• Here are some questions which you may consider.
• What did they study? What are their qualifications in the field?
• Where have they worked?
• What previous projects might have led to working on this one? If they do, where were they published?
• What do you think motivated them to work on this project?
• Are they an expert in the field?
• What institutions have they been affiliated with?
• Does the author have any affiliations (e.g., any sources of funding) which make the research they conducted biased? You may also look into who cites the author’s research. Are the same people citing the author repeatedly? If so, does the author also cite them back repeatedly? Are the citations starting to look circular or suspicious?

You are required to visit a wide variety of websites and sources to collect this information. Sources to consider are the author’s personal webpage, their institutional page, publication websites (e.g., Journal and Conference websites), and research search engines (e.g., Google Scholar or ResearchGate).

After doing so, you must be able to articulate what makes the author qualified to write this paper and why their research is credible or make a case against it.

Researcher

This is where you can get your creative juices flowing. Propose an imaginary follow-up project – not just based on the current but only possible due to the existence and success of the current paper.

Guidance of format to be used for project proposal:

1. Introduction/Problem Statement:
1. Explain the issue you are trying to solve through your project.
2. Explain how the paper has guided your motivation to create the project.
2. Background:
1. Summarize what is already known about the field. Include a summary of the basic background information on the topic gleaned from your literature review (content written by archaeologists could help).
2. Discuss several critical studies that have already been done in this area.
3. Point out why these background studies are insufficient and how your project solves the problem (the reviewer can help point out flaws within the study).
3. Method and Design (The methodology assumes participants in a study, but you can change it depending on your methodology):
1. Method:
1. Describe the general methodology you choose for your study in order to test your hypothesis(es)
2. Explain why this method is the best for your purposes.
3. Participants: Who would you test and why?
4. Describe the sample you would test and explain why you have chosen this sample. Include age, language background, and socio-economic information if relevant to the design.
5. Are there any participants you would exclude? Why, why not?
2. Procedure:
1. Explain the entire implementation process. Include visuals or flowcharts if possible.
2. Explain what the participants in your experiment do.
3. Analysis:
1. How will you analyze the results?
2. What kind of results would confirm your hypothesis?
3. What kind of results would disconfirm your hypothesis?
4. Significance and Conclusion:
• Discuss, in general, how your proposed experiment would lead to a significant improvement over the original studies and how it would benefit the field. (In other words, why should someone care? If you were applying for money to do this, why would someone fund you? If you wanted to publish your results, why would they be interesting?)
5. References:
• Cite all resources you referred to for your experiment.

Industry Expert

You must discuss how the method discussed in the paper can apply to one real-world scenario and propose a new application. Put yourself in the shoes of an employee proposing this application to their boss. Establish context (such as the industry you are working in, current solution and resources, etc.) and discuss the use-case, road map, and business advantage of utilizing the method for your application.

1. Identify the Problem:
1. Describe a problem that you could solve using some aspect of the paper.
2. Propose a viable and realistic solution to the problem that you could solve using some aspect of the paper.
2. Establish Context:
1. Imagine yourself to be working for an employer and describe the context of the situation as specifically as possible.
2. In particular, you may look into a specific company’s:
1. The industry of your employer with regards to your solution.
2. Current Solution to a problem.
3. Resources and funding available to solve this problem.
4. Data the company likely possesses.
5. Business goals and objectives.
You may make reasonable assumptions if any or all of this data is publicly unavailable.
3. Detail the Steps and Roadmap for application of your solution:
1. Go over any Background work required before implementation of your solution.
2. Systemize any Data and Resource requirements that need to be collected and established.
3. Plan out the steps you would follow to implement your solution to completion.
4. Discuss the impact of your solution:
1. Make a conniving argument (as if you are making one to an employer) as to why your solution is worth investing in.
2. Discuss advantages and how your solution is superior to the current solution.
3. Also, discuss any possible disadvantages, drawbacks, and costs relating to the application of your solution.

Hacker

Put yourself in the shoes of the technician or coder of the paper and implement a small part of the paper. This may be done on a small dataset or toy problem. You are allowed to seek out any open source implementation uploaded by the authors of the paper or a third party to assist you. We do not expect you to code the entire algorithm proposed in the paper. You may only build a small prototype that implements a core method or idea that you have identified in the paper. You must explain why you believe this to be the main focus of the paper.

You can use existing open source implementation as the backbone of your code (e.g., model building, data loading, training loop, etc.), however, you are not allowed to simply download and run an existing implementation - you should implement at least a core method from the paper. Make sure to reference any code that you have used for your implementation, even if this code is from the authors of the paper and they have a direct hyperlink from the paper to it. You may also use any libraries you find necessary, except for libraries containing functions or classes that directly solve the problem.

If the problem discussed in the paper is too complex, you may distill or simplify it while still maintaining the essence of the method. For example, if the paper focuses on three-dimensional data and the main focus of the paper is not affected by the dimensionality of the data, you may implement the code on two-dimensional data.

Another example would be while coding a deep learning paper, focusing on the main aspect discussed in the paper. If the main focus is the model, you may use activations and loss functions that you understand rather than trying to match the paper exactly. If the main focus is the loss function, try coding the loss function while using simple models. We do not expect you to achieve the same results as the paper.

Please make it clear as to which part of the paper you have identified to code. Explain why you believe this to be the main focus of interest in the paper. Explain how you went about implementing this part of the paper. In your code, identify which parts of the code you have written yourself and which you have not.

Social Impact Assessor

While (hopefully) all authors of any given research paper would want their research to have a net positive impact on this world; this may not always be the case. Many a time, the Computer Science Research Community will frame their innovations or advancements, such as Deepfakes or Driverless cars, as having a purely positive impact without giving any credence to possible negative impacts like identity theft and loss of jobs. This is akin to the Medical Research Community discussing the positives of a new treatment or drug without relaying any information about side effects. Another thing not considered is the possible future impacts of technology. It is hard to imagine that Alexander Graham Bell would know that his invention would someday lead to 1.6 million accidents in 2021 due to drivers distracted by their cell phones (especially since cars would be invented a decade later). It is your job to investigate and assess the social impact of the research. You may accomplish this by following these guidelines.

1. Discuss Self-Assessment of Social Impact:
1. Identify the paper’s self-assessed impact on this world.
2. Determine if this self-assessment is reasonable or not.
2. Peer-Review:
1. Conduct a peer review of the broader social impacts of the paper that may not have been considered by the authors.
2. Determine how the technology proposed in the paper interacts with current technology from other domains, and determine the impact of this interaction.
3. Discuss at least one positive impact that the author may not have realized.
4. Discuss at least one negative impact that was overlooked, omitted, or missed by the author.
5. Consider the impact (good or bad) of this paper on the economy, society, and/or the environment.
6. Discuss complementary technologies that may mitigate any negative impacts.
3. Consider possible future impacts:
1. Consider how the technology proposed in the paper may lead to future advancements and what would be the social impact of such advancements.
2. Consider how the technology proposed in the paper interacts with future technology from other domains, and contemplate the impact of this interaction.
3. Be creative but reasonable with your assumptions about possible future advancements that could lead to social impact in one way or the other.

References

Alec Jacobson and Colin Raffel, 2021. Role-Playing Paper-Reading Seminars.

Bob Spillman and Ian Parberry, 2000. How to Present a Paper: A Speaker’s Guide.

Lynn Santelmann, 2001. Outline for Research Project Proposal.

Purdue University. Writing a Literature Review.

Tanya Golash-Boza, 2022. 6 Tips for Giving a Fabulous Academic Presentation.