One of the most important aspects of 5271 is a research-oriented group project. The goal of the project is to introduce you to current research topics and methods in computer security. Based on the experience of past offerings of the course, the most reliable style of project is one that engages with a recently published research paper in the area or propose a specific research idea together with the instructor. So our plan is that most projects will be based on a paper published in recent two years in one of the major security conferences (IEEE Security & Privacy, ACM CCS, USENIX Security, and ISOC NDSS). Here are the links to find the papers.
- IEEE Security & Privacy (Oakland): 2017, 2018;
- ACM CCS: 2016, 2017;
- USENIX Security: 2017, 2018;
- ISOC NDSS: 2017, 2018.
By 11:55pm on Wednesday, September 19th, one member of each group should submit a one-page pre-proposal (in PDF format) with the following information:
- Who: All members of the proposed group, and a group or project name.
- What: The paper(s) your project will be based on (listing one or two related choices for discussion would be acceptable). Or, what research you propose to do.
- Why: Why is this a good paper research idea for your group to base a project around? What skills and resources can you access that will allow you to successfully complete the project?
- How: How do you intend to reproduce the paper's results? What extensions or attacks do you have in mind, and how will you evaluate them? Or how do you plan to design, implement, and evaluate your research idea.
- When: A list of at 20 minutes time slots during a typical week in which your group members are all available for a progress meeting with the instructor. A list of the instructor's available slots are shown in the provided calendar in Moodle. To sign up a meeting, go ahead to create a 20-minute meeting (event) in the calendar, including names of your group members and title of your project.
Three progress meetings and reports (5% each)
Each group schedules three 20-minute meetings with the instructor. Each member should then submit a progress report, including individual progress and contributions.
Presentation (20%) Each group will give an in-class presentation of their work during the last two lectures. This could be a presentation with slides and/or demo. Each group has 10 minutes: 8 minutes for presentation, and 2 minutes for Q&A. The presentation is graded for the whole group instead of individuals, so the entire group has responsibility to ensure the best possible presentation. Given the time constraints, we expect that only one team member will speak; however, more students can join if necessary for demo.
Presentation grades will be based on the following criteria:
- Content (3%): suggested components include motivation, problem statement/threat model, solution/attack, design/implementation, evaluation/demo, limitation/discussion, and conclusions.
- Significance(10%): novelty, research value, solid design/implematation, good evaluation.
- Organization (3%): components are organized in a coherent, eash-to-follow way.
- Presentation (4%): audience engaging; clearly presenting and answering questions.
Final report (60%) By 11:55pm on Wednesday, December 12th, one member of each group should submit an archive which includes any "deliverables" and a project report in PDF format. The report should be in the style of a computer science conference paper, and it should present the project's motivation, threat model, approach, design/implementation, evaluation results, contributions, related work, and conclusions. The report should be between 8 and 12 pages plus references, using the ACM proceedings format. A separate file ("contrib.pdf") should summarize the hours spent, and contributions of, each group member.
Report grades will be based on the following criteria:
- Originality and novelty (20%). To get full credit, your project should contain new ideas beyond the paper it engages (new design, new analysis, new implementation, new evaluation), of the sort that could be published in an academic conference (not necessarily top-notch).
- Scholarship and research value (20%). A good scholar knows the state of the art in their project area. Think about questions like: Does the paper include the important related work? How is the paper contrasted with other work? How well do the authors distinguish between what has been written by others and their own work? A sign of poor scholarship is a lack of well-explained comparisons to previous papers from top-notch venues; in particular, any good project should be able to cite, and explain its relationship to, at least 5-10 such papers.
- Evaluation (30%). To what extent do the authors evaluate their approach and establish or support their claims? For example: If you propose a new extension to defense mechanism, what are some potential ways to defeat or avoid it? How feasible are they? How did your system perform on standard benchmarks? Do they introduce any issues such as compatability? If you propose a new attack, how realistic is it? Can you carry it out empirically? Are there simple countermeasures that can reduce the effectiveness of your attack? Is your attack trivially less effective than known previously-known ones? When reporting on experiments, do you analyze their statistical significance? Do you give enough information that another group could replicate your results? If your claims are analytical, do you give enough details of the proofs? An important part of the evaluation is simply making the description of your project clear enough for others to understand what you have done!
- Individual contribution (30%). Each student should have made some identifiable contribution to the project. This component will be based on the individual student's contribution: did you attend progress meetings? Did you contribute (at least) as much to your project as others? Did you incorporate feedback received at meetings throughout the semester? A graduate class major project like this one is expected to require at least 50 hours of work per group member over the course of the semester (and that's for an average project, not an "A" project). Of course the amount of time spent isn't a sufficient condition for a good project. But our experience is that being unable to devote enough time to a project is the most common cause of poor results. That's why we ask you to keep track of the time you spend and think about how to use your time effectively.