CSCI 5451: Introduction to Parallel Computing

Parallel architectures design, embeddings, routing. Examples of parallel computers. Fundamental communication operations. Performance metrics. Parallel algorithms for sorting. Matrix problems, graph problems, dynamic load balancing, types of parallelisms. Parallel programming paradigms. Message passing programming in MPI. Shared-address space programming in OpenMP or threads.

Grade of C or better in

• CSCI 4041, OR
• CSCI 3041 AND 3061 AND Instructor consent OR
• Instructor consent.

Students are presumed to have significant experience writing and debugging programs along with some background in basic computer systems architecture such as what is offered through CSCI 2021 and CSCI 4061. Programming assignments will be in the C programming language and students without prior experience working with C programs will be at a major disadvantage.

This course focuses on parallel computing, the art of using multiple processing units to solve computational problems. The primary goal of this effort is either to solve a problem in a shorter time than is possible with a single computer or use the same amount of time to solve a larger problem. The course will primarily focus on tightly coupled parallel computers which have a relatively low overhead to communicate in contrast to networked computers in which communication time is high.

Typical topics covered in the course include the following:

• Basic parallel computing architecture
• Shared and distributed memory computers
• POSIX Threads and OpenMP for shared memory computers
• MPI: Message Passing Interface for distributed memory computers
• CUDA for GPU/Co-processor programming
• Common communication patterns in parallel programs
• Parallel algorithms for matrix operations, sorting, graphs, and discrete optimization
• Evaluation metrics for parallel computing including speedup, efficiency, and isoefficiency
• Emerging technologies such as parallel programming languages and supercomputing architectures

As a Learning Outcome of this class, a passing student will be able to carry out the following types of activities:

• Describe the differences, advantages, and limitations of various parallel computer hardware architectures.
• Convert an existing serial program into a parallel version accounting for coordination, load balancing, and efficient communication.
• Identify common communication patterns that arise in parallel programming and how several programming platforms make them possible.
• Implement a parallel programs using distributed memory, shared memory, and GPU/Co-processor paradigms.
• Evaluate the theoretical and actual efficiency of parallel programs using standard metrics.

The skills to accomplish these outcomes will be built primarily through in-lecture exercises and completion of out-of class assignments. Aspects of the assignments are described below and will comprise a combination of theory, written work, and programming on parallel systems. We will assess the learning outcomes via a series of short exams during the semester and a final exam at the end of the semester.

The course will be divided into sections which address each of the styles of parallel computation which are part of the learning outcomes. Each section will feature an assignment and related exam giving students exposure to a variety of technologies for parallel computing and underscoring the commonalities that exist between them.

Office Hours for staff will be posted on the course Canvas site. Office hours for all staff are open to all students in any section of the course governed by this syllabus. No appointments are necessary but usually students receive help on a first-come / first-serve basis.

Students are strongly encouraged to visit the professor and teaching assistant(s) during office hours to further their understanding of the material: we are here to help you learn.

The following technology platforms will be used to administer various parts of the course and communicate with course staff.

• Canvas (Course Management): used to provide links to other resources and for Overall grade dissemination
• Gradescope (Assignments/Exams): used to submit most graded work, receive grades on that work, and make requests for regrades.
• Piazza (Announcements/Async Q&A): used to ask questions on course material outside of synchronous meetings and for staff to make announcements to students.
• Email: used to set up appointments outside of office hours. Please see the notes below about which inquiries will be handled via email and which should utilize Piazza.
• Zoom (Video Conferencing): used to present audio/video synchronously such for individual online appointments outside of Office Hours.

Final grades will be determined by scores obtained on the components below according to their associated weight.

Final grades will be assigned without rounding according to the following criteria.

If circumstances require it, the grading scale may be adjusted, generally in the students' favor.

Students will receive a number of assignments during the semester. Each will involve combine writing programs, answering theoretical questions, and demonstrating knowledge of the recent course topics. Assignments are usually large and require a significant amount of work to complete.

Specific assignments may allow collaboration between pairs of students or require individual work. Pay careful attention to the guidelines for each assignment.

Late assignment submission will be penalized as follows.

• On-time submissions receive no penalties
• Submitting 1-24 hours will result in the loss of 10% absolute credit
• Submitting 25-48 hours late will result in the loss 20% absolute credit
• No submissions will be accepted more than 48 hours after a deadline
• Late penalties may be assessed later in the semester after the initial round of grading.

Example: Villanelle is running short on time and so takes an extra night to finish Assignment 2. She submits it 16 hours after the deadline. Her grader gives her 85% but later in the semester her score is lowered to 75% due to her late submission incurring a 10% penalty

There will be a series of Mini-Exams through the semester. These will occur during the lecture on the dates they occur and occupy part of the lecture. Dates for the Mini-exams appear in the schedule.

There will also be a comprehensive Final Exam at the end semester. Refer to the schedule for dates of the exams.

All exams are Open Resource Exams: unless otherwise specified you may use any course content during exams including lecture slides, your notes, the textbook, editors/compilers/shells, and any code the you have written or received from course staff. No communication is allowed during the exam (no email/texting/chat), and no Internet searches are allowed (do not "google for answers"). In addition, you may only use exams from previous versions of this course if you personally took those exams. If in doubt, ask about specifics before or during the exam.

Missing an exam results in a zero score and make-up exams will be considered only in situations involving significant life events. Proof of such circumstances will be required for a make-up to be considered.

Most coursework will be graded via Gradescope which features a Request Regrade Button associated with specific problems and criteria. This will notify the specific individual responsible grading about the dispute. Raise regrade requests respectfully and specifically: mention what you think a grader missed in your answer or why you feel a deduction was unfair. Keep in mind that graders assign credit based on what appears on the project and exams, not post-hoc explanations of answers.

If a Student and Grader are not able to resolve a grading issue to the satisfaction of both, the student can ask the grader to consult the Professor who will review the dispute and resolve it. Students should ask their grader to do this, not email the Professor directly.

When grades are published, there will generally be a 1 week window in which disputes are considered. Failing to request a regrade in that time will forfeit further opportunity to contest the grade.

Readings from textbooks and other sources relevant to each lecture are listed in the schedule. You will increase your understanding of lectures by reading associated textbook sections ahead of time, though this is not assumed. We may provide additional reading material to supplement the textbook which will be posted on the course web page.

Some coursework is to be done individually, usually programming Projects and Exams. For this individual work, the following actions constitute scholastic misconduct (cheating) and will be reported.

• Directly copying someone else's solution to an assessment problem, including student solutions from a previous semester
• Directly copying an answer from some outside source such as the Internet or friend for a homework problem.
• Making use of an Instructor Solution manual to complete problems.
• Submitting someone else's work as your own.
• Using or sharing Exam materials from another student from past offerings of this course; you may view your own past copies of exams but not share them with others.
• Paying someone for a solutions to assignments/exams.
• Posting solutions to any web site including public posts to our course web site.
• Collaborating or copying the work of others during an exam.
• Taking another student's code with or without their consent.
• Giving another student one's own code on assignments or exams
• Aiding or abetting any of the above.
• Witnessing any of the above and failing to report it to an instructor immediately.

REMEMBER: Once you give away your code/work, you lose control of it. This may lead to pain in the following non-obvious cases

• Using a public Github repository to track your code allows anyone to copy it and submit it as their own.
• Sharing code with a classmate "just to help them" may lead to them submitting it as their own, sharing it with others, or selling it for profit
• Giving someone access to your accounts or devices to help them may mean that they use your account to steal your work subsequently.
• Leaving your work open in a public space may allow someone to take snapshots of your screen and use it as their own.

All Of This Has Happened Before And Will Happen Again. Don't become one of your Professor's stories.

Refer to the following links for additional information.

• UMN Student Conduct Code
• Computer Science Department policies on scholastic dishonesty
• Office of Community Standards

Any instance of misconduct that is detected will be referred to the Office of Community Standards and will likely result in failing the course. Be advised that the teaching team will be employing electronic means to detect plagiarism. This is extremely easy with computer code so keep your nose clean.

The purpose of this course is to learn about programming and learning from one another is a great help. To that end, the following actions will NOT be considered cheating in this course.

• Collaborating with a partner on Assignments that allow it is perfectly acceptable. Limit your collaborations to within your partnership to avoid trouble.
• Discussing code at a high level with students other than a partner is fair so long as no code is shared. Take great care at the point of directly showing your work to others as your answers are subsequently out of your own control.
• Asking public questions on the course discussion board so long as limited information about your own solution is included. To convey details of your solution, use private posts.
• Asking any course staff member questions in person or online is acceptable and encouraged. Staff members may initiate small group discussions in which collaboration is fine.
• Making use of your own code or exam materials which you accumulated from past semesters of this class is fine. If you are retaking the course, make sure staff know this so that no misunderstandings occur.
• If you are unsure whether a given collaboration is fair or not, stop the activity and clear it with your instructor.

At all times keep the PRIME DIRECTIVE in mind when studying with other students. The above collaborations should be limited to getting someone over a hurdle, not carrying them across the finish line.

This course will have a number of online interactions and the following additional policies on student/staff behavior and Academic Integrity also apply: https://communitystandards.umn.edu/know-code/online-learning-expectations

The gist of the behavior policies are "Don't be a troll" and the Academic Integrity portion reflects our PRIME DIRECTIVE.

This course will include video and audio recordings of class lectures and classroom activities. These recordings will be used for educational purposes and the instructor will make these available to students currently enrolled in this course. Students must seek instructor permission in order to share either course recordings or course content/materials. Similarly, instructors who wish to share zoom recordings with other sections or classes must seek and document permission from students whose image or voice are in these recordings.