This term, I organized the CSC302 students into six teams of about ten people each, and put each team to work on an open source project. Most of the marks in the course are given to the team as a whole, so I wanted a way to assess individual contributions. Having students evaluate their peers would be easy, and in theory would reward people who’ve done more, and penalize people who’ve done less. However, peer evaluation is notoriously vulnerable to gaming and social pressure.

My thought was therefore to measure how well people understood their own contributions, rather than the actual contributions. I thought I could have everyone rate both themselves and all of their teammates. An individual’s grade would then be based on how closely their self-score matched the scores given to them by their teammates. If your teammates all gave you 2/10, for example, and you also gave yourself 2/10, you’d get a high mark for knowing that you’d done a poor job.

Well, you can probably see the flaws, and having thought about my colleagues’ comments, so can I. For one thing, it seems unfair to give people a good grade for doing a poor job. It also seems unfair to grade them twice on the same work (once during term for doing it, and once again at the end for having done it). And it would still be susceptible to gaming, though surprising students with it a final exam would reduce that risk.

All of which raises an interesting question. Peer evaluations are routinely used in industry as part of annual performance reviews; bonuses and promotions routinely depend on what your colleagues think of you. If it’s OK to use them there, why is it so difficult in a course? If anything, there is more incentive to game things in a company where you’re likely to work with people for years than in a course where you’re only working with someone in one or two courses.

Surely, this can’t be it—it can’t be that Sakai + Twitter + a blog or Wiki is what all future studies will call the “traditional” form of online courses?

It’s worth checking out some of the tools he links to: there’s a revolution waiting to happen here, and I’d really like to be part of it.

My starting point is Joel Spolsky’s comment that, “…student projects, while laudatory, frequently fail to deliver anything useful.” Respectfully, I beg to differ: About a quarter of the student projects I’ve helped supervise since 2002 have delivered software that clients actually used, and the rest have produced something even more useful: experience. (And when I say “about a quarter”, I’m talking about a quarter of 368 people from 35 countries of origin working on 136 projects.)

To make a long story short, undergraduate students can build great software if:

1. you have realistic expectations,
2. you’re patient, and
3. you realize that “how” matters more than “what”.

Let’s take those in order.

Realistic Expectations. Most undergrads are doing five courses at once, which leaves them only 8-10 hours/week per course. Putting it another way, 13 weeks of student time is equivalent to 3 weeks of full-time work. When you’re planning projects, you therefore have to ask yourself how much you would expect a new (junior) hire to do in their first three weeks on the job, particularly if (as is often the case with students) they’d never used the tools or worked in the application domain before.

It’s equally important to have realistic expectations of faculty, most of whom are working even harder than their students. A colleague of mine once summed up professorial life by saying, “We’re here to do research, they pay us to teach, and we spend our time on admin.” Keep in mind that computer science professors care about computer science, which is emphatically not the same thing as programming. Computer science is the scientific study of what computers can do; a computer scientist’s job is to invent (or discover, depending on your philosophical point of view) new knowledge. That sometimes involves writing code, but that’s like saying that mathematics sometimes involves doing integrals.

It took me a long time to understand this, and I’m not the only one to misunderstand this. I believe very strongly that being a better programmer can help someone be a better computer scientist, but if you want any traction in academa at all, you have to remember that programming really is just a means to an end.

Patience. Your project may be the first time students have written something that isn’t just going to be marked and thrown away. It may also be the first time they’ve been in a situation where 90% right is a failure rather than an A. You must not make students feel like failures for working the way the educational system has trained them to, even (or especially) if those ways are wrong.

Aside #1: one of the best students I’ve ever had explained to me several years ago why she always left her assignments until the last possible moment. “If I start early,” she said, “I’m the one who has to ask the prof all the questions to clear up what the assignment actually means. And if I start early, the odds are good that one of the prof’s answers will mean I have to re-do something. On the other hand, if I leave it ’til the night before, I can be sure that I have a stable spec.”

Aside #2: People often ask why schools don’t teach students Git, or Haskell, or GPU programming, or mobile devlopment, or whatever else is currently cool. The answer is that the curriculum is full. 4 years × 2 terms/year × 13 weeks is 4800 hours, and every single one of those hours is spoken for. Do you want to add functional programming? Cool! What are you going to take out: operating systems or B-trees?

How Matters More Than What. When I ask alumni of my project courses what the most valuable thing they learned was, none of them name a particular technology—none. Instead, they talk about teamwork, presentation skills, code reviews, time management, prioritization, communication, negotiating real requirements with real users, and building their network and portfolio. This is why I think that efforts to teach open source are slightly mis-aimed: in my experience, students don’t care if the code is open or closed nearly as much as they care about how to build complex things. It just happens that the various open source communities are both good at that, and able to talk publicly and in detail about the mechanics.

My colleagues and I at the University of Toronto have learned a lot of other things about running student projects, including:

• How to run meetings
• How to teach students how to do code reviews
• What level of tooling is appropriate/feasible
• Accountability
• How to accelerate ramp-up
• Carry-overs from previous terms
• Importance of full-time summer work (yay GSoC!)
• Industry support
• Presentations, presentations, presentations
• Scoping and re-scoping deliverables
• Recruiting students and faculty
• How to grade one-of-a-kind projects

We’re now scaling up what we’ve learned in UCOSP, a set of undergraduate capstone projects in which students from 14 universities work together in distributed teams. If you’re interested in taking part, we’d like to hear from you.

Universities already widen the gap between rich and poor, by flunking out or not admitting the poor. On-line courses tend to flunk out even more students, and mostly at the lower-knowledge and poor levels…  I think it’s possible for on-line education to be even better than existing University education, in terms of improving learning and engaging a broader range of students… [but] the work has to happen first. If [universities] disappear in favor of [online education], before we make [online education] better, [it] will lead to worse education for society, especially for weaker students.

Elsewhere, though, things are not so rosy. The students in my CSC302 software engineering course had to hand in their first assignment last week. One team brought it in two days early (earning themselves two days of grace to use in future assignments), four of the others came in on time, and one team was a day late. That’s about what I expect; what I didn’t expect, and was disappointed by, was that two teams were tossing files back and forth by email, rather than using the version control repositories we had set up for them. This is triply frustrating because (a) they’ve all had to use version control in previous courses, and (b) the only way they can submit their work for grading is to commit it to their repo. Many of these same students are in their final term, and will be entering the workforce in just over three months; somehow, we just haven’t gotten through to them.

Coincidentally, my wife was up from midnight Saturday until 10:00 am Sunday babysitting a software deployment that ultimately had to be rolled back. She works for one of the largest IT consulting firms in Canada, and her team has updated the software in question several times in the past. It still took them seven hours to get to the point where they could tell that a few key things just weren’t going to work, despite all the pre-testing they and other teams had done, and another three hours to re-set to a working state. Michael Nygard talked at length in Release It! about how to not get into this situation, and I’ve thought more than once about creating an undergrad course to teach the necessary concepts, but realistically, if our senior students still think that version control is something you use once at the end of the project, and only because the prof says you have to, it’s clear there are higher priorities.

1. Adding pivot tables to Gnumeric.
2. Upgrading PyLint.
3. Converting the Selenium IDE to a plugin architecture.
4. Improving the SpatiaLite GIS extensions for SQLite.
5. Porting Django to Python 3.
6. Helping with ILUTE (the Integrated Land Use and Transportation Engineering tool).

10-11 students will be working on each; it promises to be an exciting term.

I’ve been pleased with the results, but unfortunately managed to misplace my links to the papers that gave me the idea of trying this. I’m therefore grateful to Brock MacDonald, of the Academic Writing Centre at Woodsworth College, for sending the following:

The most thorough empirical studies of student note-taking were published in a series of papers in the 80s and early 90s by K. A. Kiewra, alone and with various collaborators. They found that students who made and studied their own lecture notes achieved consistently better test results than students who studied from notes they were given. An interesting wrinkle is that students were found to do best of all when they were given a lecture topic outline or matrix of some kind to fill in with their own detailed notes—in other words, student note-taking that’s framed or guided by limited input from the instructor leads to better results than either a) students making notes completely on their own (though this gives the 2nd best results) or b) students being given the lecture notes and not making notes of their own at all (which gives the poorest results).

Empirical evidence for the value of students writing their own assignment specs is harder to come by, because it’s less amenable to direct testing (like most aspects of teaching writing)–the support is more qualitative than quantitative, hence more open to question. The idea most often comes up as part of discussions of critical-thinking- and active-learning-oriented writing pedagogy, e.g., in Peter Elbow (his book Writing Without Teachers and many articles) and John Bean (his book Engaging Ideas); both Elbow and Bean refer to quite a bit of supporting research, though as I said it’s mainly qualitative. The value of students developing the skill of formulating good questions in general (i.e. not only in the context of devising assignments) is also supported in some of the literature on learning objectives and outcomes, e.g., in several articles by Mike Carter, some of which present case studies that show the benefits of building this skill into entire course and program curricula.

Demographics

Channels

General Background

Python Background

Exercises

Individual Chapters