Quick Quiz to Measure What Scientists Know
July 23, 2008 – 7:41 amSuppose you have a room full of scientists—hundreds of ‘em—and want to find out how they actually use computers in their work. There isn’t time to interview them individually, or to record their desktops during a typical working week, so you’ve decided to ask them to self-asses their understanding of some key terms on a scale of:
- No idea what it is.
- Use it/have used it infrequently.
- Use it regularly.
- Couldn’t get through the day without it.
My list is below; what have I forgotten, and (more importantly) how would you criticize this assessment method?
- A command-line shell
- Shell scripts
- Version control system (e.g., CVS, Subversion)
- Bug tracker
- Build system (e.g., Make, Ant)
- Debugger (e.g., GDB)
- Integrated Development Environment (e.g., Eclipse, Visual Studio)
- Numerical Computing Environment (e.g., MATLAB, Mathematica)
- Inverse analyzer (e.g., Inane)
- Spreadsheet (e.g., Excel)
- Relational database (e.g., SQLite, MySQL, Oracle)
- Layout-based document formatting (e.g., LaTeX, HTML)
- WYSIWYG document formatting (e.g., Word, PowerPoint, OpenOffice)
Now, you have the same room full of scientists, and you want to find out how much they know about software development. There still isn’t time to interview them or have them solve some programming problems, so again you’re falling back on self-assessment. This time, the scale is:
- No idea what it means.
- Have heard the term but couldn’t explain it.
- Could explain it correctly to a junior colleague.
- Expert-level understanding.
and the terms themselves are:
|
|
Once again, my questions are (a) what have I forgotten, and (b) how “fair” is this as an assessment method?
12 Responses to “Quick Quiz to Measure What Scientists Know”
I think your first scale is missing something between “Haven’t heard of it” and “use it infrequently”. Perhaps something like “I know what it is, but I don’t use it”
By Daryl on Jul 23, 2008
Howdy from Texas.
Is Q9 a trick question? Not only have I never heard of an inverse analyzer, Google hasn’t either. The first hit on “inane” “inverse analyzer” is this blog entry, and the pickin’s are mighty slim after that.
By Michael Tobis on Jul 23, 2008
The upper scale doesn’t have a good response if they know perfectly well what X is, but have never used it.
By Matthew Woodcraft on Jul 23, 2008
I’d suggest that you add a number into your first scale:
0. No idea what it is.
1. Know what it is, but never used it.
2. Use it/have used it infrequently.
3. Use it regularly.
4. Couldn’t get through the day without it.
By Matt on Jul 23, 2008
What I don’t like about your scale is that it’s loaded and doesn’t cover all the options. There’s pretty much a built-in assumption that if you know about this stuff you will use it. What about people who have good knowledge of something but either have chosen not to use it or have used it a lot and stopped because it was bad (or bad for them)? For example, I’ve not only lived in debuggers, I’ve written them; but I haven’t used one for years.
Personally, I think the most obvious missing thing on the (first) list is a text editor…but I know you think emacs users would still be using punched card if only there decent readers available
(And on a different note, I can’t resist asking, if Word is WYSIWYG system, why does it have a preview button?)
By Nick Radcliffe on Jul 23, 2008
I might add something like “automated builds” to the first list. And perhaps release management or “how do you deploy the software?”
By Matt Doar on Jul 23, 2008
> Perhaps something like “I know what it is, but I don’t use it”
Or even, “I don’t know what it is, but Greg told me I need to use it”
By Matt Doar on Jul 23, 2008
Some trick questions could be useful for filtering bozo responses. “And all of you who said that they used back-scattering inheritance in their code are big ol’ liars!”
~Matt
By Matt Doar on Jul 23, 2008
For the first list you’ve missed out:
14. Computerised Simulation e.g. Monte Carlo methods, discrete event simulations.
15. Computerised visualization software e.g. Thermal analysis.
On the second list, you could add:
* Recursion
* OO
* functional programming style.
* declarative programming style.
* Profiling.
* Refactoring.
The last list doesn’t address the team management side of software development though.
- Paddy.
* Style guides
*
By Paddy3118 on Jul 23, 2008
Your first order question is define ’scientist’. I would hazard that there are many disciplines that not pass any of these tests. eg. Medical researcher. But that does not mean that they aren’t a scientist.
Fact much of this I would pose to the team IT guy not the Principal Investigator.
Just an observation.
By JohnMc on Jul 23, 2008
“I know all about it and you couldn’t pay me enough to use it again”.
By Michael Tobis on Jul 23, 2008
This post is also a high Google hit for “Variant digression”.
define:”variant digression” and define:”variant regression” return nothing.
Yahoo returns little of interest. So … what is it?
By Sam Penrose on Jul 23, 2008