Quizzing as feedback for students

It’s important for all of us to get feedback and the timeliness of feedback matters too. Remember how it felt when you submitted something to your doctoral thesis committee to review and they took FOREVER to get back to you? Or when you posted that picture on Facebook and the folks you thought would love it didn’t even give it a like let alone a comment? Timely feedback to students is useful to their learning and could be that thing that helps them feel like they belong at Carleton.

When designing or revising your course, one way to situate the types of feedback you’ll give is by using the classic Backward Design model by Wiggins and McTighe. Specifically, it can be helpful to use their diagram for setting curricular priorities into alignment with the types of assessment you might use. We can imagine that quizzing might best align with the concepts or outcomes that are important for students to know or to have facility with in order to wrestle with the BIG ideas or “enduring understanding” of a course.

diagram for setting curricular priorities into alignment with the types of assessment
Diagram of curricular priorities and assessment types

Quizzing, and particularly multiple choice quizzing done outside the classroom (such as implemented via Moodle, auto-graded, and reported to the gradebook), can make frequent, meaningful feedback for students not only possible but efficient.

Frequent low stakes “testing” (i.e. the need to retrieve information whether in a quiz or otherwise) promotes learning (Roediger and Butler 2011). Moreover, frequent quizzing, besides promoting memory, increases the likelihood of transfer (Carpenter 2012).

You can also give feedback on these quizzes. The same Roediger and Butler–but this time in 2008–showed that while multiple choice questions improve student performance, feedback to students on their answers provides additional benefit. If that feedback is explanatory as to why an answer is wrong the transfer effect is stronger than simple feedback saying the answer is wrong (Moreno and Mayer, 2005). Crafting feedback is decidedly not efficient though! But…it may still be worth your effort in terms of student learning and if you reuse the quizzes your time investment will pay off. Moodle can help here too by making it easy to add feedback specific to each of the possible choices students can make in the quiz. And if you’re teaching a course that uses a textbook you should be aware than many textbooks provide banks of questions with answers and feedback and this can certainly lighten your load.

As always, AT is here to help you if you want to consider this pedagogical move. Please don’t hesitate to reach out to me (jrussell@carleton.edu) or any ATer if you have questions or concerns or would like to work with us!


Butler AC and Roediger HL III (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory and Cognition 36, 604-616.

S.K. Carpenter (2012),Testing enhances the transfer of learning, Current Directions in Psychological Science (Sage Publications, Inc), 21(5).

Moreno and R.E. Mayer (2005), Role of guidance, reflection, and interactivity in an agent-based multimedia game, Journal of Educational Psychology 97(I).

Roediger HL III and Butler AC (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences 15, 20-27.

Wiggins, Grant P., and Jay McTighe (2011). The Understanding by Design Guide to Creating High-Quality Units. Alexandria, Va: ASCD.

Quickstart for Podcasts

What are podcasts?

Podcasts are digital audio files which can be listened to by streaming or downloading.

Why use podcasts?

Podcasts are sometimes used in lieu of a paper assignment or to augment a paper assignment. A podcast is devoid of visual material (think NPR) and this can be useful for focusing student attention. Continue reading Quickstart for Podcasts

Janet’s Spring 2018 Update

This Spring Term online teaching and learning is much on my mind.

I’ll be presenting at OLCInnovate 2018 in Nashville, Tennessee this April. OLC, Online Learning Consortium, pairs up with MERLOT (for those of you who remember this repository) for their big conferences. I’ve got a couple presentations but for this update I’m focused on the one titled “Is online teaching and learning relevant for small residential liberal arts colleges?” It’s in the session category ‘Conversations, Not Presentations’ which means I get to talk with attendees rather than at them and I’m hopeful for some interesting and useful conversation.

For the OLC conversation I’ll be pulling from my Carleton work with CUBE, Carleton Undergraduate Bridge Experience. CUBE consists of an online 6 week summer portion and a traditional Fall Term portion. Summer CUBE has two primary goals: 1) to strengthen the quantitative skills of incoming students and 2) to connect participants to the campus community before they arrive on campus.

CUBE in turn has heavily contributed to Carleton’s LACOL (Liberal Arts Consortium for Online Learning) work on the QLAB project and this will inform my OLC conversation as well. QLAB is the nickname applied to a multi-campus development and educational research initiative to assist our students with readiness for their quantitative work across the curriculum, and to investigate the role that online resources may play in this.

And all of this online teaching and learning work at Carleton is going easier for me because of my time at Georgetown University which was  just prior to coming to Carleton. There, in my position as Director of Technology Enhanced Learning, I helped design and implement online courses and MOOCs, and in my position as adjunct professor, I taught an online course.

I won’t rely on my experiences alone though to pull off a good conversation at OLC! I’ll be tracking the current buzz about online T&L and that includes a few folks even talking about it in the context of small liberal arts schools:

Reflecting on the Original Big Idea for MOOC’s

Is online education good or bad? And is that really the right question?

Can Online Teaching Work at Liberal-Arts Colleges? Study Explores the Pros and Cons.

Wish me luck!

I Just Know…


A recent Science Post satirical article titled, “I just know” replaces systematic reviews at the top of the evidence pyramid, is a pretty funny read with a darker side.

While the article focuses on medical science (“There is no science backing up my claim that the homeopathic pill cured their cold, but in my gut I just know it did.”) it got me thinking about the teaching and learning work we do here at Carleton and our levels of evidence.

What evidence–beyond “I just know”–do we accept for what we have done in Carleton courses or for what we hoped to have done? If evidence more robust than “I just know” was available for our teaching and learning endeavors, would we want to gather it? What if it was easily available? Any instructional technology we use at Carleton can help with the collection of evidence. And with thoughtful design, that collection of evidence can be “easy” while still being meaningful.

Continue reading I Just Know…

Skills are (part of) “The New Canon”?

horizontal sign with words "new skills training"

In the Chronicle of Higher Ed* article “If Skills Are The New Canon, Are Colleges Teaching Them,” Dan Barrett suggests that after 20ish years of wrestling with a move away from a core liberal education curriculum based on classical texts (a la Harvard Classics), “a new consensus has emerged: that colleges ought to develop in students a set of skills.”

The idea is not to abandon content or disciplines but to build curricula based on specific knowledge that instills skills.

Typically the skills mentioned include those that are quite recognizable at Carleton and some even baked into the curriculum: critical thinking, writing, analytical and quantitative reasoning.

But other skills and methods, perhaps those particularly associated with turning out students ready for the 21st century, are not necessarily part of any canon. This list includes but is not limited to visual and spatial reasoning, information literacy, interpretation of meaning, using and evaluating evidence, and verbally communicating thought.

Barrett’s article shares some examples of how some colleges teach a new canon. Some curricula are tweaked and some are radically reimagined or created afresh. This is certainly happening at Carleton and we’re also seeing an approach to addressing the skills/method gap that Barrett doesn’t mention—the micro-course.

Carleton Spanish Professor, Palmar Alvarez-Blanco, has created an experimental two-credit micro-course as a companion to one of her courses. She wanted to “empower students to understand narratives found in mainstream media, research and produce their own short documentaries, and strengthen their civic engagement experience, digital storytelling proficiency, and practical career skills” and realized that combining this dense packet of skills and methods into a single week of the existing course was not ideal. Palmar is working closely with Academic Technologist Dann Hurlbert on this experimental micro-course. Their process and findings could provide the framework for micro-courses at Carleton as well as inform the revision of existing courses and creation of new courses in order to deliver on a new canon.

We can all look forward to a report from Palmar and Dann and I know I’ll be keeping an eye out for more examples to share of thoughtfully designed courses which effectively teach 21st century skills and methods as well as the disciplinary knowledge at hand.

*The Carleton library subscribes to the Chronicle and provides institutional access.  If you’re on campus, you can simply go to chronicle.com, if you’re off-campus, you can use this link to log on with you Carleton username and password: http://www.chronicle.com.ezproxy.carleton.edu/

Electronic Notebooks at Carleton: LabArchives

In the LTC Lunch: Making Learning Visible with Electronic Portfolios (Jan. 19, 2016), Deborah Gross, Professor of Chemistry, talked about how Lab Archives captures the learning that her students are doing in her lab courses.

*Deborah’s presentation begins at 14:30

LabArchives (LA) is an electronic notebook software that comes in two versions: classroom and professional. The classroom version allows you to manage a set of individual notebooks for each student, replacing the traditional lab notebook; the professional version is great for organizing your professional research and involving collaborators.

Despite its name and logo, which features iconic chemistry vessels, please don’t think LA isn’t for you or your students if you aren’t in the Natural Sciences! It really is an electronic notebook and not a lab notebook. In fact, we’ve heard from our LA rep that staff working in the area of facilities find LA very helpful.

Based on the testing that Deborah and other Carleton faculty participated in, the College has purchased access to both the Professional and Classroom editions for anyone on campus. In a sense, we’re piloting a site license for LA to see if enough folks will use it to warrant continuing in this way (or, if buying on a case by case basis is more prudent).

Our AT [un]workshop on Wednesday October 26, Electronic Notebooks for Classroom & Research: Exploring LabArchives, is a great chance to learn more about LA. You may discover, as Deborah did, that in addition to making student learning more visible both to external audiences and to the students, electronic portfolios like LA can solve logistical problems. For example, in chemistry, LA eliminated the logjam that occurs when students have turned in their physical notebooks to be graded and then don’t have them to prep for the upcoming lab, or how to deal with group projects that are recorded only in one notebook. You also may discover how an electronic notebook might help you capture, archive, and curate your own work.

We also have the following hands on training sessions coming up:
Thursday, October 20 at 3:15 – 4:00 pm in the Weitz Center, Room 027
Tuesday, November 1 at 3:15 – 4:00 pm in the Weitz Center, Room 027

Questions? Contact Randy Hoffner, rhoffner@carleton.edu

Randy Bass Visit

Randy Bass in sitting in canoe pointing in direction

It was just a year ago that Carleton hosted Georgetown Associate Provost, Randy Bass, and I’ve been thinking of some of the work directly facilitated by Randy’s visit. I’ll talk about just two examples in this post.

Liberal Education in the New Ecosystem, Randy Bass from Carleton Academic Technology on Vimeo.

In broad terms Randy talked about designing a liberal education for this moment in history. He asked us, “If you were creating a Carleton education right now, and with everything you knew about the past but also what you knew about the capacities of our current environment and the challenges we are about to face in the next 10, 20, 30 years, what would that Carleton look like?” He suggested our answers would almost certainly cause us to create pilot programs that would push against the constraints of our current Carleton model and that we would need “a different kind of approval process” to make experimentation possible. Enter example 1: CUBE.

CUBE (Carleton Undergraduate Bridge Experience) is a summer/fall experimental course taught by Melissa Eblen-Zayas from our physics department. Carleton’s Education and Curriculum Committee (ECC) made the experiment possible by flipping the usual process so that we got quick approval for a pilot with built in accountability and sunset if not approved for continuation. CUBE has two primary student goals: to strengthen quantitative skills and to support the transition to college. The summer part of the course is completely online and is followed by the face-to-face fall course that is being taught right now. Melissa and I look forward to reporting out to the community sometime this winter about this very exciting experiment and regardless of whether the course continues, I think we have taken a solid step toward that future Carleton Randy asked us to imagine. And for me, Randy’s framework and encouragement were crucial.

Example 2 comes in the form of an LTC lunch session and Winter Workshop. Both of these events are sponsored by Carleton’s Future Learning Technologies Group (FLTG) and both events center on creating flexible curricula that push against the 9 ½ week structure or our current term system. The LTC lunch session is Tuesday, October 11 and the Winter Workshop will be in early December. I can’t speak for everyone in FLTG, but I know I was (re-)inspired and (re-)energized by Randy’s question “what would a course look like that was less course-based, less term-based, separated credit from seat-time, thought about all 12 months of the year, thought about how faculty would deploy their energies in different ways? And this energy and inspiration is playing a shaping role in how I’m approaching my part of these two upcoming events.

Randy Bass in sitting in canoe pointing in direction
Randy points the way in the Boundary Waters and to the futures of higher ed!
A year ago, I really bungled my introduction of Randy before his big talk in the Athenaeum. I was tired and had left my notes behind and pulling them up on my phone just didn’t work. If I had the chance to hit “reset” I would say something like this: Just as I wouldn’t (yet) wander into the Boundary Waters without the wonderful Minnesota guide I have luckily found, I wouldn’t navigate the “future of higher education” without the wonderful Randy Bass along!

Droning on

See Janet’s Blog by clicking here.

Selfie sticks were just the beginning. Now, we have selfie drones.


The Lily drone is to be released this summer according to this report from Mashable.

We don’t yet have any drone projects in Academic Technology but we’re hopeful that the next crop of internally funded grant projects could bring us one. We’re aware that this is a promising technology but we are also aware that the College needs to move forward prudently. There are FAA rules and regulations to be mindful of and even if those are followed there is still much havoc that could be wreaked. We are working to build name recognition in AT but bad drone behavior is NOT the rep we’re aiming for!

AT for DS search

See Janet’s Blog by clicking here.

Academic Technology at Carleton has redefined itself several times over the past 20+ years and is right now undergoing another reimagining.

We have opened a search for an Academic Technologist for Digital Scholarship (AT for DS). The role of AT for DS and the person in this role will play a central role in our current reimagining process. With the AT for DS position, Academic Technology will be buffing up expertise we already have and adding expertise in particular areas of digital scholarship. Digital Scholarship is rather broad, encompassing methods and practices such as statistical analysis, online exhibitions, hybrid and online pedagogy, technology-enhanced assignments, GIS and mapping, visualizing data, text analysis and electronic text encoding, 3D modeling, network analysis, and digital publishing.

We have a great search committee:

  • Pierre Hecker, Associate Professor of English
  • Dann Hurlbert, Media & Design Specialist, Academic Technology
  • Silvia Lopez, Professor of Spanish, Director of the Humanities Center
  • Ron Rodman, Dye Family Professor of Music and Director of the Carleton Symphony Band
  • Janet Russell, Director of Academic Technology
  • Ann Zawistoski, Head of Reference and Instruction, Library

The committee will be winnowing down the strong candidate pool to those we will bring to campus visit. Please stay tuned so you can participate in this exciting process.

C-ray symbol