Quasi-experimental design

From WikEd

Jump to: navigation, search

Select some research issue that you care about that could be addressed with a quasi-experimental design and describe the design you might use. Sign your posting and separate it from the others with a horizontal line.
jal

My Quasi-Experiment for Pre-Service Teacher Development (Chris)

The Question: Will pre-service teachers benefit from professional development protocols such as collaborative inquiry and lesson study?

The Issue: Teachers engaged in these professional development activities are able to meet on a regular (daily basis) and observe one another teaching in the classroom. Discussions and analysis about teaching stem observation of classroom practice. Pre-service teachers to not have regular access to observe one another.

The Quasi-experimental Design: Pre-service teachers will form Teaching Study Groups that “meet�? asynchronously in an online community. They observe one another’s practice through electronically distributed digital video. Discussions about one another’s practice is not constrained to physical location or time of day.

The Measure: Performance Assessment of California Teachers (PACT) authentic assessment instrument. The PACT scoring instrument is currently being used as a pilot capstone assessment tool for credentialing new teachers. The assessment has undergone validity and reliability studies with favorable results.

The Non-Equivalent Comparison Group: The treatment pre-service teachers PACT scores will be compared to the scores of pre-service teachers at the other PACT institutions, which include all the other UC Teacher Ed programs, several private university programs such as Stanford and Mills College, and some CSU teacher education programs. Because the private universities and the other Ucs have similar entrance requirements and all have combined credential/Med programs, it can be argued that the individuals in these programs have similar academic backgrounds and training experiences.

Quasi-Experimental Design Ideas (Heather)

First I want to comment that quasi-experimental design possibilities (matching-only design, counterbalanced design, time-series design) seem much more realistic in terms of application than those discussed last week.

Two possibilities where I could see immediate application:

Matching-only design: A history teacher and I have been experimenting with using timed writings rather than the textbook created multiple choice tests to assess student understanding of the concepts involved in U.S. history. Initial results have been strong-- Students appear more engaged and invested in the content of the course. However, he has received some criticism from others in the department and concern that he is not adequately preparing students for the CAT-6. One possible approach to assess the efficacy of timed writings would be to assign timed writings to one of his regular U.S. history classes, while administering text-book standardized tests to the other U.S. history class (classes are "balanced" using reading test scores, thus providing matched data samples). Both groups of students would take the CAT-6 and test results could provide some measure to test the efficacy of a writing approach to assessment.

Time-series design: Debates over effective spelling and vocabulary instruction continue to rage... Despite research indicating that contextualized instruction is most meaningful, many teachers continue to drill and kill with isolated-word spelling and vocab lists passed out on Monday, written out five times each on Tuesday, used in a sentence on Wednesday, made into a crossword on Thursday, and tested on Friday. This approach is easy and to many, appears to "work" because students learn the words for the test.... I predict, however, that a time-series experiment would reveal that most students quickly forget these same words. Such an experiment could be easily conducted by pre-testing a list of words, testing the same list of words immediately after the Monday-Friday routine, testing the same list of words 3 weeks later and again 3 weeks after that. Although there is a concern that repetition of the tests themselves would skew results, I would think that this threat could be minimized by keeping the assessments short and by waiting for fairly significant chunks of time between assessments. I see this as an easily applied design that could generate meaningful discussion among teachers.

Matched group quasi-experimental design (Melissa)

In 2007, the state of California will recognize four different interpreting certifications as meeting minimum competency standards. For this experiment, I would have interpreters working in K-12 settings take a pre-test which includes interpreting a classroom lecture for the grade level they are most accustomed to interpreting. The control group for this experiment would be interpreters certified by the Registry of Interpreters for the Deaf (RID). I would have two other groups of interpreters, those who have achieved a level 4 or more on the Educational Interpreter Performance Assessment (EIPA) and those who have achieved a level 4 or more on the English Signing Skills Evaluation (ESSE). After analyzing interpreting performance on the pre-test, I would randomly select half of the interpreters from each group to observe five videotaped examples of expert educational interpreters in practice. I would then administer a post-test to all of the participants to measure the effects of the opportunity to view the 5 examples of expert educational interpreters in practice.

Quasi-Experimental Design: An Example

Time Series Design: I could evaluate a CBET teacher training module or activity. In this experiment I would:

  • Observe teacher training activity.
  • Observe teachers’ classrooms over the course of say three months.
  • Collect data in the form of pre and post teacher interviews, classroom observations, curriculum, instructional model, types of assignments, etc.


Nonequivalent Control Groups: I could evaluate the effectiveness of CBET ESL programs (CBET would be the intervention) in teaching English to adults English learners. This experiment would be based out of two community colleges that offer CBET programs to similar populations:

  • CBET San Diego City College
  • CBET Southwestern College

In this experiment I would:

  • Describe the type of curriculum being used in each college, and point out similarities and differences between both approaches.
  • Collect data in the form of pre and post CASAS (English proficiency) tests, student work, student interviews and other types of language evaluation instruments, as well as teacher interviews, classroom observations, curriculum, instructional model, types of assignments, etc.


Multiple Time-Series Design: I could evaluate the effectiveness of a particular type of CBET teacher training module in the teaching of ESL to adults. This experiment would be based out of two community colleges that offer CBET programs to similar populations:

  • CBET San Diego City College
  • CBET Southwestern College

First intervention:

  • Teacher training modules given to all CBET teachers in both colleges.

Second intervention:

  • Randomly assign teachers to control and experimental groups.
  • Collect data in the form of pre and post CASAS (English proficiency) tests, student work, student interviews and other types of language evaluation instruments, as well as teacher interviews, classroom observations, curriculum, instructional model, types of assignments, etc.

Luz

Pam’s counterbalanced regression-discontinuity design (i.e. using a cutoff criterion to identify participants) Revised 10-21-04

  • Find or develop some robust and reliable attitudinal measurement instrument (probably a survey) about writing.
  • Select 3 disaffected writers - learners who “hate�? to write.
  • Provide 1-1 interactions with a “tutor�?/writer for 2 similar and consecutive writing tasks.
  • Implementation (multiple sessions/interactions for each)
    • Treatment #1 - tutor gives feedback and instruction focused on learner’s writing.
    • Treatment #2 - feedback is constructed around learner and tutor both working on same kind of piece (e.g. joint construction of process, minilessons around issues both writers are contending with, etc.)
    • Do the ol’ counterbalance switcheroo.
  • Data collection
    • Use observational notes, audio recordings, and evaluation of writing products to examine how feedback is received and used.
    • Administer survey after each writing task is complete (eliciting responses specific to each one).
  • Question: What is the impact of 1-1 feedback on learner’s attitude toward writing if the support includes active involvement on the part of the tutor in his/her own writing? Does it have an effect on writing motivation? Outcome? Is Treatment #2 feedback more likely to be used, or used more constructively?
  • Data analysis
    • Compare surveys. Code notes and audio recordings for motivation-specific data. Evaluate writing samples and examine them for evidence of feedback usage (e.g. comparing drafts).
  • Notes: 1-1 support will likely impact writers and yield different (and better) outcomes than might normally be the case (e.g. in 30-1 settings). Given this virtually foregone conclusion (both from common sense and research), it may be that we can find out the most about what is useful in 1-1 support by examining differences between and among various types of 1-1 interaction. Research along these lines would probably also offer a much more insightful depiction of how 1-1 interactions uniquely support learning.
Retrieved from "http://wik.ed.uiuc.edu/index.php/Quasi-experimental_design"
Personal tools