Theory driven research designs - .
res-design-quant-vi-1
Theory driven research designs
(version 0.9, 1/4/05, some examples in french)
Code: res-design-quant
Daniel K. Schneider, TECFA, University of Geneva

Menu
1. Theory driven research
2
2. Experimental designs
3
3. Quasi-experimental designs
12
4. Statistical designs
24
5. Similar comparative systems design
32
6. Summary of theory-driven designs discussed
33
Research Design for Educational Technologists
© TECFA 1/4/05

---

Theory driven research designs - 1. Theory driven research
res-design-quant-vi-2
1. Theory driven research
Most important elements of an empirical theory-driven design:
Theory
compare
Hypothesis
Truth:
Test
Measures
validities
reliability
Causal analysis
• Conceptualisations: Each research question is formulated as one or more hypothesis. Hypothesis
are grounded in theory.
• Measures: are usually quantitative (e.g. experimental data, survey data, organizational or public
"statistics", etc.) and make use of artifacts like surveys or experimental materials
• Analyses & conclusion: Hypothesis are tested with statistical methods
Research Design for Educational Technologists
© TECFA 1/4/05

---

Theory driven research designs - 2. Experimental designs
res-design-quant-vi-3
2. Experimental designs
2.1 The scientific ideal
Control physical interactions between variables
Experimentation principle in science:
1. The study object is completely isolated from any environmental influence and observed
(O1)
2. A stimulus is applied to the object (X)
3. The object’s reactions are observed (O2).
O1
X1
O2
or
O1
X1
O2
• O1 = observation of the non-manipulated object’s state”
• X = treatment (stimulus, intervention)
• O2 = observation of the manipulated object’s state”.
Effect of the treatment (X): the difference between O1 and O2
Research Design for Educational Technologists
© TECFA 1/4/05

---

Theory driven research designs - 2. Experimental designs
res-design-quant-vi-4
2.2 The simple experiment in human sciences
It is not possible to totally isolate a subject from its environment
Simple experimentation using a control group :
X=stimulus,
R
X
O
experimental group
(independent explanatory variable)
O=mesure
R
O
control group
(dependant variable, to explain)
Principle:
1. Two groups of subjects are chosen randomly (R) within a mother population:
• this ought to eliminate systematic influence of unknown variables on one group
2. Ideally, subjects should not be aware of the research goals
3. The independent variable (X) is manipulated by the researcher (experimental condition)
Analysis of results: effects are compared:
effect (O)
non-effect (O)
treatment: (group X)
bigger
smaller
100 % vertical
comparison
non-treatment: (group non-X)
smaller
bigger
100 %
• What is the probability that treatment X lead to effect O ?
Research Design for Educational Technologists
© TECFA 1/4/05

---

Theory driven research designs - 2. Experimental designs
res-design-quant-vi-5
(b) Simple experiment with different treatments:
R
X1
O1
experimental group 1
R
X2
O2
experimental group 2
R
X3
O3
experimental group 3
R
X..
O..
• a slightly different alternative
• Example: First students are assigned randomly to different lab sessions using a different
pedagogy (X) and we would like to know if there are different effects at the end (O).
Problems with simple experimentation:
• Selection: Subjects may not be the same in the different groups
• Since samples are typically very small (15-20 / group) this may have an effect
• Reactivity of subjects: Individuals ask themselves questions about the experiment
(compensatory effects) or may otherwise change between observations
• Difficulty to control certain variables in a “real” context
• Example: A new ICT-supported pedagogy may work better, because it stimulates the teacher, students
may increase their attention and work input, groups may be smaller and individuals get more attention.
• In principle one could test these variables with experimental conditions, but for each new variable, one
has to add at least 2 more experimental groups, .....
Research Design for Educational Technologists
© TECFA 1/4/05

---

Theory driven research designs - 2. Experimental designs
res-design-quant-vi-6
2.3 The simple experiment with pretests:
R
O1
X
O2
experimental group
R
O3
O4
control group
• To control the potential difference between groups: compare O2-O1 (difference) with O4-O3
• Disadvantage: effects of the first measure on the experiment
Example: (a) If X is supposed to increase pedagogical effect, the O1 and O3 tests could have
an effect (students learn by doing the test), so you can’t measure the "pure" effect of X.
The Solomon design:
R
O1
X
O2
R
O3
O4
R
X
O5
R
O6
• combines the simple experiment design with the pretest design:
• and we can test for example: O2>O1, O2>O4, O5>O6, O5>O3
Note: comparing 2 different situations is NOT an experiment ! The treatment variable X
must be simple and uni-dimensional (else you don’t know the precise cause of an effect)
• There a more complicated designs to measure interaction effects of 2 or more treatments
Research Design for Educational Technologists
© TECFA 1/4/05

---

Theory driven research designs - 2. Experimental designs
res-design-quant-vi-7
A. The non-experiment: what you should not do
The experiment without control group nor pretest:
X
O
experimental group
Exemple 2-1: A bad discourse on ICT competence of pupils
“Since we introduced ICT in the curriculum, most of the school’s pupils are good at finding things on the
Internet"
There is a lack of real comparison !!
• We don’t compare: what happens in other schools that offer no ICT training ? (Maybe this is
a general trend since more households have computers and Internet access.)
• We don’t even know what happened before !
Most of the students are good ! ...
x= ICT in school x= no ICT in school
bad at web search 10 students
???
• horizontal comparison
of % ???
good at web search 20 students
???
Things have changed ! ...
before
after
bad at web search ???
10 students
• horizontal comparison
of % ???
good at web search ???
20 students
Research Design for Educational Technologists
© TECFA 1/4/05

---

Theory driven research designs - 2. Experimental designs
res-design-quant-vi-8
Experiments without randomization nor pretest
X
O
experimental group
O
control group
There is no control over the conditions and the evolution of the control group
• Example: Computer animations used in school A are the reason of better grade averages
(than in school B)
• School A simply may attract pupils from different socio-economic conditions and that usually
show better results.
The experiment without control group
O X O
experimental group
We don’t know if X is the real cause
• Example: “Since I bought my daughter a lot of video games, she is much better at word
processing ”
• You don’t know if this evolution is "natural" (kids always get better at word processing after
using it a few times) or if she learnt it somewhere else.
Research Design for Educational Technologists
© TECFA 1/4/05

---

Theory driven research designs - 2. Experimental designs
res-design-quant-vi-9
2.4 Examples
Exemple 2-2: Under which conditions does animation favor learning ?
Master (DESS) thesis by Cyril Rebetez, TECFA 2005
Note: Funded by a real research project, i.e. the student did more than usually expected !
Research question
"Notre recherche a pour objectif de mettre en évidence l'influence, de la continuité du flux,
de la collaboration, de la permanence des états antérieurs, ainsi que de vérifier la portée
de variables individuelles telles que l’empan visuel et les capacités de rotation mentale."
(p.33)
• This objective is then further developed through 1 1/2 pages in the thesis. Causalities are
discussed in verbal form (p. 34-40) and then "general" hypothesis are presented on 2 pages.
Explanatory (independent) variables, i.e. conditions
1. Animation, static vs. dynamic condition: allows to visualize transition between states.
Static presentation forces a student to imagine movement of elements.
2. Permanence, present or absent condition: If older states of the animation are shown,
students have better recall and therefore can more easily build up their model.
3. Collaboration, present or absent condition: Working together should allow students to
create more sophisticated representations.
Research Design for Educational Technologists
© TECFA 1/4/05

---

Theory driven research designs - 2. Experimental designs
res-design-quant-vi-10
Operational hypothesis (presented in the methodology chapter):
Quotations from the thesis:
• Animation
• Les scores d'inférence ainsi que les scores de rétention seront plus élevés en condition dynamique
qu'en condition statique.
• La charge cognitive perçue sera plus élevée en condition dynamique qu'en condition statique. Les
temps de discussion ainsi que les niveaux de certitude n'ont pas de raison d'être différents entre les
conditions.
• Permanence
• Les participants en condition avec permanence auront de meilleurs résultats aux questionnaires que
les participants en condition sans permanence. Les résultats d'inférence sont tout particulièrement
visés par cet effet.
• La charge cognitive perçue ne devrait pas être différente entre ces deux conditions. Les temps de
discussion ainsi que les niveaux de certitude devraient être plus élevés avec que sans permanence.
• L'influence de la permanence sera d'autant plus grande si les participants sont en condition de
présentation dynamique.
• Collaboration
• La collaboration aura un effet positif sur l'apprentissage, autant en ce qui concerne la rétention que
l'inférence. Toutefois, l'inférence devrait être tout particulièrement avantagée en cas de " grounding ".
Les participants en duo auront donc de meilleurs scores que les participants en solo.
• La charge cognitive perçue devrait suivre le niveau de résultat et être plus bas en condition duo qu'en
solo.
• Les temps de discussion devraient être naturellement plus grand en condition duo. Les niveaux de
certitude devraient également s'élever en condition duo face à la condition solo.
Research Design for Educational Technologists
© TECFA 1/4/05

---

Document Outline

• Theory driven research designs
  • 1. Theory driven research
  • 2. Experimental designs
    • 2.1 The scientific ideal
    • 2.2 The simple experiment in human sciences
    • 2.3 The simple experiment with pretests:
    • 2.4 Examples
  • 3. Quasi-experimental designs
    • 3.1 Interrupted time series design
    • 3.2 Threats to internal validity
    • 3.3 Non-equivalent control group design
    • 3.4 Validity in quasi-experimental design
  • 4. Statistical designs
    • 4.1 Introduction to survey research
    • 4.2 Levels of reasoning within a statistical approach
    • 4.3 Typology of internal validity errors
    • 4.4 Survey research examples
  • 5. Similar comparative systems design
  • 6. Summary of theory-driven designs discussed

---