Where’s your evidence?

Monday 11 May 2009

by Scott Thornbury

“Where’s your research evidence?” The question comes up periodically (and it came up last night in a discussion on Dogme at Dennis Newson’s villa in Second Life).  That is to say, is there any evidence that a Dogme approach does what it claims to do?

It’s a fair question, and any pedagogical innovation should be able to answer its critics with credible evidence. (Of course, there are always critics who will dismiss any research evidence, if they’ve got a mind to, but that – as they say – is their problem).

What kind of evidence would be needed, then? In the past, method analysts used test data to compare the effectiveness of different methods.  But test scores are inconclusive, as it’s difficult to control for all the variables that might have affected the outcome. As Richards (1990) observed, “studies of the effectiveness of specific methods have had  a hard time demonstrating that the method itself, rather than other factors, such as the teacher’s enthusiasm, or the novelty of the new method, was the crucial variable” (p. 36).

Nowadays researchers prefer to observe actual classes, using different kinds of instruments to capture evidence of the behaviours that are likely to influence learning, such as interaction, feedback, or “language related episodes” – i.e. moments during a communicative activity where learners pay explicit attention to a feature of the language. Many of the arguments supporting task-based learning (TBL), for example, are based on these kinds of findings. And since a Dogme approach shares affinities with TBL, these findings would seem to be relevant.

It’s been found, for example, that when learners are engaged in communicative tasks they tend to interact, negotiate, produce output, scaffold one another’s utterances, demonstrate fluency, and so on (Skehan 1998; Edwards and Willis, 2005). Likewise, studies of classrooms (not just language classrooms) where the interaction is modelled more closely on naturally-occurring conversation (another Dogme principle) suggests that conversation also embeds similar kinds of interactional adjustments (Ernst, 1994; Nakahama, et al. 2001; Bannink 2002).  More than in classrooms which are not task- or conversation-based?  Well, again, it’s difficult to compare across classes. But there’s been quite a lot of research to suggest that certain kinds of tasks do promote these behaviours more than others (e.g. Bygate, et al. 2001; Ellis, 2003).

So what?  How do we know that conversational interaction, meaning negotiation, output, scaffolding, fluency, and so on, are good for you? This is where a certain leap of faith is required. We still don’t know what are the optimal conditions for language learning. But I take heart from Rod Ellis’s statement (in his monumental Study of Second Language Acquisition, 1994) to the effect that “it is not yet clear which kind of instruction works best but there is evidence to suggest that focusing learners’ attention on forms, and the meaning they realize in the context of communicative activities, results in successful language learning.”  That is to say, where ‘language-related episodes’ are embedded within a communicative process, successful language learning results. This is an approach that is central to Dogme methodology.

So far, we haven’t mentioned affective factors, such as motivation. Research into learners’ attitudes suggests that learners are more motivated if content is personalised (Dörnyei and Csizér, 1999), another basic Dogme tenet. Again, you then need to show that motivation positively affects learning, but this is virtually a given.

An area, though, that is under-researched, but would be of crucial interest to a Dogme teacher, is whether there is any correlation between low materials use and the kinds of behaviours I have mentioned (interaction, meaning negotiation, personalisation, etc). Again, it would not be easy to set up a study in which all other possible variables were controlled. This is where an ethnographic study, based on learner diaries and interviews, alongside classroom observation and interaction analysis, might be more feasible, and at least provide some rich data on learners’ attitudes. It would be reassuring if learners were able to confirm our own intuitions that Dogme, if not more effective, is more engaging, more memorable, more motivating – more fun!

References:

Bannink, A.  (2002) Negotiating the paradoxes of spontaneous talk in advanced L2 classes. In Kramsch, C. (ed.) Language acquisition and language socialization. London: Continuum.

Bygate, M., Skehan, P and Swain, M. (eds.) (2001) Researching Pedagogic Tasks: Second Language Learning, Teaching and Testing. Harlow: Pearson.

Dörnyei, Z., and Csizér, K. (1999) Ten commandments for motivating language learners: results of an empirical study. Language Teaching Research.

Edwards, C., and Wills, J., (2005) Teachers Exploring Tasks in English Language Teaching. Palgrave Macmillan.

Ellis, R. (1994) The Study of Second Language Acquisition. Oxford: Oxford University Press.

Ellis, R. (2003) Task-based Language Learning and Teaching. Oxford: Oxford University Press.

Ernst, G. (1994) “Talking Circle”: Conversation and negotiation in the ESL Classroom. TESOL Quarterly, 28, 293-322

Nakahama, Y., Tyler, A. and van Lier, L. (2001). Negotiation of meaning in conversational and informational gap activities: A comparative discourse analysis. TESOL Quarterly, 35, 377-406.

Richards, J. (1990). ‘Beyond Methods’. In The Language Teaching Matrix. Cambridge: Cambridge University Press.

Skehan, P. (1998) A Cognitive Approach to Language Learning. Oxford: Oxford University Press.

 

 

 

  1. 1

    Rob

    11 May 2009 14:38

    I feel I’ve done that ethnographic study, as I’m sure have many others on the dogme list, and elsewhere. It’s likely to remain unpublished, even unscientific, as long as we teachers are trained to rely on materials and reductionist methods, not to mention atomistic testing schemes, all of which lead us further away from the type of research that would “prove” the value of a dogmetic approach. Research is never conclusive, only indicative.

  2. 2

    Sara Hannam

    11 May 2009 15:50

    Dear Scott and Rob,

    For any theory or approach in language teaching or elsewhere to be meaningful for all those who feel it is worth its salt, it needs to be explored more closely in different contexts and situations. Not just from a “how we do it” point of view, or a “why do it?” but also from a “does it work?” point of view – this means exploring when it doesn’t and seeing that data set as just as valuable as when it does work. What Scott has outlined above is how existing areas of ELT might help in a better understanding of DOGME – and he has flagged up here that it would be good to develop focused DOGME research with a more ethnographic focus – what Rob seems to suggest is that it is hard to “prove” its success if people think of research in simplistic terms i.e. scientifically provable data. Well that is true in all areas of ELT, and certainly it is still the case that new types to research are difficult to implement, but certainly not impossible anymore. What I would like to see, personally, as someone interested in different approaches to teaching and also in research, is a collection of perspectives from different localities and contexts which talk about how DOGME has been incorporated into their teaching/learning, and asking questions about how DOGME fits into their overall teaching framework. The most productive way to do this would *not* be to take the quantitative “let’s see if we can replicate this approach exactly in XXX different contexts” as this is bound to produce over-simplistic results which will not get to the nuances of the way things work or don’t work. It would be much better to try looking at situations in different countries/contexts and allow the local researcher to decide on what aspect of DOGME they want to draw out in their research. There is plenty of this sort of research going on in ELT these days so its not that unusual and its aim is depth and exploring contradiction rather than producing nice, neat results that can be squeezed into a pie chart here or a graph there.The point is not to produce the final word on something, but to see it as an ongoing project that will always produce new findings. Learner and teacher attitudes would be captured in this way, as well as perspectives of other people/processes involved offering valuable classroom and non-classroom points of view.

    Scott, in your IATEFL Online Interview, you say that you see yourself as a mediator between the academic and teaching world. That you try to make theories simpler for teachers to understand (I am paraphrasing here and hope I am reflecting what you wanted to convey). I wish that we didn’t have to see the link between research and practice as such a top down affair – but unfortunately you are right in the sense that classroom teachers don’t always see how research is relevant to them and sometimes academics lose sight of what it means to be in the classroom. But the thing is that we *need* both and I see no reason why we cannot wear different hats by being both teachers and researchers. So, now that the practical “how to” materials are out there and in circulation, it seems that the missing link is exploring this a bit more and bringing it together in a way that enables people to puzzle over the different parts of the jigsaw that is DOGME, from multi-perspectives, including helpfully critical voices that might identify its gaps. As I said, a nice edited collection from different authors would be a helpful addition at this stage.

    Sara – Greece

  3. 3

    Scott Thornbury

    11 May 2009 16:07

    Good points, Rob. It’s true, the Dogme discussion-list contains a wealth of first-hand accounts from teachers (especially yours, Rob). What’s perhaps lacking is first-hand accounts from learners.

    (And from out-of-work coursebook writers?)

  4. 4

    Seth Dickens

    11 May 2009 21:17

    Hi Scott,

    Some interesting points for me to consider as a teacher (be that a non-Dogme one) and an interesting comment from Rob, too.

    I have liked and admired the ideas and ideals behind the Dogme “movement” from afar for some time now. I can definitely recognise that many of my most enjoyable teaching moments have been the completely unplanned discussions that have morphed into an entire lesson driven by the interests and needs of the students. Dogme it may well not have been, but there was something electric there.

    I’ll admit that “evidence” that something works, or doesn’t work is of no real interest to me at this point in time. I prefer to try something out and “feel” if it seems right or wrong. Personally speaking I think digging one’s heels in for any particular methodology is not going to help teachers in general and, more importantly, students in the long run. Nevertheless, I follow the threads of this and other discussions going on around the eflblogosphere with interest right now.

    This is especially so, as I am in a phase of non-textbook teaching at the moment (have stopped my teaching job at a language school to work at a state school on a really interesting programme.) Personally speaking I still would feel uncomfortable to go into the class with nothing in my head, or in my hand… guess I should follow the Dogme list to see if there are ideas there that might inspire.

    I wonder what Rob means by “atomistic testing methods” could that be Cambridge ESOL et al, or does he mean the tests used to evaluate the efficacy of Dogme? I’ll grant that I’ve had my problems with the whole KET – CPE ladder in the past. Feels a little like hoop jumping to me.

    Well for now I shall remain happily interacting with my students in the (non-Dogme) way I have been enjoying up till now. If nothing else your points and the general discussion going on around these issues has given me pause for thought, which can never be a bad thing.

    Thanks,

    Seth.

  5. 5

    Scott Thornbury

    11 May 2009 22:05

    I ought to also mention the study I referred to on the SL discussion, about “topicalisation” and “uptake” (Slimani 1989). Slimani found that “whatever is topicalized by the learners, rather than the teacher, has a better chance of being claimed to have been learned”. That is to say, the learners’ topics have a more enduring effect on memory (their “uptake”) than topics coming from other sources (presumably also coursebooks). This is a finding that very much accords with Dogme principles.

    Sliman, A. (1989) The role of topicalization in classroom language learning. System, 17.

  6. 6

    Rob

    27 May 2009 04:42

    Hi Seth, and Sara, I enjoyed reading your comments. Somewhere, I read a text by Scott about atomistic grammar (or something like that), but, given the long day behind me, my powers of perception and recall are weak. Suffice it to say, ‘atomistic testing schemes’ would be those tests designed to require learners to regurgitate or identify bits of information with no real context or meaning around them. An example is a gap-fill at the sentence level which asks the text taker to choose between the simple past and present perfect.

    Sara, I imagine one could edit the Dogme discussion list into a sort of ‘data set’ of multiple perspectives including criticism and concrete examples of ‘how it works’ in various contexts. It’s only a matter of mining the archives.

    On to read the Robinson Crusoe debate…