Where’s your evidence?

Monday 11 May 2009

by Scott Thornbury

“Where’s your research evidence?” The question comes up periodically (and it came up last night in a discussion on Dogme at Dennis Newson’s villa in Second Life).  That is to say, is there any evidence that a Dogme approach does what it claims to do?

It’s a fair question, and any pedagogical innovation should be able to answer its critics with credible evidence. (Of course, there are always critics who will dismiss any research evidence, if they’ve got a mind to, but that – as they say – is their problem).

What kind of evidence would be needed, then? In the past, method analysts used test data to compare the effectiveness of different methods.  But test scores are inconclusive, as it’s difficult to control for all the variables that might have affected the outcome. As Richards (1990) observed, “studies of the effectiveness of specific methods have had  a hard time demonstrating that the method itself, rather than other factors, such as the teacher’s enthusiasm, or the novelty of the new method, was the crucial variable” (p. 36).

Nowadays researchers prefer to observe actual classes, using different kinds of instruments to capture evidence of the behaviours that are likely to influence learning, such as interaction, feedback, or “language related episodes” – i.e. moments during a communicative activity where learners pay explicit attention to a feature of the language. Many of the arguments supporting task-based learning (TBL), for example, are based on these kinds of findings. And since a Dogme approach shares affinities with TBL, these findings would seem to be relevant.

It’s been found, for example, that when learners are engaged in communicative tasks they tend to interact, negotiate, produce output, scaffold one another’s utterances, demonstrate fluency, and so on (Skehan 1998; Edwards and Willis, 2005). Likewise, studies of classrooms (not just language classrooms) where the interaction is modelled more closely on naturally-occurring conversation (another Dogme principle) suggests that conversation also embeds similar kinds of interactional adjustments (Ernst, 1994; Nakahama, et al. 2001; Bannink 2002).  More than in classrooms which are not task- or conversation-based?  Well, again, it’s difficult to compare across classes. But there’s been quite a lot of research to suggest that certain kinds of tasks do promote these behaviours more than others (e.g. Bygate, et al. 2001; Ellis, 2003).

So what?  How do we know that conversational interaction, meaning negotiation, output, scaffolding, fluency, and so on, are good for you? This is where a certain leap of faith is required. We still don’t know what are the optimal conditions for language learning. But I take heart from Rod Ellis’s statement (in his monumental Study of Second Language Acquisition, 1994) to the effect that “it is not yet clear which kind of instruction works best but there is evidence to suggest that focusing learners’ attention on forms, and the meaning they realize in the context of communicative activities, results in successful language learning.”  That is to say, where ‘language-related episodes’ are embedded within a communicative process, successful language learning results. This is an approach that is central to Dogme methodology.

So far, we haven’t mentioned affective factors, such as motivation. Research into learners’ attitudes suggests that learners are more motivated if content is personalised (Dörnyei and Csizér, 1999), another basic Dogme tenet. Again, you then need to show that motivation positively affects learning, but this is virtually a given.

An area, though, that is under-researched, but would be of crucial interest to a Dogme teacher, is whether there is any correlation between low materials use and the kinds of behaviours I have mentioned (interaction, meaning negotiation, personalisation, etc). Again, it would not be easy to set up a study in which all other possible variables were controlled. This is where an ethnographic study, based on learner diaries and interviews, alongside classroom observation and interaction analysis, might be more feasible, and at least provide some rich data on learners’ attitudes. It would be reassuring if learners were able to confirm our own intuitions that Dogme, if not more effective, is more engaging, more memorable, more motivating – more fun!

References:

Bannink, A.  (2002) Negotiating the paradoxes of spontaneous talk in advanced L2 classes. In Kramsch, C. (ed.) Language acquisition and language socialization. London: Continuum.

Bygate, M., Skehan, P and Swain, M. (eds.) (2001) Researching Pedagogic Tasks: Second Language Learning, Teaching and Testing. Harlow: Pearson.

Dörnyei, Z., and Csizér, K. (1999) Ten commandments for motivating language learners: results of an empirical study. Language Teaching Research.

Edwards, C., and Wills, J., (2005) Teachers Exploring Tasks in English Language Teaching. Palgrave Macmillan.

Ellis, R. (1994) The Study of Second Language Acquisition. Oxford: Oxford University Press.

Ellis, R. (2003) Task-based Language Learning and Teaching. Oxford: Oxford University Press.

Ernst, G. (1994) “Talking Circle”: Conversation and negotiation in the ESL Classroom. TESOL Quarterly, 28, 293-322

Nakahama, Y., Tyler, A. and van Lier, L. (2001). Negotiation of meaning in conversational and informational gap activities: A comparative discourse analysis. TESOL Quarterly, 35, 377-406.

Richards, J. (1990). ‘Beyond Methods’. In The Language Teaching Matrix. Cambridge: Cambridge University Press.

Skehan, P. (1998) A Cognitive Approach to Language Learning. Oxford: Oxford University Press.

 

 

 

12 responses to Where’s your evidence?

  1. Rob says:

    I feel I’ve done that ethnographic study, as I’m sure have many others on the dogme list, and elsewhere. It’s likely to remain unpublished, even unscientific, as long as we teachers are trained to rely on materials and reductionist methods, not to mention atomistic testing schemes, all of which lead us further away from the type of research that would “prove” the value of a dogmetic approach. Research is never conclusive, only indicative.

  2. Sara Hannam says:

    Dear Scott and Rob,

    For any theory or approach in language teaching or elsewhere to be meaningful for all those who feel it is worth its salt, it needs to be explored more closely in different contexts and situations. Not just from a “how we do it” point of view, or a “why do it?” but also from a “does it work?” point of view – this means exploring when it doesn’t and seeing that data set as just as valuable as when it does work. What Scott has outlined above is how existing areas of ELT might help in a better understanding of DOGME – and he has flagged up here that it would be good to develop focused DOGME research with a more ethnographic focus – what Rob seems to suggest is that it is hard to “prove” its success if people think of research in simplistic terms i.e. scientifically provable data. Well that is true in all areas of ELT, and certainly it is still the case that new types to research are difficult to implement, but certainly not impossible anymore. What I would like to see, personally, as someone interested in different approaches to teaching and also in research, is a collection of perspectives from different localities and contexts which talk about how DOGME has been incorporated into their teaching/learning, and asking questions about how DOGME fits into their overall teaching framework. The most productive way to do this would *not* be to take the quantitative “let’s see if we can replicate this approach exactly in XXX different contexts” as this is bound to produce over-simplistic results which will not get to the nuances of the way things work or don’t work. It would be much better to try looking at situations in different countries/contexts and allow the local researcher to decide on what aspect of DOGME they want to draw out in their research. There is plenty of this sort of research going on in ELT these days so its not that unusual and its aim is depth and exploring contradiction rather than producing nice, neat results that can be squeezed into a pie chart here or a graph there.The point is not to produce the final word on something, but to see it as an ongoing project that will always produce new findings. Learner and teacher attitudes would be captured in this way, as well as perspectives of other people/processes involved offering valuable classroom and non-classroom points of view.

    Scott, in your IATEFL Online Interview, you say that you see yourself as a mediator between the academic and teaching world. That you try to make theories simpler for teachers to understand (I am paraphrasing here and hope I am reflecting what you wanted to convey). I wish that we didn’t have to see the link between research and practice as such a top down affair – but unfortunately you are right in the sense that classroom teachers don’t always see how research is relevant to them and sometimes academics lose sight of what it means to be in the classroom. But the thing is that we *need* both and I see no reason why we cannot wear different hats by being both teachers and researchers. So, now that the practical “how to” materials are out there and in circulation, it seems that the missing link is exploring this a bit more and bringing it together in a way that enables people to puzzle over the different parts of the jigsaw that is DOGME, from multi-perspectives, including helpfully critical voices that might identify its gaps. As I said, a nice edited collection from different authors would be a helpful addition at this stage.

    Sara – Greece

  3. Good points, Rob. It’s true, the Dogme discussion-list contains a wealth of first-hand accounts from teachers (especially yours, Rob). What’s perhaps lacking is first-hand accounts from learners.

    (And from out-of-work coursebook writers?)

  4. Seth Dickens says:

    Hi Scott,

    Some interesting points for me to consider as a teacher (be that a non-Dogme one) and an interesting comment from Rob, too.

    I have liked and admired the ideas and ideals behind the Dogme “movement” from afar for some time now. I can definitely recognise that many of my most enjoyable teaching moments have been the completely unplanned discussions that have morphed into an entire lesson driven by the interests and needs of the students. Dogme it may well not have been, but there was something electric there.

    I’ll admit that “evidence” that something works, or doesn’t work is of no real interest to me at this point in time. I prefer to try something out and “feel” if it seems right or wrong. Personally speaking I think digging one’s heels in for any particular methodology is not going to help teachers in general and, more importantly, students in the long run. Nevertheless, I follow the threads of this and other discussions going on around the eflblogosphere with interest right now.

    This is especially so, as I am in a phase of non-textbook teaching at the moment (have stopped my teaching job at a language school to work at a state school on a really interesting programme.) Personally speaking I still would feel uncomfortable to go into the class with nothing in my head, or in my hand… guess I should follow the Dogme list to see if there are ideas there that might inspire.

    I wonder what Rob means by “atomistic testing methods” could that be Cambridge ESOL et al, or does he mean the tests used to evaluate the efficacy of Dogme? I’ll grant that I’ve had my problems with the whole KET – CPE ladder in the past. Feels a little like hoop jumping to me.

    Well for now I shall remain happily interacting with my students in the (non-Dogme) way I have been enjoying up till now. If nothing else your points and the general discussion going on around these issues has given me pause for thought, which can never be a bad thing.

    Thanks,

    Seth.

  5. I ought to also mention the study I referred to on the SL discussion, about “topicalisation” and “uptake” (Slimani 1989). Slimani found that “whatever is topicalized by the learners, rather than the teacher, has a better chance of being claimed to have been learned”. That is to say, the learners’ topics have a more enduring effect on memory (their “uptake”) than topics coming from other sources (presumably also coursebooks). This is a finding that very much accords with Dogme principles.

    Sliman, A. (1989) The role of topicalization in classroom language learning. System, 17.

  6. Rob says:

    Hi Seth, and Sara, I enjoyed reading your comments. Somewhere, I read a text by Scott about atomistic grammar (or something like that), but, given the long day behind me, my powers of perception and recall are weak. Suffice it to say, ‘atomistic testing schemes’ would be those tests designed to require learners to regurgitate or identify bits of information with no real context or meaning around them. An example is a gap-fill at the sentence level which asks the text taker to choose between the simple past and present perfect.

    Sara, I imagine one could edit the Dogme discussion list into a sort of ‘data set’ of multiple perspectives including criticism and concrete examples of ‘how it works’ in various contexts. It’s only a matter of mining the archives.

    On to read the Robinson Crusoe debate…

  7. Bruno says:

    The comprehensible output (CO) hypothesis which states that we acquire language when we attempt to transmit a message but fail and have to try again as put forward by Merrill Swain (Swain, 1985) also really points into the dogme direction.

    For more details:
    Swain, M. (1985) Communicative competence: Some roles of comprehensible input and comprehensible output in its development. In Gass, S. and Madden, C. (Eds.), Input in Second Language Acquisition, pp. 235-256. New York: Newbury House.

  8. Adam says:

    So, two years on from this article’s publication. Anyone have any evidence yet?

  9. Hi Adam – not a lot, but the good news is that (a) a teacher in the UK is setting up an action research project to monitor the effects of teaching unplugged, starting September; (b) a teacher in the US has been conducting some classroom-based research along the same lines, but has yet to present his findings, and (c) I met a woman in Oman last month who is planning a doctorate on Dogme! Watch this space!

  10. Bob Mudford says:

    As a teacher who uses dogme 100% in one-to-one classes I have often wondered about the question of “evidence” in the scientific sense.

    How would one set up a double blind test of any two teaching systems which would produce unambiguous evidence?

    You would need three groups of students with more or less identical levels of motivation, innate ability and initial knowledge of the target language.

    You would then need three equally able enthusiastic and committed teachers who would wholeheartedly support the system they were to teach.

    One group would be taught (for example) grammar translation, another dogme and the final control group would be taught basket weaving.

    One would then test the outcomes by various means at the end of 6 months, one year and two years. One would then set up various subsequent tests in order to get results for all the other methodologies. Or do them all at the same time.

    Obviously one could not realistically set this up. Human beings – both students and teachers – are simply too variable and, well, too human, to allow this to work.

    But I’m not sure where this leaves us – how can you tell that any system is “better”?

  11. Thanks, Bob, for that – you sum up the problems of doing method comparison research very well. As I quoted Richards as saying “studies of the effectiveness of specific methods have had a hard time demonstrating that the method itself, rather than other factors, such as the teacher’s enthusiasm, or the novelty of the new method, was the crucial variable” .

    This doesn’t mean it’s a complete waste of time, but only that any results will be ‘suggestive’ rather than convincing – which is probably true of any research in the social sciences.

    In the end, whether or not a system is ‘better’ than any other is going to come down to the users of the system – in this case both teachers and learners. If they THINK it’s better, that may be the best we can hope for!

  12. […] Delta Publishing Blog. Consultado el 05/12/12. 12.    Thornbury, Scott (2009-05-11). “Where’s your evidence?“. Delta Publishing Blog. Consultado el 05/12/12. 13.    Meddings & Thornbury 2009, […]

Leave a Reply

Your email address will not be published. Required fields are marked *

Delta Development Blog

This blog will be updated at least once a week, so add it to your bookmarks. You can also subscribe to the feed to be notified when it's updated.

Meet the Bloggers

Culture in our Classrooms Cover

Culture in our Classrooms

Part of the Delta Teacher Development Series. Culture in our Classrooms acknowledges the role of culture in the English Language Teaching classroom and provides lesson content which is relevant, useful and engaging for students.

The Business English Teacher Cover

The Business English Teacher

From the multi-award-winning DELTA TEACHER DEVELOPMENT SERIES. The Business English Teacher is a book not only for teachers who are thinking of making a career move into the field of business English teaching but also for those who would like to increase their skills and develop their potential.

Teaching Online  Cover

Teaching Online

Teaching Online is essential reading for any teacher interested in online teaching and course delivery. It deals comprehensively with both the tools and the techniques necessary for online language instruction.

Digital Play  Cover

Digital Play

DIGITAL PLAY - 2012 ELTONS WINNER IN INNOVATION IN TEACHER RESOURCES! Digital Play is a pioneering book on the use of computer games in language teaching. Authors Kyle and Graham are experts in teaching with technology and training teachers in innovative classroom practice.

The Company Words Keep  Cover

The Company Words Keep

Part of the multi-award-winning Delta Teacher Development Series. The Company Words Keep is a practical and thought-provoking guide for language teachers, showing how the latest insights into “language chunks” can lead to learners acquiring natural and fluent English.

The Developing Teacher  Cover

The Developing Teacher

The Developing Teacher has been awarded the 2009 Duke of Edinburgh/ESU Award for Best Entry for Teachers. The Developing Teacher suggests that teachers themselves are the most powerful agents of change and development in their own professional career.

Teaching Unplugged Cover

Teaching Unplugged

Teaching Unplugged was awarded the British Council 2010 ELTons UK Award for Innovation. Teaching Unplugged is the first book to deal comprehensively with the approach in English Language Teaching known as Dogme ELT.

Being Creative Cover

Being Creative

Part of the Delta Teacher Development Series. Being Creative takes you on a journey that reveals how all teachers have the potential to become creative. Whether you are experienced or new to the classroom, Being Creative allows your teaching to take flight.

The Book of Pronunciation   Cover

The Book of Pronunciation

Part of the multi-award-winning Delta Teacher Development Series. The Book of Pronunciation is a definitive account of the key role pronunciation plays in teaching and learning, providing a highly authoritative but hugely accessible overview of the essential elements of English pronunciation as well as a wide range of classroom practice.

Previous

Next

Archives by date

  • November 2016
  • October 2016
  • July 2016
  • June 2016
  • May 2016
  • April 2016
  • March 2016
  • February 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • July 2015
  • June 2015
  • May 2015
  • April 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • November 2014
  • October 2014
  • September 2014
  • August 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • February 2014
  • January 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • June 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • January 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012
  • July 2012
  • June 2012
  • May 2012
  • April 2012
  • March 2012
  • February 2012
  • January 2012
  • December 2011
  • November 2011
  • October 2011
  • September 2011
  • August 2011
  • July 2011
  • June 2011
  • May 2011
  • April 2011
  • March 2011
  • February 2011
  • January 2011
  • December 2010
  • November 2010
  • October 2010
  • September 2010
  • August 2010
  • July 2010
  • June 2010
  • May 2010
  • April 2010
  • March 2010
  • February 2010
  • December 2009
  • November 2009
  • October 2009
  • September 2009
  • August 2009
  • July 2009
  • June 2009
  • May 2009
  • April 2009
  • March 2009