EMF5 Day 2 | Jane Allemano: Authenticity in Speaking Tests | Talk summary

My final talk summary from E-merging Forum 5 is a summary of a plenary talk by Jane Allemano on assessing speaking, in which she analyzed what it means to be ‘a good speaker’, compared that to what’s tested in many modern exams and suggested how we can bring assessment closer to what we need to be testing. 

Two questions to ask about a test:

• Does the test test what it’s supposed to test?
• Does it have a positive effective effect on what happens in the classroom?

Before we can answer the first question, we need to understand what it is that we need to test.

What we need to test when assessing speaking

Ostensibly, the factors that we need to take into account (and they are assessment criteria of most modern high-stakes exams, e.g. IELTS or Cambridge Exams suite) are:

• Grammar
• Vocabulary
• Pronunciation (though this is a contentious thing – We don’t need to be native-like, but what is comfortable intelligibility – which depends on the listener?)
• Coherence and cohesion – organization of speech, ability to produce a long utterance
• Interactive competence – responding, turn-taking, agreeing-disagreeing, listening, body language

However, are they all equally important? What’s also crucial?

You talk to your friends differently from the way you talk to your boss, which is an example of choices we make while speaking. Where does that fall in the criteria outlined above? Nowhere, it’s absent.

These choices are studied by pragmatics: 

• ‘The study of how meaning is created in context’ McCarthy (1991) Discourse Analysis for Language Teachers
• Context dependent rather than context independent’ Levinson SC (1983) Pragmatics Cambridge:CUP

Let’s look at those choices from the perspective or Hyme’s Speech acts.

Act: a joke
Event: conversation
Situation: Party
Community: friends
Outcome: easy to make a joke because there’s shared understaning of what counts as a joke

Act: a joke
Event: presentation
Situation: International conference
Community: the audience
Outcome: you might choose not to make a joke because you’ve got no understanding of what counts as a joke

Here are some other pragmatic functions that me might want to realize in daily communication:

we do	in order to gain
amuse/impress	appreciation
inform	understanding
promote action	support
exchange	cooperation, you’ll want your listeners to take part in the conversation.

Doing this successfully is not just about semantics.

Take, for example directness in Russian vs English. Living in Russia, J.A. was getting increasingly annoyed with the Russian way of giving advice because it came across as an order.

• The Russian way of giving advice: ‘You need to take the dog to the vet’ (not even saying ‘I think’). ‘You don’t eat enough meat.’
• The English way of saying this: I’ve just noticed that the dog is limping a bit. Maybe you’ve seen this? I think that maybe you might want to take it to the vet.

(J.A. mentioned that there’s a PhD thesis on that, if you’d like more detail.)

Factors that inform good choices: 

• physical context (time and space)
• social context (status, relationship, shared knowledge)
• function (persuasion, entertainment, advice)
• culture (convention – e.g. Relevance, organization).

So what do good speakers have?

• Socio-cultural knowledge
• Genre knowledge
• Discourse knowledge…
• [and so on]

and use appropriate

• framing
• register
• tone
• [and so on]

What do we in fact test?

Let’s go back to Speech Acts to to analyze a typical exam situation.

Act: giving an opinion
Event: Discussion
Situation: Exam
Community: judged and the judged

Emerging issues. In an examiner-test-taker interview:

• restricted range of types of talk
• unbalanced power distribution (examiner manages the interaction, suddenly saying ‘Thank you’ and asking you another question)
• interlocutor frame (examiners adhere to a script so that all candidates get the same language from the examiners and so that; examiners can’t respond, even backhannel or encourage)
• focus of the examiner (how you’re speaking, not what you’re saying).

An answer:

A paired or group format?

Pluses:

• reflects the real world (most of the time you’re speaking to your peer);
• encourages interaction in the classroom, so this has positive washback on the classroom;
• allows a broader range of language functions because candidates are not just answering question;
• creates more speaking time for the candidates; allows more interactional symmetry.

Is this a fair test? Not always. Research findings:

• The test takers have more turns, compared to the examiners
• The test takers has more opportunities for initiating topics
• There is more collaborative development of topics
• The effect of background variables is higher
• There’s more variety of speech functions

Variables:

• language level
• personality (extrovert/introvert), familiarity (husband and wife might be candidates – lots of shared knowledge that they don’t overtly say, so they say less + they’re not used to speaking to each other in English)
• cultural capital
• gender
• status

However, the significant amount of research show that those factors don’t make that much difference. Also, one might argue about these variables that this is life!

So, suggested approach:

• A range of task types, only one of which is paired
• Use of analytical and global criteria and scales

Further issues

Does introducing paired tasks make the test situation completely life-like? Not quite. If we look back at pragmatic intentions of everyday life, they are to amuse, inform, instruct, etc. However, during the test, the candidate’s intention is to demonstrate linguistic and socio linguistic competence (Simpson 2006), and this mismatch might affect the candidates’ level.

To demonstrate, Jane Allemano gave an example of a test she observed. Candidates were given the task to describe the place where they lived. They were clearly not interested neither in the topic nor in each other and were going through the motions of answering the question. However, at one point of the conversation, one candidate mentioned that they moved from a certain London neighbourhood because it wasn’t safe. Another candidate got interested, because they were staying in that same neighbourhood. Interestingly, as soon as the discussion turned into a genuine one, with candidates reassuring each other, their level of English went up.