The validity of automated scoring software and its application in ELT contexts

This was the title of the closing plenary at this year's VUS-TESOL conference, given by Professor Timothy L. Farnsworth.  What follows is a summary of what he had to say.





What is automated scoring?

• Computer software that automatically assigns scores to writing or speaking samples.
• Essays can be assigned scores instantly by computer.
• Test takers can call a testing centre and take an oral test without speaking to a human.
• Scores can be reported instantly.
• Some level of feedback is given to test takers.
• There is a variety of software available.

How does a computer grade a test?

1.  Natural Language Processing (NLP)

• software identifies and counts linguistic features.
• software does not attempt to gauge content in any way.
• used for testing writing.

2.  Speech recognition

• software compares the speech sample to a large database of samples of the same test questions.
• faster responses are 'more fluent'.
• used for testing speaking.

E-rater (ETS)

• automated scoring of timed essays
• uses NLP
• currently used in a limited way to rate TOEFL and GRE
• used for formative assessment (e.g. TOEFL practice online)
• individual assessment
• students submit essays, receive scores and re-write them as many times as they want in order to improve their score

E-rater takes an essay and counts:

• the number of words
• the number of sentences
• the number of paragraphs
• sentence length
• the number of unique words used versus the total number of words (lexical diversity)
• the number of low-frequency words (lexical depth)
• the number of prompt-specific words (topic appropriateness)

The computer doesn't try to understand the essay, but it does look at grammar:

• dependent/independent clauses
• passive voice
• subject-verb agreement
• plurals
• sequencing words
• logical relations
• mechanics (punctuation, for example)

What is a good essay according to E-rater?

• It's long - longer is always better!
• It has a standard structure.
• It has many longer sentences with a lot of dependent clauses.
• It has many explicit organisational words.
• It has a lot of obscure vocabulary - for example, indubitably would score much higher than surely!
• It has a wide range of vocabulary.

This is not necessarily a good thing!  Good English writing is often simple, clear and concise.

What does E-rater not notice?

• Untruths
• Grammatical errors
• Lexical errors
• Flawed arguments
• Insanity!

Therefore, ETS doesn't use E-rater as the sole scorer for tests.  Rather, it is used as the second human in order to save money.  More than ten years of research hasn't solved the problems with E-rater - it's incredibly hard to get a computer to understand language!

Criterion

This is an E-rater application designed for in-class use.  Students' essays are instantly scored using E-rater software.  Students are given individual scores and extra resources to refer to about their errors.

Versant

This is the first fully automated oral language test used commercially.  It is a Pearson product.  The test is taken in a computer lab or over the phone (speaking to a computer).  The computer automatically rates the speech and produces scores.  It is used widely in business and increasingly in schools.  There are many versions with multiple uses and languages - for the aviation industry, for example.

The test is fifteen minutes long and includes:

• repeating sentences
• scrambled sentences
• oral multiple choice

All responses are totally scripted with only one possible right answer.  There is an optional 'free response' answer, but this is not scored.  Answers are scored on:

• fluency
• pronunciation
• sentence mastery
• vocabulary
• grammar

Speech is captured by microphone and compared to a large database of human-scored responses.  The database includes responses from native speakers from different countries, and English learners from different countries and of all proficiency levels.  Scores are given in the range of 'most similar' to the sample.

What is a good Versant response?

• It's fast (fluency score)
• It's clear
• It's accurate
• It has native-like pronunciation

This last criteria is the most contentious.  We talk about 'global English' now and, for most of us, comprehensibility is much more important than native-like speech.

What Versant doesn't measure:

• the range of vocabulary used
• extended speaking
• pragmatics - cultural awareness, for example
• the ability to interact with others

Advantages of these systems

Reliability

• computers don't get tired
• computers aren't biased for or against individuals
• scores are more consistent than with human raters

Practicality

• it's less expensive than using human raters
• scores and feedback are obtained instantly

Research shows that when test takers are 'acting in good faith', scores are roughly equivalent to those of human raters.  Even though the scores are very similar, however, they are arrived at in very different ways.

Problems

Automated tests can be 'gamed' or tricked.  Versant scores, for example, can be quickly raised by coaching.

Positive effects on teaching

• Students can get more and faster feedback.

Negative effects on teaching

• The form of the test can influence what happens in the classroom.
• Teachers tend to focus on what is tested at the expense of communicative teaching.
• There can be a decreased focus on the quality of the content.
• There can be an increased focus on grammatical accuracy and low-frequency vocabulary.
• There is more oral repetition in order to increase the students' speed of response.
• There is less time spent on developing critical thinking.
• There is a decreased focus on the pragmatic.

To conclude

Despite the obvious drawbacks, computer scored testing is in all our futures.

Power of the image: ways to use photographs in ELT

This was the title of a presentation at this year's VUS-TESOL conference, given by Paul Grainger from National Geographic Cengage.  What follows is a summary of what he had to say.





Using images as the basis for discussion

• Show an image quickly and then hide it.  Ask the students what they remember about it.
• Show the image for longer and pose questions - who are the people?, what are they doing?, what happened before the picture was taken?, what do you think happened next?, how does this picture make you feel?

 

The history of the image

• Cave drawings
• Printing press
• Typewriter
• Visual literacy

We have gone from text rich images to those where the picture is more and more important.

The power of the image

Today pictures are uploaded and shared on social media, giving them an immediacy and profound impact.  As the saying goes, 'a picture is worth a thousand words'.  As teachers, we need to exploit this, even at low levels when we can teach new vocabulary with images.  People think using images, so seeing comes before the use of words.  As Aristotle said, 'without image, thinking is impossible'.  We remember far more about texts if they are illustrated - the 'picture superiority effect'.

Images can be:

• surprising
• shocking
• funny
• sexy
• powerful
• disgusting
• frightening
• ambiguous

After 72 hours, we retain 65-70% of visual information, whereas we remember only 10% of something we've read or heard.  After a year, the retention rate remains at 65-70% for information we've seen, but drops to only 1% for written or aural information.

The best images to use in ELT are those which:

• provoke an emotional response
• arouse interest
• generate discussion

Ideas for using photographs

1. Captions - If you can make your students laugh, you can make them do anything!  Show them a photo of people or animals and ask them to imagine what they are saying.  This is a great warmer - it engages students and stimulates creativity.
2. Creative question and answer practice - Show pictures of people and get students to work in pairs to ask and answer questions about them.  The person answering the questions pretends he knows the people in the picture and invents a whole back story for them.  For lower level students, you could limit the questions to the grammar point being studied.  You can do this activity with any picture in the coursebook - before a listening, for example.
3. Personal photos - Tie these in with the topic of the unit.  Show your own photos before the start of the unit as an introduction.  Get students to ask questions about them.  Students can also use their photos.  This is very engaging for students as they get to know more about their teacher as well as being able to share personal stories with their classmates.
4. Introducing a new topic or new vocabulary - Use photographs to generate interest in a new topic or to provide a visual reminder for new vocabulary. 
5. Pre-listening/pre-reading - Get students to focus on an image before they do a reading or a listening.  This really helps with their prediction skills and can be particularly useful when helping students with exam strategies.

Advantages of using images

• It motivates students
• It makes the material more memorable leading to higher retention rates
• It is a natural approach
• You can present the usual in an unusual way
• It's fun!

Using PowerPoint for Teaching English

This was the title of a presentation at this year's VUS-TESOL conference, given by Khau Huu Phuoc.  What follows is a summary of what he had to say.






Teaching with information communications technology (ICT)

ICT involves using:

• mobile phones
• computers
• audio-visual systems
• the internet
• software, such as PowerPoint or Hot Potatoes

Teaching with PowerPoint

When using books, it's difficult for us as teachers to refer to a specific point nin a text.  With PowerPoint, we can:

• Make something appear - for example, in gapfills where we can show the answer so that students are clear about what is correct (we can make words appear in any order or we can number the gaps and then fill them by clicking on the number making the exercise very easy to follow), or when teaching new vocabulary, we can click on a part of a picture to make the word appear.
• Move something - for example, in matching sentence halves or in inserting relative clauses.  You can put in sound effects for right and wrong answers - audio feedback is very engaging for students.
• Make something disappear - we can hide answers, which is particularly good when doing vocabulary tests.  For example, we can make words disappear as a clock ticks down to zero.
• Use custom animations (although we must have a purpose for using animation!) - there are four types of animation available with PowerPoint - entrance, emphasis, exit, and motion path.

All of these tools can focus students' attention on what we want them to learn.  As teachers, we can highlight places in a text very easily.  It saves a lot of time writing on the board and is very engaging for students.

Flipping the classroom: using a blended learning approach to actively engage students inside and outside of class

This was the title of a presentation at this year's VUS-TESOL conference, given by Rebecca Fletcher.  What follows is a summary of what she had to say.

 

How have teaching and students changed in the 21^st century?

• Students take pictures of the whiteboard, rather than taking notes.
• Students’ attention spans are getting shorter.
• Teachers need more activities to engage students. 
• We now have interactive whiteboards.
• The classroom is much more student centred and collaborative.
• We use laptops and tablets rather than paper.

21st century student
 

What is blended learning?

 

Blended learning is teaching in the classroom mixed with learning outside the class.

Flipping the classroom – how does it work?

Flipping the classroom allows the school to become a place for talking, doing group projects and getting individual help from the teacher, and lets home become a place for doing pre-learning, such as watching instructional videos, and self-study.  Flipping what the student does means that they do the work ahead of time, come to class and debrief.  Students interact with the material before they come to class.  It empowers students to direct their own learning by coming to class ‘genned up’.

Why digital?

 

Digital is omnipresent in all aspects of life – we need to embrace it.  Flipping the classroom supports student centred learning and helps students make connections between the real world and the classroom.

What materials can be ‘flipped’?

• video
• audio files
• powerpoints
• documents
• images
• links to websites

It’s important to have a wide variety of materials.

Social networks

Use social networks to:

• support self-paced learning
• practise new language in an engaging environment
• maximise authentic input
• build a learning community

Using Facebook with your students

Students will be using English outside the classroom in an authentic way.

• Create a page for each of your classes.  Students can ‘like’ the page and comment on the links.
• Put images on Facebook and ask students, ‘what do you think will happen next?’ or ‘what happened just before this picture was taken?’
• Students post comments on an image and then come into class and discuss them.
• Students can write collaborative stories on Facebook.

For security, students should create a Facebook account to use only for students.

Where can you find materials to flip?

• You can create your own or, better still, get students to create them.
• Use publisher-created materials.
• Use ELT websites.

Recommended websites  

• BBC Learning English - especially 'The Flatmates', an ongoing soap opera with subtitles.  Students could watch it at home and then re-create scenes in class through role-play.
• EnglishCentral.com
• LyricsTraining.com 
• BreakingNewsEnglish.com
• Film-English.com

Learning Management Systems

LMSs or VLEs (Virtual Learning Environments) are software programmes which can be used to grade and monitor students.  One of the most well-known is Edmodo.  Teachers can assign work and track their students' progress.

Blogs

We have progressed from PPP (present, practise, produce) to PPPP (present, practise, produce, publish).  Blogs are student-centred with student-generated content.

To conclude

Flipping the classroom saves time both inside and outside of the class.

Digital inspirations for the young and not so young: motivating learners, motivating teachers

This was the title of the opening plenary at this year's VUS-TESOL conference, presented by Heather Barikmo and Marcus Artigliere.  What follows is a summary of what they had to say.





They addressed these two questions:

• What is motivation?
• How can current instructional technology shape motivation for both teachers and learners?

External versus intrinsic motivation

We have to consider both external motivation (the expectations laid down by the principal resources available) and intrinsic motivation (our own sense of curiosity and willingness to try new things).

Teachers give up on new technology more quickly if they are only subject to external motivation.  The desire to use new technology has to come from within the teacher.  The presence of the technology alone is not enough to motivate us.

Constructivist education

This is the idea that learners ultimately construct their own knowledge.  It is all about situated learning - that is, learning which is context related.

As teachers, we don't often use technology to construct learning - we tend to use it just as a tool.  It is often hard for us to change the way we teach to digital natives.

Practice

How and why should we use technology?  Some examples:

• Using i-Pads to discuss facts and opinions about animals.  For example, 'Can animals think?' - students create digital books using screenshots of PowerPoint presentations.  They personalise their learning and increase their efficacy.  It's engaging for the students because learning becomes much more self-directed.
• Developing digital stories - another way students and teachers can develop a constructivist approach to learning.
• Creating digital post-its of the phonetic pronunciation of new vocabulary.
• Writing notes on an IWB over a projected image.

Projects like this increase digital literacy and learners become more self-reliant.  They are invested in their own learning and results.  Students can also help teachers when using technology in class.  They can become the teachers.  This role reversal can be highly motivating for both students and teachers.

The constructivist approach with students

1. Vocabulary on the street - students use mobile phones to take photos of words they see or make notes on words they hear.  They then e-mail these to the teacher who makes a presentation of this 'found' vocabulary to show to the rest of the class.  In this way, students create the word lists rather than them coming from coursebooks.  This is very motivating for them.

2. Blogs as a class space - class blogs can be used by the teacher to share presentations with students and extend the classroom time.  They can also be used to practise web-based reading, with a focus on hypertext where students click on links to learn more about a topic and then come back to the original text.  Students can use the comments section of the blog to give feedback on classroom material.

3. Interactive maps - use mapping software to learn more about places discussed in class.

By using these ideas, students continue to look at course content long after a course has finished.

The constructivist approach with colleagues

Faculty blogs and wikis - wikis can be password protected to turn them into filing cabinets for the faculty.  They can also be developed to serve as a textbook.  Current staff can be given author rights to post on a faculty blog and these rights can subsequently be removed when a teacher leaves.

The idea of using technology in our classrooms is a global concept which can not be ignored!
 

8th VUS-TESOL Conference, City Hall, Saigon, 21st July 2013

 

Having attended the 7th conference last year, I was keen to go to this year's meeting, the theme of which was 'Incorporating Technology in ELT - Why and How?'  Whilst it didn't attract quite the same calibre of 'star' names as last year (Alan Maley was plenary speaker), the conference was very successful and packed full of interesting and useful presentations and workshops.  I will write up the most interesting sessions in a short series of blogposts to follow this one, but you can read a summary of the day's proceedings on the VUS website here.

Coaching and Mentoring in ELT

This was the title of the sixth in a series of monthly CPD webinars hosted by the British Council.  You can read more about the programme here.

This webinar was presented by Loraine Kennedy and what follows is a summary of what she had to say.

Loraine began by giving us some definitions:

Coaching

'Coaching is a developmental process by which an individual gets support while learning to achieve a specific personal or professional result or goal.'

Mentoring

'Mentoring is a means of providing support, challenge and extension of the learning of one person through the guidance of another, who is more skilled, knowledgeable and experienced, particularly in relation to the context in which the learning is taking place.'
                                                          Andrew Pollard, Reflective Teaching

Distinction between coaching and mentoring

• Coaches use questioning and listening techniques to bring out the full potential of the individual, whereas mentors act as advisors, suggesting new paths for the individual to take.
• To mentor effectively, you must possess an in-depth appreciation and knowledge of the subject on which you are advising.
• Often the relational positions of mentor and individual being mentored are equivalent to that of teacher and student.
• In a coaching event, the positional relationship is on a par as the coach's role is to create an environment for the individual to learn for themselves.
• A mentor is often an expert working with a novice.
• A coach is more for experienced people - they explore ideas together, but the individual comes to a realisation themselves.

Spectrum of coaching styles

A mentor is more likely to be at the directive end of the coaching spectrum, but non-directive styles are much more powerful and the learning gained is much more likely to stick.

Questions to ask

• Does your organisational culture support coaching?
• Are you part of a learning organisation?
• To what extent does your organisation support the growth of individuals?
• Does your organisation believe in collaboration?
• Does your organisation allow enough time for coaching to take place?
• What needs to change to allow the process of coaching and mentoring to thrive?

If you have the right environment, then coaching and mentoring can happen.

Coaching and mentoring situations in ELT

• Professional development plans
• Giving feedback on an observation
• Resolving work problems - fostering team spirit and maintaining individual morale
• Managing change
• Enhancing team effectiveness
• Counselling students on their progress and learning objectives

Qualities of an effective coach

• Patience
• Enthusiasm
• Honesty and integrity, including confidentiality and trust
• Friendliness
• Genuine concern for others
• Self confidence
• Fairness
• Consistency
• Flexibility
• Resourcefulness

Skills and abilities

• Communicating, especially listening
• Questioning
• Analysing
• Summarising
• Setting goals and objectives
• Establishing appropriate priorities
• Giving and receiving feedback
• Relating to people at all levels
• Planning and organising

Underlying principles of being a coach

1. Ensure you fully understand what coaching is - What is coaching as opposed to mentoring?  What is coaching as opposed to managing?  Be clear about when and why you're coaching.
2. Check your perspective on people - you always have to see the potential in people.  If you're dwelling on their past mistakes, or thinking that they're 'no-hopers', then you're not in the right frame of mind to be that person's coach.  You have to always be thinking about the coachee's strengths and how best to move him or her forward.
3. Learn and practise an effective coaching model - start out with a coaching model, but don't feel restricted by it.
4. Engage your boss - everyone in the organisation has to have bought in to the coaching idea.
5. Understand and value personality differences - we know that everyone's different and has different motivations.  We know that everyone is at a different stage of their career development.  We must value this and take each coachee from their own starting point.
6. Prioritise your time and stay focussed - time allocated for a coaching situation needs to be used for coaching!
7. Stop putting out fires - don't do things for your coachee!  Create a consciousness about being responsible for yourself.  A coach helps, but doesn't do!
8. Seek regular feedback - feedback should be given and received!
9. Listen, listen, listen!
10. Keep growing and developing - when you are a coach, you need a coach!! You'll be a better coach when you are being coached.  Things are constantly changing.  You never reach the finish line.
11. Be fully prepared - you always need to know where you got to in the last coaching session and how you're going to move forward.
12. Focus on progression, not punishment - what about the ones who don't want to progress?  What about the ones who've lost motivation?  If handled well, coaching can appeal to everyone.  It's about moving forward, not looking back.
13. Use the right questions
14. Remember that the coachee knows the answer already - if you keep quiet or explore the options with the coachee, it's amazing how many times they will come up with the answers themselves.
15. Break down big goals into little steps
16. There is no right and wrong - there are many rights!  My right may be different to yours, but both are equally valid.
17. Maintain forward momentum - keep focussed on solving problems.
18. Be a good coachee!

Ground rules for being a coach within a coaching relationship

• Clarify the purpose of the coaching relationship from the start.
• Set and agree the objectives for each 'coaching conversation'.
• Be honest.
• Build your understanding of the coachee's world - what stage of their career are they at?  How is their home situation?
• Build your understanding of the coachee's relationships - how do they fit into the team?  Do they get support from their family/their friends/their colleagues?
• Insist on action.
• Insist on accountability.

Peer coaching

• Identify a partner you trust and who trusts you.
• Ensure that both of you want to learn.
• Set a schedule for conversations.
• Each identify their learning objectives.
• Divide the time between coaching and being coached equally.
• Strive for objectivity, not empathy - empathy is important, but you need to be frank and honest.
• Avoid moaning and grumbling.
• Focus on positive action.
• Agree to hold each other accountable.

Tools and models

1. ADKAR change management model

It is important that the steps are carried out in the order shown.

 

2. GROW model of coaching

 

This is a good model to start with.

 

3.  Achieve coaching model

 

4. Reflective practice

5. High level listening skills

• Intend to understand
• Pay close attention
• Defer judgement
• Explore for deeper meaning and understanding
• Concentrate
• Don't interrupt inappropriately
• Get inside the other person's frame of reference
• Listen with your eyes and mind, not just your ears
• Listen for meaning and feelings
• Beware of your body language signals

Avoid being patronising

• Find out what the other person knows and feels, don't presume.
• Understand the person's character and style of working and respond appropriately.
• Be careful with words and expressions that can be interpreted negatively and watch the tone of your voice.
• Don't insult someone's intelligence even if they don't know something you know - don't make them feel small.
• Avoid using suggestion questions - e.g. don't you think you should answer the phone?
• Withhold unsolicited advice - wait to be asked or ask if you can make a suggestion.

Courses

• Coaching Academy
• Centre for Coaching
• Coaching.com

Books

• 'Coaching Questions: a Coach's Guide to Powerful Asking Skills' by Tony Stoltzfus available here.
• '101 Coaching Strategies and Techniques' edited by Gladeana McMahon and Anne Archer (Routledge)
• 'The Manager's Coaching Toolkit' by David Allamby (Pearson Business)
• 'Coaching for Performance: Growing Human Potential and Purpose' by John Whitmore (Nicholas Brealey Publishing)
• 'Reflective Teaching' by Andrew Pollard

Websites

• National framework for mentoring and coaching 
• Excel group development

Oxford Big Read - an introduction to setting up a class library and using readers

This was the title of a recent webinar hosted by Oxford University Press and presented by Verissimo Toste.  He based his talk on his experience of setting up a class library for 25 - 30 teenage students.  The idea was that students chose the books they wanted to read and did so at their own pace. They were encouraged to use readers as a tool to learn the language; the scheme was supposed to appeal to them as language learners, not as readers.  What follows is a summary of what Verissimo had to say.

The importance of reading to learn English

• Remember to tell your students regularly how important reading is.
• If you go to the gym and do exercise, you will get fitter.  You don't need to know what the muscles are or what anatomy means.  In the same way, if you read, you will learn - you don't necessarily need to know how every grammatical structure works.

Reading needs to be:

• voluntary
• routine - i.e. 15 minutes a day, not one hour once a week.
• beyond the classroom

Comfort leads to routine

• Stories must interest students from the beginning - they must choose the book.
• Stories must be appropriate for the level - that is, knowledge minus one. No more than 2 or 3 words per page should be new or difficult.  The student shouldn't need to use a dictionary.
• Establish a reading routine of 15 minutes per day.

Selecting stories

• There is a huge amount of choice out there!
• A library needs about 1.5 books per student.
• Get catalogues from publishers and use them as a reading comprehension exercise.
• Allow the students to choose the books.  Tell them to:
• look at the cover
• consider the title
• read the back cover
• look through the illustrations
• sit and read a page comfortably
• Use the 'find your level' page of the OUP website.

Reading in class

• Create a social environment.
• Use reading as a five-minute activity at the beginning of a class whilst you're setting up the room.
• Encourage students to pick up their readers if they finish an activity early
• Have ten or fifteen minutes of reading at the end of every class.
• Build a habit where reading becomes a part of every class - this routine may take up to three months to establish.
• Encourage students to talk about their books and share ideas - because everyone is reading a different book, they will want to talk to each other about them.
• Make students aware that these books have been written for them.
• Talk to students about where and when they read outside of class.
• Focus on the students who are reading and build the numbers up month by month.

Enthusiasm leads to involvement

 

Here are some reading related activities aimed at generatingi students' enthusiasm which will get them involved and lead to more language learning.

 

1. Posters

 

This is a good first activity.  Students make a poster of the book they are reading, to include the title, an illustration and a sentence or some key words. The posters are displayed in the classroom in order to help other class members decide which book to read next.

 

The posters don't have to be done on paper - they could be digital (using Glogster, for example) or they could be powerpoint slides.

 

2. Make a film poster from the book

• Give the book a new title?
• Who would be the stars - celebrities? classmates?
• What images could be used to best illustrate the book?
• How about making a trailer?

This is a very engaging activity which allows students to use their imagination.

 

3. Wordle

 

Make a word cloud from a text from the book.

 

4. Snap

• Students choose ten sentences from the story and copy them into their notebooks.
• Students decide which is the keyword in each sentence and underline it.
• Students write each sentence on a card without the keyword (like a gapfill).
• They write the keywords on different coloured cards.
• They play the game of 'snap' in pairs.
• The games relating to each book can be kept and re-used.

5. Speaking and interviewing a character

• Students choose a character from the book they are reading.
• They write questions to ask that character.
• They role-play interviewer and interviewee with a partner who has read the same book.
• Students can make up answers if all the information they need is not in the book.

6. Write a postcard to a character in the book

• Students read and reply to each other's postcards.

These activities create enthusiasm.

To conclude

Why should students read a lot?

• To extend their contact with the language
• To reinforce classroom language
• To contextualise language
• To increase motivation
• To expose students to new experiences
• To give them a feeling of achievement