FAQ

General

1. What is Visible LearningPlus?

Visible Learningplus is an in-depth school change model of professional learning and development. It is based on the principles of Visible Learning that have developed from John Hattie’s research and his two books: Visible Learning (2009) and Visible Learning for Teachers (2012). It takes the theory of this research and puts it into a practical inquiry model for schools to ask questions of themselves about the impact they are having on student achievement.

2. How should I use Visible Learning as an educator?

The book serves as a basis for discussion on using evidence to inform your school’s practice. One example might be on how feedback can be modified within the classroom. This can assist teachers to optimise their feedback and heighten students’ awareness of the benefits of effective feedback. It also creates awareness of how feedback might be getting through to students

3. What barriers are there to Visible Learning?

Firstly, there are many misconceptions about the Visible Learning research. Secondly, the mis- or over-use of an effect size alone in the Visible Learning research has produced negative commentary and we have found both to be a barrier to participation.

1. Why is the research focussed to one dimension of schooling i.e. student achievement?

The Visible Learning research has achievement as its focus. The research covers the typical range of achievement expected in schooling contexts. It includes many outcomes related to numeracy and literacy, as well as social studies, science etc. It includes many formats including standardized tests, state tests, teacher made, researcher made tests, etc. Of course there are many other outcomes of schooling such as retaining students’ interest in learning, affective, health outcomes as well as respect for self and respect for others. John has kept to one outcome as it is so important.

2. How do you allow for variability in design and quality for the studies gathered in the Visible Learning handbook? How can we be sure that the results are robust?

Others have investigated this issue and with one major exception quality of study is just not a big enough variable to make a difference to the conclusions. The difference is when the effect size is very small (<.10) then quality of the studies becomes more important. In the second book some of the low quality meta-studies have been discarded. Again, the effects of low and high quality were investigated and in general across the meta-analysis level (certainly not necessarily at the individual study level) it is rare to find quality making a difference (except when the effect is low).

3. Many of the high effects in the book: student feedback, linking prior and subsequent learning, going from known to unknown are from the early 1970s. Are we running the risk of just putting numbers to old ideas?

Research is re – search; perhaps you are right it is ‘searching again’. The book Visible Learning is a literature review. From that John Hattie aimed to extrapolate a story in the hopes of forming a contribution. The interpretations of these successful ideas from the 70’s are currently still not shared by all of our colleagues.

4. What are your thoughts about research that suggests that class size does matter?

Quote from John Hattie: This is NOT what I say. I summarise PAST experiences (as research focuses on what has happened in the past) and the preponderance of evidence shows that reducing class size DOES INCREASE achievement but the effect is small. Relative to other influences it is a small but positive effect.

The key question is to ask “why is the effect of class size so small, particularly given the many claims that we have all made about why it should be much larger”? If your arguments about the influences that really matter were implemented surely it would be easier in smaller classes. Smaller class sizes have been tried many times in many places but the effects remain small.

5. Most of the studies in the book were carried out in in highly developed English speaking countries. Is the research therefore applicable to non-English speaking, non-Western countries?

Yes, it is indeed the case that many of the studies are from developed countries and generalising to other countries needs much caution. Relative to high income countries, academic achievement in low-income countries can be more affected more by students’ social status and less by teacher quality. It is for this reason not recommended that this research is used in developing, non-western countries.

6. On which studies is the “pie” from Visible Learning based on?

The pie is based on John Hattie’s analyses of the sources of variance that impact on learning. Both the Best Evidence Synthesis by Adrianne Alton-Lee and the recent meta-analyses on apportionment of variance shows that the pie is reasonably representative of the various influences. Sometimes the point is missed which is that the largest source of variance we have control over relates to teachers, not schools. And some forget to mention the phrase “that we have control over”. Of course, the pie shows the largest source of variance is among students.

7. The table of 'effect sizes' in the book clearly identifies the effects of one element (e.g. direct instruction) and the effect size of another (d=0.82) on student achievement, but what consideration is given to the inter-correlation between those areas identified? As one example, teacher-student relationships has an effect size of d=0.72, and reducing class size an effect size of only d=0.21. However, what effect does reducing class size have on the ability for teachers to develop deeper relationships with their students?

We agree that the overlap between areas is critical. It is not as simple as combining the influences to get an increased effect. It is important to create a story about the underlying common parts such as success factors, student assessment capabilities, knowing thy impact etc.

1. Why has John Hattie changed the term “Self-reported grades” to “Student expectations”?

In the Visible Learning book (2009) the term self-reported grades was used. Since publishing that book the title of this influence has been changed to ‘student expectations’ (different from teacher expectations).

The notion is that students are very adept at predicting their performances which on the one hand is great, but too often this prediction sets the bounds to which they invest in the learning. It is our role as educators is to show them they can exceed their self-expectations.

2. What is the difference between ‘expectations’ and ‘student expectations’?

One is focused on student expectations of themselves and the other focuses on teacher expectations of students. Please refer to page 121 – 124 of the Visible Learning book (2009).

3. I understand the explanation that students are very good at predicting their performance in terms of their learning but don’t understand the mechanism or link that has caused such a large effect size. Does it mean that as a teacher I actually limit their learning when I don’t let students set appropriate and challenging learning goals? Do I as teacher need to nurture and encourage students to have high expectations?

Yes, aiming to raise student expectations about what they can achieve is the key and providing success (worked examples, etc.) can increase the expectations that students have, along with associate increased self-efficacy or confidence to reach these higher expectations. It is indeed not more complicated than that, but it is too rare as students learn that they need to hand something in, work till the bell, and many other performance (and not mastery) notions of learning.

1. How does a teacher in a classroom measure effect size and how exactly do you calculate it?

Please refer to 'calculating and effect size: a practical guide'. You can also view a video on how to calculate an effect size. Please go here: http://vimeopro.com/cognition/visible-learning-plus

2. I am not a mathematician. Is there an easier way to calculate an effect size?

Yes, however we do suggest you learn the manual version first.

Please visit:

http://visiblelearningplus.com/infohub and download the Visible Learningplus effect size add-in. This will allow you to easily calculate an effect size in excel. For instructions on how to use and install the add-in please visit our video channel: http://vimeopro.com/cognition/visible-learning-plus

3. If we were to look at using effect-sizes in our own schools, how would we calculate the standard deviation to make it easier?

We encourage you to watch one of our videos on ‘how to calculate an effect size’ which also covers how to calculate a standard deviation. There is also a very good paper on this topic (by Schagen and Hodgen).

Video: http://vimeopro.com/cognition/visible-learning-plus

Paper: http://www.educationcounts.govt.nz/publications/schooling/36097/4

4. Why do you use an effect size of d=0.40 as a cut-off point and basically ignoring effect sizes lower than d=0.40?

Effect sizes below d=0.40 aren’t ignored however a decision has been made to not look at what works (d=0.00 – 0.40) but what works best (d>0.40). Ninety percent of all effect sizes in education are positive (d > .0) and this means that almost everything works. The effect size of d=0.4 looks at the effects of innovations in achievement in such a way where we can notice real-world and more powerful differences. It is not a magical number but a guideline to begin discussion about what we can aim for if we want to see student change. (Hattie, p15-17)

5. What is the preferred timescale over which an effect size can be calculated?

Work in schools and subsequent analyses indicate that a minimum of an 8-12 week period is needed to show effects, if any, to be detected using effect-sizes.

6. Is there a bias when using effect sizes in favour of lower achieving students?

There could be a bias when the design of the test creates:

  • A ceiling which does not allow high achievers to show any progress.

  • When the students score close to zero on the first test, and if the pre and post-test measure are of different difficulty (e.g., a hard then an easy tests or vice versa).

7. What caution should I take when calculating an effect size?

Effect sizes are more accurate when calculated from a large sample. Thus caution should be taken when interpreting effect sizes from small samples (<30). One should also take particular caution when the sample group has outliers i.e. students with exceptionally high or low scores. When calculating an effect size it is best to compare your results including and excluding outliers to see if it makes a difference. When comparing pre-test and post-test scores, it is more useful to ensure that all students are tested and that scores from the same group of students are compared. Finally, one should always look at the context when interpreting effect sizes.

8. Why can an effect size of d=0.4 be gained in a shorter timeframe?

There are a few reasons:

  1. Teachers often chunk curriculum and teach it in bursts. Students usually only get exposure once to a topic. The assessments the students receive is related to these chunks. This means that d=0.4 is appropriate for shorter timeframes covering material students will only see once a year.

  2. Often the difficulty of the assessment(s) reflect the nature of material covered. This means that progress of d=0.4 can be expected.

 

1. What is Keller’s PSI?

Please refer to page 171 of the Visible Learning book. You can also visit this page: http://en.wikipedia.org/wiki/Keller_Plan

1. What defines 'achievement'? Is it student results in standardized testing or are there other measures?

Achievement relates to the subjects typically taught in schools, with the preponderance in math and reading (followed by Science, Social Studies) and fewer in the Arts. In terms of the Visible Learning research, there is a good mix of standardised tests, state tests, researcher constructed and teacher made tests.

2. Is it realistic to expect that the bottom 50% of learners improve?

Yes. Our expectation is that all students can and will improve and show progress over the course of a year.

The key is the mindset that we as teachers are change agents and so many are successful at changing students in positive directions. In any debate about national standards, the measurement of progress/improvement is the key. Some students may not “meet” the national standard but we expect them to be continually showing progress.

1. What is meta-analysis?

Meta-analysis is the use of statistical methods to combine results of individual studies. In this case the Visible Learning book synthesizes research based on what happens in classrooms for students from pre-school to tertiary.

2. What are some of the common problems with the use of meta-analysis?

In short there are 3 main issues with meta-analysis as described below.

  • You are comparing disparate studies so care is needed to consider possible moderators to any overall conclusion

  • Meta-analysis is focussed on seeking averages and thus it is important to allow for the variance or spread of results.

  • Studies only report past findings

Many of the above described problems have been well considered and please refer to the visible learning book for an explanation of these issues. (Hattie, p.10-12)

3. Why do you focus on averages and therefore ignoring the complexity of the classroom and the wide variety of results?

Classrooms are places of many complexities and while there are common themes sometimes averages may not do it justice. However the issue is that generalisability of the overall effect is an empirical issue and there are far fewer moderators* than are commonly thought (Hattie, p 10). It is important to appreciate that the 50,000 studies summarised in the book cover a very wide group of classrooms, with remarkable complexity and the aim of many of the studies was to understand these complexities. John further states: "Of course, I look forward to syntheses of qualitative research on classrooms to complement Visible Learning."

* In statistics, moderation occurs when the relationship between two variables depends on a third variable

 

1. What are the impacts of acceleration on student learning?

Kulik (2004) concluded in his research that accelerated students did just as well as the bright students in the grades into which they moved. Also accelerated students have higher educational ambitions. When looking at the effect size we see that accelerated students surpass the performance of the non-accelerated students of an equivalent age and intelligence by nearly one grade level (d=0.88).

One of the reasons it is one of the least used methods for gifted students is due to the typical claim that it is not beneficial from a social and interpersonal perspective. However it seems that not accelerating students has a negative social impact.

2. Does acceleration work?

Acceleration optimises the challenges whereas enrichment spreads kids sideways. There is no reason not to consider acceleration (timetabling and other logistic problems are often blamed). Please refer to the section “Acceleration” of the Visible Learning handbook for more information.

3. Do streaming and retention both have undesirable impacts on students?

Yes, both show very little progress in student achievement. Streaming shows an effect size of d=0.12 and retention d=0.16. They may enhance teacher working conditions but have very little impact on students.

1. How important is homework in the Visible Learning research and is there a difference between primary/elementary and secondary/high school?

Yes. There is a difference between homework in primary (d=0.15) and secondary education (d=0.64) where the effect size is a lot higher in secondary education. Overall homework shows an effect size of d=0.29 which means that there is no visible effect on student achievement.

2. Are you recommending that homework be abolished in elementary/primary schools?

No. Schools need to consider the type of homework they give. The evidence hints that the probability is that homework as we traditionally do it in primary schools may have a low impact so it is worth checking. Homework needs to be:

  • Linked to the lesson

  • Monitored by the teacher

  • Short in duration

  • Not include new learning as this disadvantages those who most need a professional teacher.

1. Why are some of the other school variables such as policies, buildings, school size and curriculum ruled out in the Visible Learning research?

They are not ruled out, however they are among the least beneficial influences on student achievement. The most powerful effects on the student relate to features within the school such as the climate of the classroom, peer influences, and the lack of disruptive students in the classroom, etc.

2. Where can I see the Visible Learning methods you describe in practice?

Many schools around the world have excellent Visible Learning practices in place. It is not possible for us to organise personal tours however we do provide though the tool kit and seminars video clips of good practice. If you are interested in a tour, please contact your local Visible Learning partner (click here) who may be able to help you.

1. What is the difference between transformational and instructional leaders?

Instructional leadership (d=0.42) refers to those principals who have their major focus on creating a learning climate free of disruption, a system of clear teaching objectives and high teacher expectations for teachers and students

Transformational leadership (d=0.12) refers to those principals who engage with their teaching staff in ways inspire them to new levels of energy, commitment, and moral purpose such that they work collaboratively to overcome challenges and reach ambitious goals.

It is Instructional Leadership that has the most effect on student outcomes.

2. What can a school leader do to shift the mind set of teachers and parents and make a difference?

Lots, but let's concentrate on one. The evidence shows that if school leaders see themselves as “evaluators” then they are likely to have the highest chance of shifting the mindframes of teachers towards greatest impact on student learning. These school leaders need to be clear about what it is that they want students to know and be able to do and knows what the success indicators are.

Constantly ask:

  • How do I know this is working?

  • How can I compare ‘this’ with ‘that’?

  • What is the merit and worth of this influence on learning?

  • What is the magnitude of the effect?

  • What evidence would convince you that you are wrong?

  • Where the evidence that shows this is superior to other programs?

  • Where have you seen this practice installed so that it produces effective results?

  • Do I share a common conception of progress?

By developing evaluation plans about the impact of teaching and learning, then your own effects can lead to others seeing the value of these questions, providing evidence about the impact on student learning and teachers teaching, and create a healthy respect for ensuring all making the difference.

1. What should I as an educator focus on?

The major theme of Visible Learning is “know thy impact” and this is what we would like educators to focus on.

In a nutshell:

  • Everything works. So asking does X improve achievement is a trivial claim.

  • We need to aim for a larger magnitude change than just “any change” and we know what is average (d=<0.40) so this is the minimal target.

  • The key is for school leaders, systems and teachers to know the impact they are having and to work from this. If it is less than d=0.40 you have to ask yourself is this is worthwhile carrying on with. If this is more than d=0.40 then carry on with what you are doing.

  • “Know Thy Impact” forces the moral question: impact on what? This is more than achievement. This extends to making schools an inviting place to learn, retention through to the final year of school, moving from surface only to also include deep processing, enjoyment and passion, etc.

2. What is microteaching?

Microteaching involves teachers reviewing videotapes of the lesson after having taught it, see http://en.wikipedia.org/wiki/Microteaching. Our view is that the video of the students’ reactions is much more powerful for a teacher to observe than a review of what the teacher did/said/taught. It should be all about the students’ learning and less about the teacher teaching. What did the students react to, etc.?

3. How can we change the teachers’ thinking about what effects are most significant in a child’s learning?

Asking teachers to monitor and evaluate the impacts of their own teaching on student learning is the key. Go to our website and listen to John Hattie’s summary on ‘Know Thy Impact’

4. Our curriculum people will see d=0.60 for professional development and say “Whew, we’re already doing that”, but we’re not getting the results we would like. What about professional development makes the difference?

Opportunities for teachers to engage in professional development is one way to change thinking. For an analysis of the most effective types of professional development see p. 119-121 of the Visible Learning book (2009).

5. Where do you start to ensure that you are not repeating what has already been taught?

Have dependable assessment information from a variety of sources from previous teachers that is trusted and used. Without this, we continue to get repetition of content, with many students knowing what is being taught to them. Schools need dependable ways of assisting teachers to learn from each other and trusting each other’s data.

6. How does the teacher’s preparation (i.e. planning and preparation of lessons) affect the students’ learning progress?

The key issue here is for teachers to cooperatively plan their lessons, with particular emphasis on:

  • their notions of what success from the lessons look like

  • the quality of the evidence about this impact

  • evidence of the knowledge and understanding that students start the lessons with and

  • the magnitude of the change expected (and whether it is sufficiently challenging)

We do not suggest merely more time for planning, especially if this is done alone, or worse sitting with others in the staff room working alone. We advocate a focus collective impact. We realise this is not easy but the work we have done on this pre-planning has certainly led to higher impact.

1. If students in a class are all setting different targets what sort of learning intentions do we need to set (whole class, individual)?

The effects are stronger at the individual level, but the ways of setting and monitoring targets can be taught as a class unit. To manage this effectively, students who have common needs can be grouped together and taught as a small group. Often targets are set around a common need which allows the teacher to set a more global learning intention and differentiate through the success criteria. For example, the whole class may need to focus on punctuation with one group of students learning to use capital letters and full stops correctly with another group learning to use semi colons commas for phrasing.

2. How much time can a student have out of classroom engaged in other activities (e.g., sport teams) before it has a detrimental impact on their learning that outweigh the positive impacts of extra curricula activities?

All of us like to protect our “time” in teaching and maximise student time in learning. Successful time is more related to engaged and productive time and not spent time. When we consider that 70-80% of our classroom time teachers are talking then the productive and engaged time it is very low indeed. Maybe we need to reduce “our” time and increase “their” time. Consider sport (etc.) many students see the purpose of schooling in these terms, derive much satisfaction and learning, and the art is how we integrate and capitalise on these activities in our class time. For some students little outweighs the positive impact of these extra curricula activities so we need them indeed. The key for the classroom teacher is productive time – how could you construct a program of study (with clear learning intentions and success criteria) for learning in interrupted time periods.

3. What effect does the use of computers/interactive whiteboards have on learning of students.

The potential of interactive whiteboards (IWBs) has often been argued in terms of developing more effective demonstrations, presenting a variety of representations and aspects of display more generally. The claim is that they are attractive to both teachers and children, and capture and hold pupils’ attention much more strongly than other classroom resources. One study concluded ‘While our findings support some of the claims being made for IWBs, they do not suggest a fundamental change in teachers’ underlying pedagogy’ (Sinclair & Coulthard, 1975, p. 454). In terms of impact on pupils’ attainment, the White Boards appeared to have a negligible effect (Higgins et al., 2005).

In short, the effects have been negligible, but the potential of IWBs remains untapped. Our guess is that they are used as adjuncts to a teacher’s current teaching, and it is only when teachers change their teaching to optimize the power of the multi-modal methods, that IWB will show a difference. Reviewing the literature on interactive whiteboards” Steve Higgins, Gary Beauchamp, and Dave Miller (Learning, Media and Technology, Vol. 32, No. 3, September 2007, p. 213–225)

1. Why do you say “Inquiry based teaching doesn’t work”.

The average effect of inquiry based teaching is d=0.31, but there are some clear issues that are raised in the book Visible Learning.

There are many forms of Inquiry teaching, but too often it is implemented without recourse to the content domain. Teaching generic skills of inquiry, which is the common method, leads to this low effect size. Visible Learning promotes the use of SOLO taxonomy. SOLO taxonomy has five levels: no idea, one idea, many ideas, relate the ideas, extend the ideas. The second and third are typically surface and the latter deep processing. Inquiry is more specific to relating and extending (and similar cognitive processes) but it needs ideas to work on. One of our findings is that such processing does not transfer across subject domains which is why Inquiry teaching within (not across) subject domains is likely to have a much greater effect than as a generic method.

A lot of the problems are advocates that promote a method of teaching (e.g., Inquiry teaching) which then takes a life of its own and ultimately the teaching of the content is downplayed. Subject matter vocabulary is critical, knowledge of ideas essential and this too often gets missed when some follow the advocacy of method. Of course, if firmly embedded within a content domain, inquiry is critical. We prefer "problem solving methods" as this directly invokes a notion of "problems - about what".

So we are not suggesting inquiry based does not work but previous research, based on research using this method, has not been that powerful in its impacts.

Secondly, all work is probability statements. Based on prior implementations it is probably that Inquiry Learning only has a d=0.31 effect. The better situation is asking about your own impact; if you gain d>0.40 using your methods then there is every reason not to change.

2. Do you think Bloom’s Taxonomy is an effective teaching and learning tool?

Quote from John Hattie: No. In my opinion it is outdated. There is no evidence for the Bloom hierarchy (neither the earlier nor more recent modifications) and it mixes strategies for thinking (analyse, create) with levels of understanding (knowledge). In Visible Learning we advocate the use of SOLO taxonomy.

Please refer to the following papers on this topic:

Hattie, J. & Purdie, N. (1998) The Solo model: Addressing fundamental measurement issues. In Dart, B. & Boulton-Lewis, G. M. (Eds.) Teaching and learning in higher education. Camberwell, Vic, Australian Council of Educational Research.

Hattie, J.A.C. & Brown, G. T. L. (2004). Cognitive processes in assessment items: SOLO taxonomy (Tech. Rep. No. 43). Auckland, NZ: University of Auckland, Project asTTle.

http://www.tki.org.nz/r/asttle/pdf/technical-reports/techreport43.pdf

http://www.educationcounts.govt.nz/publications/schooling/36097/4

3. Inquiry learning is embedded in the NZ Curriculum. How do we make it to the ‘Problem Solving’ teaching that gives d=0.61 vs. inquiry at d=0.31?

Too often inquiry is content free; too often inquiry is considered best taught via constructivist, facilitator methods; too often inquiry is not enquired about itself. Carl Bereiter has found that inquiry is often taught more successfully using direct teaching methods, by teachers who consider they are change agents not facilitators, and through well planned even scripted lessons. We must not confuse the outcome (inquiry learners) with methods of teaching that abrogate responsibility by the teachers to actually make the difference. A much more effective mantra would be to adopt “problem solving” approaches as this emphasizes the problem, the problem clarification, the notion of success that comes from resolving problems and asks for a better balance between surface, deep, and conceptual understandings.

4. Are inquiry learning approaches limited to poorly conducted early experiments?

No. Contrary to some reports about Visible Learning, we have been particularly attentive to the quality of studies. The quality of the studies did not make a difference – this is a finding not claim. Inquiry based learning is low as too often it ignores that you need to have surface knowledge to “Inquire” about anything.

There are many different ways to use effect sizes but here we focus on progress – not on comparisons between classes, teaching methods etc.

Imagine a class of students have been administered a similar or the same test relating to the curriculum in February and June. We can use the data from these two tests to calculate an effect size. This effect size helps us to understand the impact of our teaching over this period.

The easiest way to calculate an effect size is to use Excel. Here is the formula:

Effect size = Average (time 2) - Average (time 1)

standard deviation

Consider this example:

like0