Cooking the (visitors) books: the Australian War Memorial struggles with statistics – again

David Stephens

‘Cooking the (visitors) books: the Australian War Memorial struggles with statistics – again’, Honest History, 7 February 2017

‘There has always been an element of myth-making and myth-peddling in the Memorial’s work; it should not extend to its statistics.’

Twelve months ago, in an article which is quoted above, Honest History did some detailed analysis of how the Australian War Memorial counts its visitors – both real people up the steps and visitors to the Memorial’s website. We used statistics from the Memorial’s Annual Report for the year 2014-15, and from Alexa (an internet industry analysis site), the Australian Bureau of Statistics, and the ACT Government.

Now we can update the key points of that analysis, using the Memorial’s Annual Report for 2015-16. The evidence this time around shows a slight dip in real visitor rates – and a tendentious conclusion – plus continuing carelessness and slipperiness in the Memorial’s website visitor statistics. As we have done previously, we welcome comments from the Memorial and will reprint them without amendment.

Real visitors

Last time, we concluded that the Memorial’s Australian-based real visitor numbers – people going up the steps – have remained remarkably stable from year to year over a quarter of a century.[1] About four per cent of Australia’s population visits the Memorial in any one year – including lots of repeat visitors – but around 96 per cent of the population does not.

Having considered the Memorial’s 2015-16 report, we have no reason to change our conclusion from last time; 26 years of statistics gives pretty much the same result as 25 years. In the 12 months ending 30 June 2016, 1 010 910 Australian-based visitors came to the Memorial, down 4.9 per cent on 2014-15 (1 062 817), and 4.2 per cent of Australia’s population as at 30 June 2016.

The long-series graph from 1990-91 is below. Figure 1rev and the accompanying Excel spreadsheet shows 26 years of Australian-based visitors at an annual average of 811 558 visitors and an average of four per cent of the Australian population in any one year. The sharp increase in 2014-15, like the one in 1993-94 (the year when the Tomb of the Unknown Australian Soldier was dedicated), needs to be set against the annual average and the long-term trend.

How did the Memorial interpret the 2015-16 figures, though? ‘Record numbers of Australians and international visitors walked through the front gates, spurred by the centenary of the start of the First World War and the magnificent new First World War galleries’, says Kerry Stokes, Chair of the Memorial Council, on page 2 of the 2015-16 report. There is a similar claim in the body of the report: ‘The number of visitors walking through the front doors has increased on last year’ (page 14). (Note that these reports are supposed to be for financial years. It is possible that Mr Stokes’s rather garbled account is referring to a figure for calendar year 2015, though such a figure is not provided. The reference on page 14, however, is explicitly to ‘[t]he 2015-16 financial year’.)

The 2015-16 report (page viii) does indeed include as the top line under ‘Visitors’ a figure of 1.232 million, which is 90 000 higher than the 1.142 million on the equivalent line of the 2014-15 report (page xiv). But the 2015-16 visitors figure includes 145 264 who came to touring exhibitions outside Canberra, whereas the 2014-15 visitors figure does not include touring exhibitions. Even if we compare like with like the result is still the same: the 2014-15 report also has a figure of 1.282 million for attendance at ‘Memorial exhibitions and travelling exhibitions’ so even that figure is 50 000 higher than the equivalent figure for 2015-16.

Figure 2 illustrates the point. The Memorial’s own statistics contradict its claims about an increase: comparing like with like shows a decrease in visitors across the financial years, not an increase. The only way the number goes up, year on financial year, is if we compare ‘apples’ and ‘oranges’.[2]

Figure 2: comparative statistics, Australian War Memorial visitors, 2014-15 and 2015-16

Sources: 2014-15 report, p. xiv; 2015-16 report, p.viii. ‘Mitchell’ is a reference to visitors to the Memorial’s storage facility at Mitchell, ACT (3008 of them in 2015-16; 2014-15 figure not given).

Website visitors

In our article last year, we said this: ‘We … characterise the Memorial’s statistics about the virtual visitors to its website as a mixture of confusion and spin, even obfuscation’. We found:

• confusion, deliberate or inadvertent, between website visitors and visits: a visitor can make a number of visits; the 2014-15 report gave more prominence to the (higher) visits number;
• occasional use of figures for views to get an even bigger number: views figures are almost always larger than visits because most visits involve viewing a number of pages of the website;
• failure to use the key statistic of unique visitors: no matter how many visits a visitor makes, if he or she is using the same device and browser he or she only counts as one unique visitor, so that a unique visitors number will almost always be lower (and thus less impressive) than a visitors number;
• lack of information about bots (information-seeking automated web crawlers, rather than real people), bounce rate (looking at one page of the site then leaving), duration of visit, views per day, and internal usage (by officers of the Memorial): these figures add more depth to website statistics but tend to take the gloss off numbers for visitors and views.

Has the Memorial done any better in its 2015-16 report?

• The Memorial reports 5 495 313 visits to its website in 2015-16, which the Memorial admits (correctly) is 25.5 per cent down on last year (pages viii, 42, 44) but it also claims in an adjacent paragraph on page 42 ‘a 48 per cent increase in visits to the website this year’. It is not immediately clear where this number comes from – but see below.
• The Memorial separately records (page 42) a figure for 3.29 million website visitors. Strangely, this is exactly the same as the visitors figure for 2014-15, which is either a massive coincidence or an egregious error. Assuming it is an error, that leaves the report lacking a correct figure for website visitors in 2015-16, so we cannot make a year-on-year comparison for that figure.[3]
• Further comparison of the reports for 2014-15 shows that the mysterious ’48 per cent increase in visits’ words also appear in the 2014-15 report – when they were true – as well as in 2015-16, when they are false. Indeed, there are many other similarities between the two years in the words appearing under ‘Deliverable 1 An engaging website with accurate information’. Sometimes ‘alternative facts’ arise not out of bloody-mindedness but carelessness. At least one hopes that is the reason here.

Figure 3: Australian War Memorial website visitors: comparison of Annual Reports[4]

‘There was a 48 per cent increase in visitation to the website this year … This year has seen a 62 per cent increase in visitation to collection items, with 3.29 million visitors looking at this material compared to 2.09 million last year. There was a 31 per cent increase in visitation to the biographical information on the site, up to 12.9 million views from 9.87 million last year.’ (2014-15 report, page 35)

‘There was a 48 per cent increase in visits to the website this year … This year has seen a 62 per cent increase in views of collection items, with 3.29 million visitors looking at this material compared to 2.09 million last year. There was a 31 per cent increase in visits to the biographical information on the website, with 12.9 million views compared to 9.87 million last year.’ (2015-16 report, page 42)

• The Memorial reports 28 million page views of online research facilities, compared with 20 million last year but, perhaps recognising the limited value of this statistic, does not give it prominence. (The 2014-15 report had it under ‘Highlights’ on page xiv; the 2015-16 report buries it on page 54.)
• The Memorial again gives no figure for unique visitors. Figure 4 repeats what we said a year ago about unique visitors:

Figure 4: ‘Unique visitors’: extract from previous Honest History analysis

‘[T]he number of unique visitors to awm.gov.au might be a much lower figure than the number of visitors and much, much lower than the number of visits. There could be, for example, 329 000 unique visitors (actual humans) making ten visits each, to add to 3.29 million visitors, or there could be a million unique visitors each making 3.29 visits, to add to the same figure. If the 3.29 million is actually unique visitors why not say so rather than call them visitors, which is technically a different metric? Is there a lower, less impressive unique visitors figure that is not being revealed?’

• While the Memorial this time reports an ‘average session duration’ of four minutes and 20 seconds (page 42) it says nothing about the effects of bots on statistics, or about bounce rates, or about how much of any of its website statistics is attributable to internal users rather than members of the public. We refer readers to our previous analysis regarding bots and bounces.

Conclusions

When Honest History pressed the Memorial in 2016 for a response to our previous visits analysis, we received this reply: ‘The Australian War Memorial is happy to let your opinion stand as it is. Thanks.’ We were not sure what to make of this response – in the ‘post-truth’ era it is common to disparage an interpretation robustly supported by evidence as merely an ‘opinion’ – and it is disappointing that the Memorial disdains engagement with constructive critics.

This time around, the claim of an increase in real visitors – ‘visitors walking through the front doors’ – is not supportable by the Memorial’s own statistics. This suggests at least sloppiness, if not an intention to mislead. Similarly, there is still a degree of carelessness, even obfuscation, in the Memorial’s website statistics. Most organisations will gild the lily in presenting statistics but they will usually be cleverer at it than the Memorial is – and they will avoid clanging cock-ups like using last year’s statistics.

All that said, the gullible will be taken in if they do not do their own forensics, and that includes Ministers, members of Parliament and government bean counters. We note that the Memorial recently received ‘$4.0 million over four years from 2017-18 … to ensure that the AWM can continue to deliver its core activities under the Australian War Memorial Act 1980’. We would be interested to know if the statistics in the Memorial’s Annual Report formed part of its case for receiving this money.

In 2016, when we sent each member of the Memorial’s Council a copy of the first edition of Honest History’s Alternative Guide to the Australian War Memorial, the Chairman of the Council, Mr Stokes, said we were free to provide material to Council members. Accordingly, we have sent a copy of this article to each member of the Council. We have also drawn it to the attention of the Minister for Veterans’ Affairs, the Minister for Finance, the Labor Shadow Minister, the Greens spokesperson, and the Chair of the relevant Senate Estimates Committee.

Notes

[1] The Memorial advised us in 2016 that about seven per cent of its visitors each year were tourists from overseas. We then discounted total annual visitor figures by seven per cent to get a figure for Australian-based visitors. Figure 1rev shows both figures.

[2] We emailed the Memorial on 2 February to ask what statistics supported the statements in the Annual Report. No response had been received by deadline 6 February.

[3] If – and we have no way of knowing this – the proportion of visits to visitors in 2015-16 was the same as in 2014-15 (approximately 45 per cent) then the visitors figure in 2015-16 would have been about 2.5 million (45 per cent of the visits figure, 5 495 313).

[4] There has been minor fiddling in the 2015-16 report with words – ‘visits’, ‘visitation’, ‘views’ – but more important needed revisions have been overlooked. For example: the 2015-16 report repeats (with minor editing) some good news from the previous year (even though it is prior to the reporting period); a paragraph on page 42 about ‘continual improvement’ has undergone editorial titivation but still repeats the dated information quoted in Figure 3 about 3.29 million visitors.