Tech in Government: Digitisation is an essential strategy for the preservation of data
Address given by Mr Matt Anderson PSM, Director, Australian War Memorial for the Tech in Government Conference, 8 August 2023.
The Australian War Memorial is a shrine, a world class museum and an archive.
In the past year we have had nearly 1 million visitors on site, including 96,000 school children. 70,000 visitors have attended the daily Last Post Ceremony.
380,000 have visited our travelling exhibitions.
We have been collecting for over a century, with millions of items in our National Collection and archives.
In 1925 we had 50,000 photographs.
Today, we have 4 million.
In 1925 we had 100 hours of film
Today, we have 7,000 hours.
Currently, the Memorial’s digital holdings sit at around two petabytes of data.
That is roughly the equivalent of 40 million four-draw filing cabinets filled with text.
Over time, including two decades of digital preservation programs and projects, we have built up an extensive online collection as well.
And each year we accept an average of 30,000 items into the Collection.
The Memorial’s mission is to lead remembrance and understanding of the Australian experience of war. We operate to allow Australians to understand the causes, conduct and consequences of war, and through that understanding, to meaningfully commemorate the sacrifices that war and conflict inevitably require.
Our rich and varied collection is the most effective way to assist our visitors in that progression from curiosity through to meaningful commemoration.
A great institution requires a great collection. And making that collection both relatable and accessible.
The Memorial, while an institution of national significance and a local landmark, is also of course, intensely personal. From our inception, people have come to the Memorial to mourn at the Memorial, for a lost loved one whose grave was overseas.
They have come to the museum to transport themselves to the places of their loved ones, to experience through the exhibits what their loved one may have experienced.
And they come to research: To read contemporaneous accounts of conflicts, in the words of someone who was there.
An important part of the Memorial’s current development, is the construction of a purpose built research centre, immediately adjacent to the Bean Building, where the bulk of our photos, film and sound are physically archived. It will be for families, students and professional researchers/historians.
The Memorial has a wealth of online digital content. Millions of records help people discover film that they can download, photos and art to view, interviews to listen to and books and archival records to read online.
4.8 million unique visitors to our website,
15.2 million page views,
75,000 hours of youtube watch time.
9,800 research enquiries.
The Memorial has progressively digitised millions of pages of official records as well as tens of thousands of pages of personal letters and diaries.
Some of the originals are on display.
When you stand in our Gallipoli gallery, you see the Lone Pine diorama that puts you firmly in that time and place. And across from it, you can read a letter from Private Thomas Anderson Whyte to his fiancé, Eileen. The letter is written in pencil, the handwriting is graceful and even. It describes his love for her, and tells her not to grieve too long should he not return. It is dated 24 April 1915. Private Whyte was wounded in the Gallipoli landing on 25 April and died that evening. This letter, the original paper, that tells a very human and very personal story of Gallipoli, will need to be rested in order to preserve it. But it, along with 127 other pages of his letters have been digitised and will be freely available even when the original is taken off display.
The Memorial’s digital collections are freely available online and are heavily used by the public, offering valuable resources particularly for all humanities disciplines.
Stabilisation of the original paper object ahead of digitisation
But before we even start the digitisation processes on many, many occasions we have to stabilise the original document – removal of sticky tape and corrosive metals, repair of paper, flattening of the object and rehousing are part of the digitisation process.
For example, the War Diaries of the 2nd AIF and CMF forces during and immediately after the Second World War contain approximately 35,000 files – several million pages!
One of the great achievements of the current digitisation project was to rehouse over 31,000 files into archival wallets and folders ahead of digitisation. Where required, acid free paper was also interleaved in files to reduce damage caused by the high acid content in some papers. In this process over 2,500 old boxes which were high in acid content were replaced with stable acid free boxes.
This was a huge achievement in ensuring the long term viability of the original physical files. They will be stored for posterity and will no longer be handled by the public who, once digitisation is complete, will have access to them in digital form.
The Series contains a wide range of operational material, maps and reports, including copies of messages, minutes of staff conferences, general staff memoranda, appreciations, circulars, procedures, administrative instructions, intelligence summaries, signal instructions and unit histories.
This collection will form the backbone of future thesis, books and documentaries towards major Second World War anniversaries.
Digital preservation has different purposes, depending on the collection type.
- Photographs, film and Sound material deteriorates by its very nature. This requires a change in format with a goal of ultimately replacing the original. No matter how well stored magnetic tape is, for example, it will deteriorate. There is a growing recognition of the need to digitise magnetic tape due to the natural life of the material.
- The principles underpinning the digitisation of paper records is to preserve the originals from handling.
Paper records are on open public access. This means the public can come to the Memorial and view original records.
We have collections from the Boer War, First World War through to recent conflicts – documents from 100 years old, to current day.
As mandated in the Archives Act 1983, Official Records must be issued to the public unless a copy can be made available. This is essential for research, using primary sources, but puts the documents themselves at risk.
The digital preservation principle for our paper records is to make a high resolution copy and publish the image on the website for public access to limit public handling. The original is then housed – forever - in a stable storage environment and the public - and my son! - get a useful image to their desk top or mobile device.
The Memorial’s digital preservation strategy ensures high-quality copies are made to manage the security and preservation of vulnerable collections and preservation of digital assets in a Digital Asset Management System.
- Historical collections include items that are inherently vulnerable to deterioration over time. Deterioration is delayed by good conservation and storage but preservation is achieved by high-quality digital copies of paper records and photographic prints, glass plate negatives, a variety of photographic film types including acetate and nitrate.
- Digitisation delivers a record of physical objects and artworks at the time of capture which provides a benchmark for future conservation. Physical objects in appropriate storage are less vulnerable but high-quality imaging provides a basis for planning conservation and future remediation.
- Digitisation enables the establishment of systematic, consistent metadata standards for objects in the collection, and more effective collection management, online searching and discovery.
- Access to digital copies rather than the original mitigates damage to original material.
I’d like to expand on this last point:
Access to digital copies rather than the original mitigates damage to original material and loss of historic data contained within the records
The Memorial’s archival paper records span to over 100 years old and are in constant and heavy use. Certain eras of paper are of poor quality.
Second World War paper, for example, was subject to rationing and shortages in supply and as a result is more vulnerable to deterioration than paper produced in earlier eras. Second World War era paper can be very fragile.
Paper records created by prisoners of war are especially vulnerable to deterioration over time.
The Memorial, for example, is currently digitising the original records of Sister Betty Jeffrey who was a nurse stationed in Malaya with 2/10th Australian General Hospital during the Second World War. She was evacuated to Singapore, and then again on the Steamship, Vyner Brooke and was among the survivors of its sinking. She was captured and became a prisoner of war surviving in various camps including Palembang on Sumatra.
This more than a piece of paper.
Jeffrey recorded her experiences as a prisoner of war in notebooks and on scraps of paper detailing the conditions. She recorded how she survived in the camp and drew many sketches depicting camp life. Not surprisingly, many of these entries related to food, but they also detailed the death of friends and her own physical and mental battle for survival.
She also kept a diary written in two children’s exercise books that she managed to steal from the Japanese. Her diary was written with a pen that she shared with her friend Tweedie. She often had to hide the diary by sewing it into her pillowcase and it sometimes stayed there for periods of six weeks and, in one instance, many months. Jeffrey later used her diary and the scenes depicted in her sketches to write her best-selling book White Coolies.
This is SO much more than a piece of paper!
Amongst Jeffrey’s music is a sketch depicting waving hands and barbed wire. This sketch was drawn from House Two, the only place where women could see their relatives in the men’s camp. The women would wave to the men in the distance each morning and evening.
Jeffrey described how on Christmas day they sang to the men "oh come all ye faithful". She describes how the men stopped and listened and waved hankies, shirts and hats, and an echo of thank you was also heard. Two days later the men sang the same song back to the women and "Everyone wept."
For the prisoners of the Japanese during the Second World War, paper was a rare and valuable treasure. Identity records, such as a marriage certificate, a birth certificate or a list of names, were regularly confiscated and destroyed on discovery.
Surviving searches and destruction, these paper records exist today as part of the most valued treasures in the Research Centre’s collections. These personal, contemporaneous stories can turn a lesson about history into an empathetic exercise.
You can see the hankies waving and imagine “an echo of a thank you”.
Betty’s Jeffrey’s prisoner of war records, created on scrounged paper within the prisoner of war camps, hidden and retrieved, sticky taped and exposed to all manner of conditions, were fragile then, and more so now.
Having survived the POW camps against all odds, we must protect them so they can continue to speak to future generations.
How can we balance access and conservation?
We have started by making a copy of this record and providing access to the copy, which will ensure survival for centuries going forward.
Digitisation enables the establishment of systematic, consistent metadata standards for objects in the collection, and more effective collection management, online searching and discovery.
There’s little point having a document if it’s not discoverable.
Enhanced cataloguing is an essential process for digitisation and delivers enriched public access online. Digitisation includes capturing metadata which is the critical information for linking collections and making them discoverable online, and provide a basis for systematic conservation and preservation of the National Collection.
Digitisation enables deep indexing of data and linking of collections. The Memorial is currently serving an enormous public appetite for access to its data.
Going back to the numbers – 9700 research centre requests. 4.8 million visits to our website.
2 petabytes of data.
The data, created from two decades of digitisation and indexing of the collections, are a rich vein for research and understanding the Australian experience of war.
The Memorial’s digitisation programs are aimed at making more effective use of its data to enable the public to discover and search the collections more independently and successfully. This will provide efficiencies in resourcing public enquiries and improve regional access to the collections.
Digitisation is also focused on unlocking and sharing data in the web environment for broader public research and reuse.
The Memorial’s digitised collections are linked to 912,995 names which are published on the Memorial’s website and used for linking data and collections. The metadata created from digitisation enables the possibility of linking the Memorial’s collection with other cultural institutions - in particular the National Archives and the National Library - and creating a more powerful and extensive research capability for all Australians in understanding the experience of war and military operations.
It enables processing by artificial intelligence for interrogating large data sets which will help in visualising the collections and making them more useful.
Conclusion - Infrastructure and the cost of a digital archive
As I conclude, Digitisation for collection preservation is practiced by most major cultural institutions and the benefits are well demonstrated over the last two decades.
The downside is that the process is a slow and a resource intensive process for the majority of museums, archives and libraries.
Digitisation generally requires investment in bespoke and costly equipment. To be sustainable, investment in a digital asset management system is required. Safeguarding the digital images is an important aspect in safeguarding the original collection.
It can be difficult to gain corporate sponsorships for digitisation mainly due to the longevity of the process and the cost.
There is so much more to digitisation than scanning the document: - cataloguing, conservation, standards, image capture, archival arrangement and description, metadata and image preservation are all essential parts of the process.
The true whole of life cost to digitise one file of a private record collection held at the Memorial is estimated to be around $1,000!
Collection preservation, therefore, becomes a matter of prioritising – for the Memorial this is based on public demand and targeting material that is in high use or high value.
The Memorial has invested in and developed a digital preservation infrastructure through its Digital Asset Management System (DAMS) and can sustainably publish digitised collections and metadata through the Collection Management System (CMS)
The Memorial has made significant investments in the infrastructure needed to deliver a modern digital archive. As a result practices are more efficient, sustainable and effective. The Memorial has invested in a scalable platform to publish digitised collections which is delivered through the CMS and the DAMS. This enables the Memorial to streamline the delivery of digitised collections from capture through to publication on the website.
The current processes work efficiently and are scalable, should sponsorship or increased funding for digitisation become available, we will be able to expand the scope of material we can conserve in this way.
The Australian War Memorial Act, 1980 compels me to make accessible the National Collection available.
I hope that each and every Australian makes at least one trip to our stunning heritage building at some point in their life. However, I also know that digitising our collection allows me to take the collection to the far reaches of Australia, without endangering it.
It preserves the material for future generations, and through the magic of metadata, will allow for connections to be made between collection items.
Regional schools and remote communities; all of the places where proud cenotaphs mark their sacrifices of the great wars, can dive into the records behind those names and find the whole story.
Thank you to Robyn Van Dyke, Head of the Research Centre at the Australian War Memorial for leading the digitisation project, and providing background to this presentation.
The founder of the Memorial, Charles Bean said “Here is their spirit, in the heart of the land they loved and here we guard the record that they themselves made”. Digitisation guards that record in a new way, for a new generation.
It also puts the National Collection where it belongs, and where it came from; in the hearts and hands of all Australians.