Margaret Phillips, the digital archiving manager for the National Library of Australia (NLA), told ZDNet Australia the NLA will archive in the vicinity of 20-30 adult sites, which would equate to less than one percent of the archive. "We haven't started doing this yet so we don't have a good idea of the sort of sites that are out there or what we need to archive to get a representative sample."
According to Hitwise 40 percent of sites in its database fall into the "adult" category, and the category claimed about 20 percent of Australian traffic for the month of July.
However, Phillips said that the decision to archive adult material did not mean that the storage needed would increase by 40 percent. "We take a very selective approach," she said of the library's archiving process. "We make sure the site is complete and all the files are there."
The archive is still relatively small, but the NLA recognises that the amount of data is going to increase dramatically over time. "We are expecting that we will be assisted by developments in technology, and storage will become cheaper," said Phillips. "We'd be expecting that we could compress this data more in storage."
Currently the archive is 353GB, and it grew by 32GB in July and 18GB in June. It contains 13.5 million files, and archives 2,700 titles. Ninety of these titles are commercial, and the library is required to restrict access for a period of time so as not to adversely affect the business of the publishers.
Phillips said the purpose of archiving online sites was to fulfil the NLA's role of archiving the countries output.
"The National Library is interested in collecting publications on the Internet in two main categories," said Phillips. "Those publications which have research value in their own right, for example government publications and peer-reviewed journals. The second category is those sites that collectively build up a picture of Internet publishing in Australia today."
The library also periodically focuses on specific topics, such as gun control or the Sydney Olympics, and collects a wide range of organisational and community sites. "We try to cover the whole picture, to capture the plethora of Australian sites on the Internet at this time."
"There's no doubt that as time goes this is going to be a very expensive undertaking for the library," she added.
