Tangible Augmented Reality for Archival Research: Using Augmented Reality to Research Cultural Heritage Items
Daniella Kalinda, Ryerson University, Canada, Lucas Hrynyshyn, Ryerson University, Canada, Gabby Resch, Ryerson University; University of Toronto, Canada, Anitha Nathan, Ryerson, Canada, Ravit David, Scholars Portal, Univ. of Toronto, Ontario, Canada, Ali Mazalek, Ryerson University, Canada
Abstract
In this paper, we investigate how innovative technologies in the field of cultural heritage, particularly augmented reality, can be used to advance research in a highly digitized era. We will describe advancements in digital recreations of cultural heritage. We will also describe a formative study we undertook to understand current challenges in cultural heritage research, as well as to understand current practices in research and insights gained from interaction with physical cultural heritage materials. We then propose two designs based on analysis of data from the formative study. Finally, we conclude with possible directions for the integration of augmented reality in cultural heritage research.Keywords: augmented reality, cultural heritage, tangible augmented reality
Introduction
Cultural heritage items can be interacted with in novel ways using emerging digital technologies like augmented reality (AR), virtual reality (VR), and mixed reality (MR). The rise in popularity of augmented reality, in particular, shows promising opportunities for integration of this media for people from all walks of life- including cultural heritage researchers. With respect to cultural heritage research practices, current methods for presenting digital archival content often lack the richness, versatility, and longevity that is necessary for researching physical artifacts or ephemera (Champion & Rahaman, 2019). Our project aims to address some of the challenges that scholars face when working with ephemera and memorabilia collections, and to enhance scholarly engagement with cultural artifacts by enabling research and discovery through emerging technologies, and augmented reality in particular.
In this paper, we describe advancements in digital recreations of cultural heritage and present a formative study we undertook to understand current challenges in cultural heritage research, as well as to understand current practices in research and insights gained from interaction with physical cultural heritage materials. We then present the design of a mobile augmented reality prototype based on analysis of data from the formative study. We conclude with possible directions for the integration of augmented reality in cultural heritage research.
Background and Related Work
This section provides an overview of the roles, value, and challenges of cultural heritage preservation, and an assessment of how augmented reality can be used in cultural heritage research.
Cultural Heritage
Cultural heritage refers to the traditions (tangible and intangible) that are transferred from one generation to another. Tangible heritage includes objects such as artifacts, ephemera, and buildings, whereas intangible heritage involves norms, practices, and stories (Brumann, 2015; Kenderdine, 2015).
Tangible objects such as artifacts (three-dimensional cultural heritage objects) and ephemera (paper-based cultural heritage) are kept in archives, libraries, and museums where they are preserved and can be used for research. However, with limitations in storage and access to these archives, the cultural heritage field has recently shifted toward digital methods and emerging technologies to improve access.
Challenges Facing Cultural Heritage Access and Interactions
First-hand public access to cultural heritage is limited by several factors. For travel and tourism, cultural heritage centres are attempting to increase their number of visitors, improve the visitor experience, and reimagine public perception of cultural heritage sites as exciting, innovative, and entertaining (Chung, Han and Joun, 2015). One of the ways the industry has achieved this is by incorporating emerging technology as part of the visitor experience, with a specific focus on augmented reality. Methods for preserving cultural heritage have also evolved with the advancement of technology. Many libraries are taking initiatives to digitize their collections and increase access to their historical items so as to remain relevant in a digital world and to keep the library as the center for knowledge acquisition (Jones, 2017).
Some heritage researchers assert that ephemera are most meaningfully understood in a collection, posing another challenge to visualization and representation of ephemera through technology (See Appendix D; Table 1.2). For example, a single postcard with “John” as the sender’s name does not illustrate any sort of need for privacy or identity protection, until the viewer of the collection understands that numerous postcards sent from concentration camps all use this same name as an alias for their personal protection. Therefore, the value of ephemera as understood as a part of a whole collection makes them both unique and difficult to work with.
The cataloguing of donated special collections can often be non-linear. Items in a collection may be randomly assorted in a scrapbook, carefully selected by an experienced collector, or inherited by a family member (Anghelescu, 2001). While a collection can provide valuable insights on a collector and the motivations behind the collections, librarians and archivists often struggle to classify these special collections in a meaningful way that supports the research process.
A Collaborative Preservation Initiative
Various libraries have taken the initiative to digitize their archives (Smith, 2016). The Thomas Fisher Rare Book Library at the University of Toronto, which holds part of the Mernick Farley collection of Jewish Ephemera, is currently seeking to digitize its collection in an innovative way that will help heritage researchers undertake insightful research of primary source cultural heritage from remote places. Our project is being conducted in partnership with this initiative. Our goal is to create intuitive and engaging interfaces for meaningful digital research experiences. In order to do so, we investigated ways that innovative technologies have been used for the dissemination and preservation of cultural heritage, including which modern technological methods were most appropriate for this task.
Augmented Reality
Augmented reality is an enhanced digital visualization technique that superimposes virtual objects onto a physical environment, through handheld or wearable display, giving the appearance of being integrated in the physical environment (Milgram et al.,1994; Wu, Lee, Chang and Liang, 2012). This dynamic interaction with the physical environment may be the reason augmented reality has become increasingly popular in many areas including entertainment and education (Kim, Kang, Choi, Choi and Hong, 2017).
In a systematic review of augmented reality for educational purposes, Akcayir and Akcayir (2017) reported sixty percent of augmented reality applications are built for mobile devices, twenty-four percent for desktops, and sixteen percent are created for head mounted devices specifically designed for augmented reality. The authors argue that there are five factors that may make mobile devices more prominent: they are more cost-effective, easier to learn to use, portable, more likely to encourage social collaboration, and built for independent operability (one’s own ability to utilize the physical hardware). Furthermore, it is possible that the dynamic interaction between the physical and virtual environment may improve processing and problem-solving skills, critical thinking and on-the-spot decision making (Akcayir and Akcayir, 2017). The growing prevalence of augmented reality also demonstrates that the technology is maturing and will become easier to use. This trend suggests that augmented reality could be applied beyond the field of entertainment and used in areas such as education, training, and research.
Some of the commonly reported challenges with augmented reality are “lagginess” (a term indicating a delay between player action and the action’s digital result) and issues with rendering visualizations using trigger (or target) recognition (Wu, Lee, Chang and Liang, 2012; Akcayir and Akcayir, 2017). Improvements in wireless connections may decrease these challenges. However, most studies of augmented reality have been short-term and some researchers suggest that more assessment protocols should be taken to analyze long-term effects of AR usage (Botden and Jakimowicz, 2009).
Design and Implementation of Augmented Reality
The integration of PTC’s Vuforia plug-in with Unity 3D has helped novice developers build augmented reality applications fairly easily. They can quickly prototype different applications and test them using this game engine. For example, researchers with little to no programming experience built a mobile augmented reality application for children with obesity and tested its effectiveness in decreasing obesity and iterating as necessary (Kim, Suk, Kang, Jung, Laine and Westin, 2014). Game engines such as Unity 3D have increased access to developing mobile augmented reality applications for less experienced developers. Furthermore, this game engine can also be used to interact with tangible objects in a virtual environment.
Annexing Reality, a toolkit developed by Hettiarachchi and Wigdor (2016) on Unity3D, enables designers and developers to match augmented virtual objects to similarly-shaped physical objects in one’s environment. An AR user can grab a physical object and see the corresponding virtual object move in the simulated environment. The developers and designers who assessed this toolkit found it easy to learn and were satisfied with the tool overall. However, they reported that it was difficult to predict the weight of physical objects. Furthermore, for the AR user, erroneous head movements led to lagging, making interactions more difficult. Overall, despite some challenges, AR users found interacting with physical objects in the simulated environment to be more realistic.
Augmented Reality in Cultural Heritage and Tourism
Augmented reality applications have been used in the tourism industry to help improve the visitor experience. These applications are generally location-based and often intended for guided tours (Jung and tom Dieck, 2017; Dow et al., 2005). Augmented reality has successfully been integrated in creating innovative experiences in institutions like museums and heritage sites. Examples include integrating augmented reality with outdoor heritage in parks for sustainable tours (Ciolfi, 2013) and cultural visits (Kenderdine, 2015), incorporating natural interaction in augmented reality in museums (Kyriakou and Herman, 2018), and interactive and context-based applications in Egyptian museums (Nofal, 2013). Such designs may be useful for increasing public engagement, making the site experience more engaging for the modern visitor. They also point to interesting research applications, including the use of augmented reality for the interpretation of archaeological sites (Amin et al., 2012). These various examples suggest that augmented reality may be a relevant medium for academic cultural heritage research.
Formative Study
Methodology
The goal of this study was to discover the current working procedures researchers use, and to identify challenges in the interactions for researching cultural heritage. We hypothesized that researchers are looking for more intuitive and interactive ways of researching artifacts. We expected the interviews to provide insights into the areas where heritage researchers are lacking intuitive and interactive opportunities for engaging with digital artifacts. All the interviews were conducted in person and were audio-recorded. This ongoing research is in partnership with the Thomas Fisher Library at the University of Toronto. These findings are used to inform the design of an augmented reality prototype for research purposes.
Data Collection
The method for data collection used was open-ended interviewing (Yin, 2015, p. 138). Excluding demographic questions, the questions were open-ended and followed a qualitative interview format. A questionnaire was used as a guide for the interview but the order and phrasing of the questions differed depending on the context of the interview (Yin, 2015, p.142). Conversational-style interviews are suggested to help researchers recollect their work candidly (Yin, 2015, p. 142). Finally, data was analyzed using a five-step process: compiling, disassembling, reassembling, interpreting, and concluding (Yin, 2015, p. 185). These approaches are reviewed in the data analysis.
Participant Recruitment
We recruited participants using snowball sampling. All participants were sent a consent form to review once they had agreed to join the study through email response. They were offered compensation of $10 for their participation. Participation in this study included both senior and junior researchers with a graduate degree, as well as those currently working on their graduate degree who work with cultural heritage items. Researchers without a graduate degree and who do not work with cultural heritage items were excluded from participation.
Interview
A total of 8 participants were involved in this study. This includes 4 curators, 2 history professors, and 2 digital archaeologists. The study was conducted in either the participant’s office or in a booked meeting room to ensure privacy and an audio-recording of high fidelity. The study lasted between 30 to 50 minutes. The signed copy was kept by the investigator and a copy of the consent form was given to the participant. The investigator used the questionnaire as a guide for the interview. Out of the 37 available questions (see Appendix A), a total of 14 interview questions and five demographic questions were used between interviews. Any follow-up question was assigned to its original interview question. For example, questions 22 to 26, under the insights category of the questionnaire (see Appendix B), were all placed under question 22 for data analysis. Asking each question during the interview separately became redundant whereas grouping such questions into one question facilitated candid answers on the part of the interviewee (heritage researcher). These “grouped” questions (having a total of 14) were analyzed to find trends (see Appendix B).
Data Analysis
To analyze our findings, audio-recorded interviews were transcribed and, once verified, the recordings were deleted. Each interviewee was assigned an alphanumeric code. The questions were numbered to correspond to the questionnaire. The interview answers for each participant were reorganized into a new spreadsheet. Answers to a question that appeared twice or more were highlighted and summarized in an “emergent themes” section for that answer. The themes found across various answers were summarized to find common trends (see Appendix D). These steps were repeated as necessary.
All participants’ answers for each question were used to find trends in the workflows, desired insights, and common challenges during cultural heritage research. In the next section, the results will be discussed, then they will be interpreted in the following discussion section.
Results
After conducting our data analysis, we categorized our findings into three main sections: common practices in research, practices in tangible versus digital search, and researchers’ proposed solutions for enhancing research.
Research Workflow: Advantages and Disadvantages
The reported tools researchers use during research include pen, paper, laptops and desktops for note-taking. The software used include Excel, Word, Dropbox (Appendix D; Table 1.3; T7 and T8), and email applications. Only digital archaeologists in our study reported using 3D scanners (Table 1.4).
The motivations for research varied between researchers. While three curators and both digital archaeologists claimed to work collaboratively (Table 1.2 ; A4-A8) on projects as opportunities came, the history professors and one curator reported that their work was self-driven and they selected what they researched based on their area of specialty (Table 1.2; A1-A3). Therefore, while the ways that researchers collaborate vary, collaboration generally happens for the identification of miscatalogued items, for advice and clarification as part of a larger project, or to get another expert’s take on a topic they are less familiar with (Table 1.2; T20, T32 and T33). These conversations may take place by email as they analyze the cultural heritage item, or in person, about the general topic overall.
Pre-visit: Researchers reported starting their research online to find cultural heritage items (whether ephemera or artifacts) that are relevant to their project (Table 1.2; T4). During the pre-visit phase, researchers do as much preparatory work before the archival visit. This includes accessing various databases online, usually on a museum, library, or another archival institution website. It is reported to be more efficient due to time and budget constraints (Table 1.2; T15), but less reliable because the available information online is not “necessarily accurate” (Table 1.4; T8). The researcher will also generally contact colleagues within their network for clarification on whether certain collections are in their institution although not visible online (Table 1.2; T33 &T20). During this time, they begin organizing their findings in Excel spreadsheets, Dropbox folders or Word documents (Table 1.3; T7 and T10). They may also collaborate with colleagues either in person, over the phone or via email to decipher the provenance of an ephemeral item. However, researchers mentioned that not all cultural heritage items that are in an archive are available online (Table 1.4; T12), which can lead to a necessary archival visit for the researcher to find the archives onsite (Table 1.2; T2, T3A, T3B, T11 and T25). During the pre-visit stage, they decide which archive is worthwhile visiting and make preparations accordingly. Therefore, the pre-archival visit stage is used by the researcher to gather general data about their project themes and to find relevant archives for further research.
Archival visit: A researcher sifts through various boxes of data to find archives relevant to their research. Here, researchers take an “exploratory” approach to sort through the archives and analyze each item and create their own filing system for reference. Researchers generally take photos of items in the collection with their own mobile devices and create their own database for items that are relevant to their topic of research (Table 1.2; T18). They note details such as the box number, date, and object number of the collection. This way, they may easily refer back to the physical collection and know how to find each item easily.
Researchers use either an object and device-centric focus or a narrative focus to studying cultural heritage. Some researchers were more interested in the history of the object and its implications in society while others were more focused on the individual lives impacted by these objects. For example, one researcher studied the introduction of consumer cameras on social life, whereas another researcher studied the letters of a photojournalist at different stages of her career. While both used cultural heritage to make inferences, they took different focal points to arrive at their conclusions. All researchers, therefore, agree that the story behind the object is what is most important, but they do so according to different focal points (Table 1.3; T12-T14). Therefore, the purpose of studying any item of cultural heritage is not only to study the physical object but to draw implications about its significance.
Post-visit: Once researchers have visited the archives, they use their new-found insights to either redirect or reinforce the stories they are telling about these objects. They link the items to greater social structures and address the implications of the items carried within their historical context.
Heritage researchers who are professors use the photos they took at an archive for use in their publications (Table 1.1, T10; Table 1.2, T18; Table 1.3, T10). Curators generally opt to use the physical objects themselves in their exhibitions (Table 1.2; T19). In other cases, the research is ongoing and can lead to other archival visits for long-term projects, such as books.
Research Search Methodology
One of the main reported advantages of visiting archives and working with cultural heritage objects is that it offers researchers a more meaningful research experience (Table 1.2; T2, T3A, T3B, T11 and T25). The primary motivating factor for visiting archival institutions is to find ephemeral cultural heritage that is of relevance to a participant’s/researcher’s topic of study. Generally, a collection is sought out because it is unique, relevant to one’s research, and/or inaccessible online. This visit may be a part of collaborative work or a self-driven project. The visit to the archival institution was considered pivotal to most researchers and as the moment in their research that either solidifies their hypotheses or challenges them. Depending on their research specialization, researchers travelled either within the country or overseas to see these artifacts (Table 1.2; T3A and T3B). Although central to their research, most reported that one visit was enough for their research. Whether one’s work is more self-driven or collaborative seemed to be contingent on the researcher’s style. Some curators preferred the collaborative approach and worked mainly on requested items. However, other curators were more self-driven and enjoyed an independent research process. Professors tend to be more self-driven having a few collaborations on occasion.
Neither historians, nor digital archaeologists or curators, visited archives only out of curiosity or interest. However, they did mention visiting public exhibitions out of curiosity or interest. That is, while each researcher is personally motivated and curious about their work and tends to work on self-driven projects, the visits to an archival institution tend to be driven by a strong external motivation such as an upcoming publication or curated exhibition.
The archival visit is considered very important and researchers strategize (through researching online and contacting their network) to ensure their visit is worthwhile (Table 1.2; T4). Physical archives give the researcher the opportunity to make conclusions about the special collections that they have limited ability to make prior.
Participants stated that the majority of their research was conducted online because of ease, access, and vastness of directed search (Table 1.2; T27 and T29). Furthermore, it is considered to be more resourceful as it requires less time and a smaller budget (Table 1.2; T5). As a participant stated, “the find function is very valuable.” That is, being able to type pointed keywords in the search function helps researchers navigate copious amounts of information in a directed and efficient way. Online research, therefore, is considered a necessary precursor to archival research (Table 1.2; T4). This helps prepare the researcher with as much background information about the artifact in order to prepare for the tangible search. Further, while the archival search may be copious, the tangible analysis is considered more meaningful and more rewarding for their research.
Another advantage of visiting archives is that researchers have access to all the cultural heritage that is not available online. Further, the way that the collections are organized, researchers can make uncanny connections between items that usually are not possible online. Therefore, currently miscatalogued cultural heritage items in special collections can be beneficial for making unexpected discoveries. From the benefits of online and tangible research, as well as their disadvantages, researchers reported the solutions they thought would work best for their research.
Reported Possible Solutions for Current Challenges
The most common proposed solution was to have more cultural heritage items accessible and available online (Table 1.4; T12, T21-23). The second proposed area for improvement was to have a more accurate cataloguing system. The third suggested solution was to have one common platform for researching cultural heritage across multiple institutions. Many found that pictures online of the object in all angles sufficed for what they were working with. As long as they had the option to zoom in and out and see the recto and verso of documents and other two-dimensional ephemeral items, they found it sufficient for their pre- and post-research. For three-dimensional cultural heritage items such as artifacts, they found that the physical visit is necessary if the item is indeed relevant to their research. Some agreed that three-dimensional imaging of the artifact could be helpful but maintained that it would probably not replace the physical visit. One mentioned, “I’m very comfortable dealing with materials.”
Researchers were aware of how augmented reality has been used for cultural heritage. Most only saw a use for it for exhibitive or teaching purposes. None had used augmented reality for their own work but understood how it worked. Even less experience was reported about the use of virtual reality. One curator mentioned, “my imagination is probably better than VR.” When commenting on the use of augmented reality or virtual reality in their research they reported that they only saw a use for it in exhibitions or for teaching purposes, but not for research. One curator had incorporated AR in one of their exhibitions, portraying the exhibition in three different time stages. She stated that the project was “successful and beautiful” but, with a large group using one provided mobile device, the quality of the experience diminished as people waited longer than expected to use the application. However, another curator found that using AR for her exhibitions on African art pieces would be less appropriate for her audience. She found that the AR “looked too much like a video game” and would take away from the experience she was trying to create. They were unsure how AR or VR could be relevant to researching cultural heritage. However, researchers remained open to the possibility of such technologies for research because, “it could be a generational thing.” Another mentioned, “I started my career before there was an online.” While most participants who are experts in their field do not envision using a different technology to do their research, they have adapted to researching online even though they started their careers only researching through physical archives. The possibilities afforded by this reality are explored further in the prototype design.
Mobile Augmented Reality Prototype
From our study, we found that researchers were looking to have greater access and better information about cultural heritage items. While doing large digitization projects and item-by-item re-cataloguing of the items were beyond the scope of this project, we decided to start our development by tackling a different angle of the project. We sought to make the pre-archival visit digital search more meaningful by adding tangibility to the digital search experience, as well as adding features for collaboration and recommendations for an easier cross-platform search. In this section, the design of an augmented reality application for researchers is discussed. We call the application TARA, for “Tangible Augmented Reality Archives.” The design explores how virtual objects that represent artifacts from a cultural heritage archive can be superimposed over a tangible cube using a mobile augmented reality application. For simplicity, the cultural heritage in the applications only include artifacts and not ephemera.
Design
TARA (see Appendix C) consists of an augmented reality application on a mobile device that is coupled with a tangible paper cube with one QR code on each of the sides. The purpose of this interactive cube is to offer researchers a tangible experience during their digital search. In the application, the camera on a mobile device detects one side of the QR code and a three-dimensional visualization of an artifact is then activated. The digital artifact covers the cube so that turning the cube around in the physical environment also turns the artifact in the virtual environment (Figure 1). In this way, users can manipulate the virtual artifact by performing the corresponding physical manipulation on the cube, e.g., turning the cube upside down to see the bottom side of the virtual artifact, or bringing the cube closer to zoom in on the virtual artifact. Users also have the option to scroll between artifacts, get more information about an artifact, share or save the artifact, get recommendations of similar artifacts, and search through a database (Figure 2 and 3).



Low accessibility and inaccurate cataloguing were often mentioned as needs for improvement in the digital search. Therefore, this application is intended to be used by researchers during the pre-archival visit stage. We expect this to give them a better visualization of the cultural heritage they are researching, so that this may help them decide whether or not to make the archival visits. However, some questions we still found ourselves with were: how do we help make more cultural heritage accessible digitally? How can we help both researchers and librarians with their cataloguing of data?
Our next steps with this prototype involve user testing with cultural researchers and investigating if, and how, this application would be useful for research.
Future Directions
We aim to finalize the implementation of the Tangible Augmented Reality Archives (TARA) application and conduct user studies. From these studies, we will be able to learn how researchers might integrate TARA in their work. Future iterations of the TARA application could include the archival visit in its user experience—namely, helping with accessibility and cataloguing. During their archival visit, researchers would be able to use their mobile device with the augmented reality application and could take a three-dimensional scan of the artifact or ephemera they are interested in evaluating. With this, they would have the option of saving it to their own gallery and/or adding it to the repository. They could then create their own catalogue for each artifact, take notes, and with smart recognition of these notes, the application itself could detect keywords found in the researcher’s analysis and automatically create categories (or tags) to facilitate finding this artifact for future research on multiple platforms. Users could write their own artifact description and request changes in the current cataloguing of the artifact to be sent to the archivist of the library. This application could be useful for librarians as well in the digitization of their files.
Conclusion
In this paper, we investigated the value of using emerging digital technologies like augmented reality in cultural heritage research. We presented a formative study investigating current practices and challenges in archival research of artifacts and ephemera. Results of this study influenced the design of an augmented reality application called TARA (Tangible Augmented Reality Archives) that, we propose, might assist with cultural heritage research. In our formative study, researchers in the heritage field were interviewed about their workflow, insights, and challenges during their research. We found that there are 3 major stages in research, the pre-archival visit, the archival visit, and the post-archival visit. Each stage has a focus, namely, directed research in the pre-archival visit, exploratory search when visiting archives, and the post-archival search brings together the insights from the previous stages for an exhibition or publication. While the archival visit tends to only happen once, it was considered especially valuable for studying the cultural heritage item. Researchers noted that working with the tangible object allowed them to draw insights about the item that is not possible through a two-dimensional, primarily visual, digital search. Researchers suggested improvements centered around availability and accessibility of archives online, and they expressed skepticism about the relevance of AR for research. However, they also acknowledged the way in which online search has transformed their research practices, and were optimistic that AR has the potential to be used in cultural heritage research. More broadly, the reported desire for greater access to information about cultural heritage items online, as well as the reported importance of tangibility and online visualization of cultural heritage items, together suggest that emerging technologies like AR may present promising approaches for online cultural heritage research.
Acknowledgements
We thank Loryl MacDonald and Brock Silversides from the Thomas Fisher Rare Book Library for their collaboration and extending the usage of the collection of Jewish ephemera. We would also like to thank student collaborators Ben Ashley, Zeeanna Ibrahim, and Milad Monavvarian for their work on this project.
References
Akçayır, M., & Akçayır, G. (2017). Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review, 20, 1-11.
Mohammed-Amin, R. K., Levy, R. M., & Boyd, J. E. (2012). Mobile augmented reality for interpretation of archaeological sites. In Proceedings of the second international ACM workshop on Personalized access to cultural heritage (pp. 11-14). ACM.
Anghelescu, H. G. (2001). A bit of history in the library attic: challenges of ephemera research. Collection management, 25(4), 61-75.
Botden, S. M., & Jakimowicz, J. J. (2009). What is going on in augmented reality simulation in laparoscopic surgery?. Surgical endoscopy, 23(8), 1693.
Brumann, C. (2015). Cultural heritage. In International encyclopedia of the social & behavioral sciences (pp. 414-419). Elsevier.
Chung, N., Han, H., & Joun, Y. (2015). Tourists’ intention to visit a destination: The role of augmented reality (AR) application for a heritage site. Computers in Human Behavior, 50, 588-599.
Champion, E., & Rahaman, H. (2019). 3D Digital Heritage Models as Sustainable Scholarly Resources. Sustainability, 11(8), 2425.
Ciolfi, L., Petrelli, D., Goldberg, R., Dulake, N., Willox, M., Marshall, M., & Caparrelli, F. (2013). Exploring historical, social and natural heritage: challenges for tangible interaction design at Sheffield General Cemetery.
Dow, S., Lee, J., Oezbek, C., Maclntyre, B., Bolter, J. D., & Gandy, M. (2005). Exploring spatial narratives and mixed reality experiences in Oakland Cemetery. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology (pp. 51-60). ACM.
Hettiarachchi, A., & Wigdor, D. (2016). Annexing reality: Enabling opportunistic use of everyday objects as tangible proxies in augmented reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 1957-1967). ACM.
Jones, E. (2017). The public library movement, the digital library movement, and the large-scale digitization initiative: Assumptions, intentions, and the role of the public. Information & Culture, 52(2), 229-263. doi:http://dx.doi.org.ezproxy.lib.ryerson.ca/10.7560/IC52205
Jung, T. H., & tom Dieck, M. C. (2017). Augmented reality, virtual reality and 3D printing for the co-creation of value for the visitor experience at cultural heritage places. Journal of Place Management and Development, 10(2), 140-151.
Kenderdine, S. (2015). Embodiment, entanglement, and immersion in digital cultural heritage. A New Companion to Digital Humanities, 22-41.
Kim, S. K., Kang, S. J., Choi, Y. J., Choi, M. H., & Hong, M. (2017). Augmented-Reality Survey: from Concept to Application. KSII Transactions on Internet & Information Systems, 11(2).
Kim, S. L., Suk, H. J., Kang, J. H., Jung, J. M., Laine, T. H., & Westlin, J. (2014). Using Unity 3D to facilitate mobile augmented reality game development. In 2014 IEEE World Forum on Internet of Things (WF-IoT) (pp. 21-26). IEEE.
Kyriakou, P., & Hermon, S. (2019). Can I Touch This? Using Natural Interaction in a Museum Augmented Reality System. Digital Applications in Archaeology and Cultural Heritage, 12, e00088.
Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12), 1321-1329.
Reitsma, L., Smith, A., & Van Den Hoven, E. (2013). Storybeads: preserving indigenous knowledge through tangible interaction design. In 2013 International conference on culture and computing (pp. 79-85). IEEE.
Nofal, E. (2013). “Taking advantages of augmented reality technology in museum visiting experience.” In Proceedings of 6th International Congress “Science and Technology for the Safeguard of Cultural Heritage in the Mediterranean Basin,” 3, 1-8. Also available at https://lirias.kuleuven.be/2802267?limo=0
Wu, H. K., Lee, S. W. Y., Chang, H. Y., & Liang, J. C. (2013). Current status, opportunities and challenges of augmented reality in education. Computers & education, 62, 41-49.
Yin, R. K. (2015). Qualitative research from start to finish. Guilford Publications.
Appendix
Appendix A: Questionnaire
DEMOGRAPHIC:
- Name
- Research institution/University
- Research specialization?
- Expertise – number of years as researcher/specialist?
- What gender do you identify with?
DEFINITION: - Do you know what ephemera and archives are?
- How would you define both terms?
MOTIVATION: - [For researchers], what are the determining factors that motivate you to visit a distant library to see its ephemera?
PROCESS: [For researchers, librarians, and metadata specialists] - Could you give us a ratio of how often you handle physical ephemera versus how often you search online through online publications? For example, ejournals, and museum’s websites
- Can you explain what a typical research session [for a paper] looks like on any given day? (Get case studies).
- When researching, are you generally looking for one specific ephemera or do you search by another category? (E.g. era, location, usage, evolution in technology)
- If there were no limitations to how you’d research, what would a research session look like? (E.g. start with geographical map to pick location, timeline, etc.)
- What would you imagine are the benefits of working with physical ephemera?
- What do you think are the benefits of researching digitally?
- What would you say are the top 3 problems you have when researching about ephemera through online publications?
COLLABORATION: [For researchers] - How do you collaborate with other researchers while doing your own research? / How do you discuss what you discover from literature reviews or your findings from the physical collections?VISUALIZATIONS: [For all]
- [For researchers], how is the data you are working with currently presented to you? (E.g. open-source linked data vs. library’s search engines).
- What are the benefits of the current publications?
- What are their constraints?
- [For librarians], if a researcher is unable to make it to your library but is interested in one of your objects, what would make it possible for them to access it remotely?
- [For metadata specialists], what is the current categorization process for the ephemera at your institution?
- [For metadata specialists], Can you break down the digitization process in steps from archives to final digital platform?
- [For all], how do you visualize the data you are working with?
INSIGHTS: [For researchers] - What insights are you looking for when researching through a specific collection of ephemera?
- Do you look for more information about the context the ephemera was in? If so, what contextual information are you looking for?
- What physical properties are important for a researcher to know- especially if they cannot access the object in person but only digitally?
- Are you interested in knowing the meaningfulness of the object to the original owner? The collector?
- How do you make the relation between a piece of ephemera and contemporary society? Through literature reviews? Discussions? Are there other means?
TECHNOLOGY: [For researchers] - How do you record your research findings? This includes note-taking, database management, etc.
- What technologies are you used to using?
- E.g. Hardware: e.g. laptop and mouse, archaeological tools;
- E.g. software: Excel, MySQL, Omeka, Islandora, etc.
- What technologies are you used to using?
- How do you approach various ephemera during the digitization process?
- What are the visualizations you encounter for metadata? (e.g. digital repositories, open-source linked data visual representations, etc.)
- How have these visualizations helped your research?
- Do you have any struggles with the current visualization techniques?
- If so, what do you think could help with this process?
- Is there a different way of working with the data that you would find more suitable for research on ephemera?
-
- What functions and features would you want to have that would make the tool valuable in your research?
- For example, would you want the interface to work differently? To use different kinds of technology or tools to interact with digitized ephemera?
-
- What is your experience with AR/VR?
- Do you think an AR/VR experience will benefit or inhibit your research process? How do you imagine this would happen?
- Would you be interested in incorporating new or different technologies in your research process that can help simplify working with digital ephemera?
Appendix B: Analyzed Interview Questions
1) How do you define ephemera, archives, and artifacts?
2) What are the determining or motivating factors for you to visit ephemera or artifacts at an archival institution?
3) How often do you handle physical ephemera or artifacts compared to how often do you research online?
4) Can you explain what a typical research session looks like for you (in preparation of a publication or exhibition)?
5) What are the benefits of working with physical ephemera or artifacts?
6) What would you say are the top 3 problems you have when researching about ephemera through online publications?
7) How do you collaborate with other researchers while doing your own research?
8) How is the data you are working with currently presented to you? What is the cataloguing like? What are the benefits and constraints of this system?
9) How do you organize or visualize the data you are working with?
10) What kind of insights are you looking for when researching through a specific collection of ephemera?
11) How do you record your research findings? What tools do you use? This includes note-taking, data management, and the digital hardwares and softwares used.
12) Is there a different way of working with the data that you would find more suitable for research on ephemera?
13) What is your experience with augmented reality and virtual reality?
14) Would you be interested in incorporating new or different technologies in your research process that can help simplify working with digital ephemera?
Appendix C: Figure of User Journey with the TARA application

Appendix D: Results Tables with Emergent Themes
Participant Identifier | Participant’s Definition of Ephemera | Participant’s Definition of Artifact | Participant’s Definition of Archive |
A1 | T0B: Unaware of this term in relation to HCIs | T1: Physical/3D Object
T2: Important to CHIs T3: A Representation of Time T4: Context of Item/Location/Culture is Important |
T0A: No Response |
A2 | T5: Impermanent and Disposable
T6: Paper Based |
T1: Physical/3D Object
T7: More Durable than Ephemera |
T8: Place to Keep Documents
T14: Archives hold document and ephemera |
A3 | T5: Impermanent and Disposable
T6: Paper Based |
T1: Physical/3D Object | T8: Place to Keep Documents
T10: Holds items like Paper, Prints, Newspapers, Artworks, Photos T11: Archive Contents convey important information over aesthetics |
A4 | T5: Impermanent and Disposable
T10: Examples such as Paper, Prints, Newspapers, Brochures T4: Context of Item/Location/Culture is Important T6: Paper Based |
T4: Context of Item/Location/Culture is Important
T12: Sometimes Preserved T5: Sometimes impermanent / disposable |
T13: Museums hold artifacts
T14: Archives hold document and ephemera |
A5 | T14: Archives hold document and ephemera | T1: Physical/3D Object
T4: Context of Item/Location/Culture is Important |
T0A: No Response |
A6 | T14: Archives hold document and ephemera
T15: Importance of item only realized after passage of time T5: Impermanent and Disposable |
T0A: No Response | T14: Archives hold document and ephemera
|
A7 | T14: Archives hold document and ephemera
T10: Items like Paper, Prints, Newspapers, Artworks, Photos T6: Paper Based |
T13: Museums hold artifacts | T13: Museums hold artifacts
T14: Archives hold document and ephemera |
A8 | T4: Context of Item/Location/Culture is Important | T0A: No Response | T10: Holds items like Paper, Prints, Newspapers, Artworks, Photos |
Table 1.1: Participant Definitions of the Terms “Ephemera,” “Artifacts,” and “Archives”
Participant Identifier | What is your motivating factor/reason for visiting a site? | How often do you visit a site versus researching online? | What does a typical research session involve (process)? | Are there benefits to physical ephemera? | How do you collaborate when researching (if at all)? |
A1 | T1: To Identify Activities Performed, Time Period and Other Information contained within artifacts | T10: Doesn’t visit sites anymore
T11: Physical Literature and Site Visits were used more often than finding information in a digital catalogue T12: Personal Research involved digitizing physical objects |
T16: Layout a grid to map out locations of artifacts
T17: Identify Artifacts after excavation T12: Personal Research involved digitizing physical objects |
T0A: No Response | T0A: No Response |
A2 | T2: Would visit the site if information couldn’t be found elsewhere
T3A: Sites in Canada make up majority of visits T4: Pre-visit analysis like searching online catalogue before an in-person visit is necessary |
T0A: No Response | T4: Pre-visit analysis like searching online catalogue before an in-person visit is necessary
T13: Large portion of research done online T18: Photography exhibits/analysis are of primary concern |
T22: Online artifacts may be of lower quality than in-person
T23: Online artifacts may contain a smaller selection of a collection than in-person |
T30: No collaboration occurs normally. If it does, it is a simple idea exchange between colleagues. |
A3 | T5: Would visit archives for primary research
T6: Important to physically see a research object in person T7: Best or Only way to access artifacts is in-person sometimes T8: Viewing exhibits as an interest |
T10: Doesn’t visit sites anymore
T13: Large portion of research done online |
T0A: No Response | T6: Important to physically see a research object in person
T24: Tangible properties like weight, shape, engravings, etc are best appreciated and studied in person T25: Some online information may be less credible/erroneous than visiting a reputable site in person T18: Photography exhibits/analysis are of primary concern T20: Discuss research prompts with colleagues in pre-visit stage |
T31: Conferences and speaking events between peers are the main form of collaboration |
A4 | T5: Would visit archives for primary research
T8: Viewing exhibits as an interest T3B: Sites in Africa and Europe make up majority of visits instead of Canada |
T0A: No Response | T4: Pre-visit analysis like searching online catalogue before an in-person visit is necessary
T7: Best or Only way to access artifacts is in-person sometimes |
T6: Important to physically see a research object in person
T24: Tangible properties like weight, shape, engravings, etc are best appreciated and studied in person T23: Online artifacts may contain a smaller selection of a collection than in-person T26: No Benefit to Digitizing ephemera, in person is vastly superior |
T32: Frequent collaboration with curators of exhibits or collections if information is misorganized or if more access is needed, etc |
A5 | T1: To Identify Activities Performed, Time Period and Other Information contained within artifacts
T6: Important to physically see a research object in person |
T10: Doesn’t visit sites anymore
T13: Large portion of research done online T14: Started out by doing site visits, now finds research through literature and online |
T0A: No Response | T20: Discuss research prompts with colleagues in pre-visit stage
T6: Important to physically see a research object in person T1: To Identify Activities Performed, Time Period and Other Information contained within artifacts |
T33: Very collaborative process between colleagues and research specialists |
A6 | T5: Would visit archives for primary research
T8: Viewing exhibits as an interest T9: External factors/ clients motivate a visit |
T11: Physical Literature and Site Visits were used more often than finding information in a digital catalogue
T14: Started out by doing site visits, now finds research through literature and online T4: Pre-visit analysis like searching online catalogue before an in-person visit is necessary |
T19: As a curator, participant is in charge of rotating exhibits
T1: To Identify Activities Performed, Time Period and Other Information contained within artifacts T9: External factors/ clients motivate a visit |
T0A: No Response | T32: Frequent collaboration with curators of exhibits or collections
T33: Very collaborative process between colleagues and research specialists |
A7 | T5: Would visit archives for primary research
T7: Best or Only way to access artifacts is in-person sometimes T4: Pre-visit analysis like searching online catalogue before an in-person visit is necessary |
T14: Started out by doing site visits, now finds research through literature and online
T13: Large portion of research done online |
T0A: No Response | T27: Easier to access and more convenient to use online artifact collections
T28: Ephemera and 2d documents are fine to view online but 3D artifacts would be useful to view in-person |
T34: Mostly a solitary effort, but collaboration is used when studying an area that is not familiar or difficult to approach |
A8 | T5: Would visit archives for primary research
T6: Important to physically see a research object in person T1: To Identify Activities Performed, Time Period and Other Information contained within artifacts |
T4: Pre-visit analysis like searching online catalogue before an in-person visit is necessary
T13: Large portion of research done online T15: Detriments of site visits such as budget, time it takes, etc |
T18: Photography exhibits/analysis are of primary concern
T4: Pre-visit analysis like searching online catalogue before an in-person visit is necessary T20: Discuss research prompts with colleagues in pre-visit stage T5: Would visit archives for primary research T21: Research would be demonstrated through academic talks or curation of exhibits |
T27: Easier to access and more convenient to use online artifact collections
T6: Important to physically see a research object in person T24: Tangible properties like weight, shape, engravings, etc are best appreciated and studied in person T29: Organization of items in folders and a hierarchy as well as saving on money is also a benefit of online archives |
T34: Mostly a solitary effort, but collaboration is used when studying an area that is not familiar or difficult to approach |
Table 1.2: Emergent Themes for Participants Collaboration, Motivations and Process
Participant Identifier | How is the data you encounter presented to you? | How do you visualize/ analyze data you are working with? | What insights are you looking for? |
A1 | T0A: No Response | T7: Use a graphical or spreadsheet database to define relationships and identify patterns
T8: Scanning and Digitization of ephemera occurs |
T12: Object Centric – Looking for who created and who used certain tools, as well as their location of origin, time period, etc. |
A2 | T1: Inconsistency in cataloguing – sometimes fully done and sometimes not
T2: Rarely, information is incorrectly recorded for a catalogue |
T9: Mostly primary research, archival books and making notes on descriptions and relevant information | T13: Device Centric – Looking for how certain early devices functioned, the inventor, etc. |
A3 | T0A: No Response | T0A: No Response | T0A: No Response |
A4 | T0A: No Response | T10: Dropbox/Shared Drive used to organize groups of photos and information between researchers and colleagues
T7: Use a graphical or spreadsheet database to define relationships and identify patterns |
T0A: No Response |
A5 | T0A: No Response | T11: Projects organized according to a “production pipeline” model with budget, schedules, file organization. | T14: Narrative Centric – Focused on the story/author’s life behind an artifact |
A6 | T4: Files are encountered that are filled with clippings and various types of ephemera | T0A: No Response | T12: Object Centric – Looking for who created and who used certain tools, as well as their location of origin, time period, etc. |
A7 | T5: Data process begins by searching internal repository
T4: Files are encountered that are filled with clippings and various types of ephemera T6: Collaborators and Curators of other exhibits and archives often aid in data collection process. |
T10: Dropbox/Shared Drive used to organize groups of photos and information between researchers and colleagues
T7: Use a graphical or spreadsheet database to define relationships and identify patterns |
T12: Object Centric – Looking for who created and who used certain tools, as well as their location of origin, time period, etc. |
A8 | T1: Inconsistency in cataloguing – sometimes fully done and sometimes not
T6: Collaborators and Curators of other exhibits and archives often aid in data collection process. |
T0A: No Response
|
T14: Narrative Centric – Focused on the story/author’s life behind an artifact |
Table 1.3: Methods that Participants use for Visualization and Insights
Participant Identifier | What kinds of technology and tools are you using throughout your process? | What would be your preferred way of dealing with data/ artifacts? | What is your experience with AR/VR? | Would you like to add new or different technologies to your research? |
A1 | T0A: No Response | T8: In Person is the main way to experience ephemera, digital makes it harder to replicate the collection accurately
T9: AR/VR would be promising for visualizing ephemera T10: 3D Printing at a very high degree of quality and accuracy has promise. |
T9: AR/VR would be promising for visualizing ephemera
|
T0A: No Response |
A2 | T1: Computer, Keyboard and Mouse
T2: Antique Cameras and associated analysis technologies |
T11: Print Catalogues and Collections are primary media used | T18: Researcher sees little use for VR/AR and has limited experience dealing with VR/AR | T0A: No Response |
A3 | T3: Archaeological Tools – Carbon Dating, Microscopes and Ultraviolet light systems | T12: It would be beneficial if more artifacts, photos and their associated information and metadata appeared online. | T19: 3D Imaging, AR and VR have potential, but in certain instances the effort isn’t worth it for particularly plain ephemera or artifacts or basic items | T21: High degree of detail, zoom and photo manipulation would vastly improve online catalogues and would be very exciting to this researcher |
A4 | T0A: No Response | T8: In Person is the main way to experience ephemera, digital makes it harder to replicate the collection accurately T12: It would be beneficial if more artifacts, photos and their associated information and metadata appeared online.
T13: In-depth examination of artifacts should appear in person but some online information and basic visualizations are still useful |
T18: Researcher sees little use for VR/AR and has limited experience dealing with VR/AR | T22: Researcher is very “old school” and doesn’t feel the need to add new technologies to their arsenal |
A5 | T4: Book Research
T1: Computer, Keyboard and Mouse T5: 3D Scans, Drone scans and Radar Scans |
T14: Important to get differing perspectives, the ideal method of data analysis involves working closely with cultures and people who have lived experience in this area | T20: Researcher has a lot of experience developing for VR and see promise, but believes AR still has a long way to go before being available and viable on a large scale | T23: Technologies that pair with AR/VR and specifically focus on smell and sound would vastly improve ephemera and artifact collections by delivering a heightened sensory experience that is currently unavailable with traditional research methods |
A6 | T6: Excel Spreadsheets and Shared Online Drives
T1: Computer, Keyboard and Mouse |
T15: Having a variety of artifacts in a more centralized and accessible location is important, no matter the technology utilized for this. | T18: Researcher sees little use for VR/AR and has limited experience dealing with VR/AR | T0A: No Response |
A7 | T1: Computer, Keyboard and Mouse | T8: In Person is the main way to experience ephemera, digital makes it harder to replicate the collection accurately
T9: AR/VR would be promising for visualizing ephemera T16: Must have a tangible way to rotate and interact with ephemera to simulate holding the artifact in real life |
T9: AR/VR would be promising for visualizing ephemera
T19: 3D Imaging, AR and VR have potential, but in certain instances the effort isn’t worth it for particularly plain ephemera or artifacts or basic items |
T12: It would be beneficial if more artifacts, photos and their associated information and metadata appeared online. |
A8 | T7: Tripod, Camera and Associated Equipment | T17: Multiple databases accessible online via a unified platform with universal keywording | T18: Researcher sees little use for VR/AR and has limited experience dealing with VR/AR | T0A: No Response |
Table 1.4: Participants Opinions on Emerging and Preferred Technologies for Research
Cite as:
Kalinda, Daniella, Hrynyshyn, Lucas, Resch, Gabby, Nathan, Anitha, David, Ravit and Mazalek, Ali. "Tangible Augmented Reality for Archival Research: Using Augmented Reality to Research Cultural Heritage Items." MW20: MW 2020. Published January 15, 2020. Consulted .
https://mw20.museweb.net/paper/tangible-augmented-reality-for-archival-research-using-augmented-reality-to-research-cultural-heritage-items/