adventures in augmented reality

Spatial Importance

Greeno’s 1991 paper highlights the idea of conceptual domains where we each hold a virtual environment in our mind that envisions relationships between objects. Greeno was particularly interested in what he called “number sense,” a term that refers to people’s ability to treat numbers as symbolic objects that can be manipulated and understood. He went on to highlight the importance of these conceptual domains; people build mental models to understand the behaviors and properties of actual phenomena. The mental objects themselves become not tangible as if real. Subsequently, individuals with high spatial ability tend to perform better in mathematics (Battista, 1994). Battista hypothesized that spatial activities could improve students’ mathematical thinking and was proven later to be correct. In an experiment conducted in Norway, Nordvik, et. al, (1998) showed that women who initially scored low on spatial perception tests performed higher after engaging in computer “games” that required the women to rotate and manipulate objects in three dimensions. Later tests revealed the increase in ability appeared to be permanent.A likely vehicle for improving mental modeling is the three dimensional environment of Virtual Reality (VR). To build successful conceptual domains, initial mental models must be accurate, otherwise the student will have backtrack and relearn concepts and skills or remain confused. Children come to the classroom with a basic understanding of how the world works based upon the child’s own observations, misconceptions, and absorbed facts (Winn, et. al., 2000), but a solid foundation is imperative for successful scaffolding. For example, most children come to the classroom unaware of the sun as anything else then a small, glowing disc (unless otherwise instructed) that the child will often illustrate as a yellow circle with rays. The child has already established a working model with which to understand the nature of the sun as it relates to the child’s life. VR models can catch such difficult to illustrate concepts early on and establish accurate models that will aid in later knowledge building.

Many concepts in science are impossible to observe first hand. Atoms are too small to examine and planetary bodies are too far and too large to fully appreciate. Barab, et. al’s., (2000) virtual solar system demonstrates the advantages of using extreme scale in education. By interacting objects roughly the size of a baseball, the student can easily observe and create workable mental models of such non-intuitive physical phenomena like the earth’s rotation around the sun. Winn refers to the concept of modeling objects that are abstract in phenomena and lack tangibility, as “reification.” Through reification, the student is able to virtually experience phenomena impossible or very difficult to simulate in the real world.

Exploring the Virtually Tactile

Aside from the obvious advantages of creating interactive virtual science for students to encounter, is the alluring concept of presence. The act of actually “being there,” and “experiencing” science rather than just passively observing or interpreting through textbook descriptions enables the student learn and discover for him/herself – a happy nod to constructivism. The importance of presence is paramount to the success of VR and positively correlates with learning success (Winn, et. al., 2000). Additionally, the exciting nature of VR most likely elicits an emotional response and encourages motivation. LeDoux’s research has linked adrenaline release (a result of excitement) with high memory recall.

The excitement of interactivity is a large part of the VR experience, but how important is the ability to interact and move VR models in aiding student learning? According to Dede, et. al. (1997), quite important. MaxwellWorld, a virtual world designed by students from the University of Maryland, is a virtual world where students can witness and manipulate electromagnetic phenomena and reported these findings: “Manipulating the electric field in 3-D appeared to play an important role in students’ ability to visualize the distribution of energy and force. For example, several students who were unable to describe the distribution of forces in any electric field prior to using MaxwellWorld gave clear descriptions during the post-test interviews and demonstrations.” Manipulation improved student recall and created clear, working models for the students to internalize. Diezmann, et. al., (1999) believes the importance of movement to spatial intelligence is akin to Vygotsky’s inner speech. For example, Greeno reports showing one physicist a diagram of blocks and a pulley, and then asking him exactly was happening in the picture (Greeno, 1991). Although it was clear the physicist was recalling sophisticated mental models, he used gestures to explain and represent forces at work.

During scientific exploration, student gestures often mimic manipulations and understanding achieved during tactile discovery learning. This sort of “kinesthetic inner speech” is integral to complex scientific comprehension (Roth, 2001). Successful mental models allow for successful scaffolding which acts as a hook for more complex comprehension (Taylor, 1996) and may link the disconnect between what Scaife & Rogers (1996) call, “knowledge in the head” and “knowledge in the world” or internal and external representations. Scaife & Rogers acknowledge the paucity of data regarding visual processing and new media (animations, VR), but advise designers to create visuals based on interactivity. They conclude that “solvers” use diagrams by mentally transforming parts of the diagram to reason through tasks. “In doing so the solvers no longer need to solve the problems entirely in their head but can work them out by interacting with the diagrams” (Scaife & Rogers, 1996; pg. 193) The increased speed at which people do this versus non-visual diagrams is a result of computational offloading (Scaife & Rogers, 1996).

Creating Mental Models through Interactivity

The advantage of two-dimensional diagrams is fluidity; students can mark and make notes that aid in the understanding of depicted concepts. But what is this accomplishing, and how can interactive diagrams achieve the similar results in learning? “Diagrams that have been already constructed allow the user to leave cognitive traces, i.e. mark, update and highlight information. However this is a limited function. There is no possibility of interaction or feedback—the user cannot test new configurations. In contrast, when interacting with animations and virtual reality objects there is more scope for providing feedback but less for leaving cognitive traces. For example, various parameters of a computer-based model can be set in a virtual reality or three-dimensional simulation (e.g., microworlds) and the outcome directly observed. Graphical representations should be designed with a view towards how they support different kinds of cognitive tracing and levels of interactivity.”(Scaife & Rogers, 1999 pg 288)

A VR application that enables cognitive tracing through interaction is a logical step in the evolution in the technology. Although VR has shown to aid in student learning (Winn), little research exists that examines visual processing and subsequent effects on mental models. What occurs in mental modeling is not only a reinterpretation of real world objects displaying realistic behaviors, but also understandings that are contrived and imagined. Battista uses the example of the child who envisions a rectangle as a “square that’s been stretched.” Bauer and Johnson-Laird (1993) explain: ‘‘in the case of the diagrammatic problems, the subjects form a visual representation of the diagram, and in their mind’s eye they can imagine moving the pieces or switches (i.e. they carry out visual transformations of images).” The ability to do such mental manipulation is repeatedly regarded in the literature as an important part of abstract comprehension. VR reinforces, corrects, and builds upon mental models by externalizing an internal process. One would wonder if student shortfalls in the areas of math and science would greatly benefit from VR to aid in mental modeling.

A combination of visual and audio-textual information and its subsequent effect on memory recall have been documented (Tversky, Hegarty, Moreno et. al.,), although with differing conclusions. Tversky (1998) believes that the memory increase is a result of visual processing. She later makes the argument that building a mental model through verbal (text, auditory) is just as powerful as seeing the image itself. One study conducted by Moreno & Mayer (2002) concluded that speech was more effective than text in a VR lesson, and found that was the case regardless of delivery medium. Hegarty (1996) maintains that visuals reinforce the image of the labels; rather it is the shape of the text that is imprinted. What emerges from these findings is the importance of visual diagrams and the importance of textual/verbal reinforcement to successful mental modeling and subsequent comprehension. Although research in the use of diagrams or remains largely unexplored and is virtually non-existent as it applies to VR, a strong argument for diagram use in VR continues and more research is needed.

AR Design

The role of immersive virtual reality in education, although extremely interesting and exciting, continues to elude the educational system due to cost and its legendary resistance to new technologies. However, the prospect of using portable Augmented Reality (AR) applications appears to be far more feasible. Using the University of Washington developed ARToolkit (Augmented Reality Toolkit), it is now possible to create and distribute AR applications using a home computer without any additional equipment other than a camera. The camera is suspended above an image marker that signals the computer to project a programmed object in the vicinity of the marker. The result is a dynamic three-dimensional displayed on the screen. The user is then able to interact with the virtual object by watching the monitor while picking up the actual marker.

Presence in AR is arguably not quite as high as with use in VR. However, Moreno & Mayer (2002) found less immersive environments (AR) to be just as effective as full immersion (VR with headset) in their study. AR is without many of the hang-ups that sometimes discourage VR use such as nausea, dizziness, and discomfort due to equipment use. What AR does allow is inexpensive and tactile virtual modeling that engages the student and encourages discovery. The success of two-dimensional diagrams in memory recall as well as Scaife & Rogers recommendation to combine visuals and interactivity specifically to support cognitive tracing, result in a strong argument to examine the use of interactive diagrams in VR.

IDEA

The ramifications for such a tool are exciting, particularly in the realm of science education. Inspired by books such as The Incredible Book of Cross-Sections and other complex cut-away images, I set to create a simple AR model of the sun that allowed for internal stellar exploration. This model would theoretically allow students in the range of grade 1 to 3, to tangibly pick up and “look inside” the sun. The inside of the virtual model would contain a cross-section with each strata labeled and color-coded. The sun example, along with the rest of the solar system, would be contained in a book containing age-appropriate information, colorful graphics, and AR markers for each planet. Theoretically, the result would be a complete cognitive package of visual/textual/kinesthetic learning materials resembling that lauded by Scaife & Rogers and Roth.

RESULT

Running ARToolkit proved far more complicated than it initially seemed. The greatest problem was rounding up the necessary resources and researching where to begin. However, after much web surfing and many consultations with programming friends (and the husband), the ARToolkit with VRML support (only in version 2.5 and NOT 2.7, argh) was quite friendly to use. Three-dimensional modeling was created using the open source program Blender. Textures and diagrams were created using Adobe’s Photoshop and Illustrator. Because of time restraints, only the sun was created and tested. Readability continues to be an issue, but can most likely be resolved with more work. Observed color and texture clarity were remarkably high. Unfortunately, the camera proved quite jerky and lag in interaction remains quite problematic. The virtual sun’s range of motion was only around 10 degrees which made the object difficult to “explore.” Future plans to expand this initial curriculum model begin with the planets in the solar system and would later include more complicated structures like galaxies and supernovae.

FURTHER ADVENTURES

With additional time and expertise, further applications could apply to more complex structures such as those depicted in Diestry’s amazing cross-section illustrations on the extreme end, and basic science applications at the more basic level. Creating a model with a much greater range of motion seems feasible using a cube instead of a flat plane. Images pasted on all but one side could possibly create such an object if transition points (where the edges of the cube meet) could be calculated and displayed. Aside from theoretical creations, an underlining research goal is to further explore the importance of kinesthetic motion and its connection to envisioning mental models as reported by Dede et. al., (2001) and Greeno (1989).

Bibliography

Barab, S.A., Hay, K.E., Squire, K., Barnett, M., Schmidt, R., Karrigan, K., Yamagata-Lynch, L., & Johnson, C. (2000). The virtual solar system: Learning through a technology-rich, inquiry-based, participatory learning environment. Journal of Science Education and Technology, 9, (1), 7-25.

Battista, M., (1994) On Greeno’s Environmental/Model View of Conceptual Domains: A Spatial/Geometric Perspctive Journal for Research in Mathematics Education, Vol. 25, No. 1 86–94

Bauer, MI & Johnson-Laird PN., (1993)How diagrams can improve reasoning. Psychological Science 4 , 372 – 378 .

Dede, C., Salzman, M., Loftin, R. B. & Ash, K. (1997). Using virtual reality technology to convey abstract scientific concepts. In M. J. Jacobson & R. B. Kozma (Eds.) Learning the sciences of the 21st century: research, desMahwah, NJ: Erlbaum.

Diezmann, M. Carmel & Watters, J. James (2000) Identifying and Supporting Spatial Intelligence in Young Children. Contemporary Issues in Early Childhood, November, 2000

Greeno, J. G. (1991) Number Sense as Situated Knowing in a Conceptual Domain. Journal for Research in Mathematics Education 22, 170–218

Roth, W-M. (2001). Situating cognition. The Journal of the Learning Sciences, 10, 27-61

Hegarty, M., Just, M.A., Constructing Mental Models of Machines from Text and Diagrams Journal of Memory and language 32, 717–742 (1993)

LeDoux, J. (1999) The Emotional Brain, MIT Press

Moreno, R. & Mayer, R. E. (2002) Learning Science in Virtual Reality Multimedia Environments: Role of Methods and Media Journal of Educational Psychology Vol 94, No 3, 598–610

Nordvik, H., Amponsah, B., (1998) Gender differences in spatial abilities and spatial ability among university students in an egalitarian educational system Sex Roles, a Journal of Research

Scaife, M., & Rogers, Y. (1996). External cognition: How do graphical representations work? International Journal of Human Computer studies, 45, 185-213.

Stock, W.A., Kulhavy, R.W., Peterson, S.E., Verdi, M.P., (1995) Mental Representations of Maps and Verbal Descriptions: Evidence They May Affect Text Memory Differently Contemporary Educational Psychology 20, 237-256

Taylor, H. A., Tversky, B., (1996) Perspective in Spatial Descriptions Journal of Memory and Language 35, 371–391

Tversky, B., Franklin, N., Taylor. A. Holly, Bryant, J. David (1994) Spatial Mental Models from Descriptions. Journal of the American Society for Information Science 45(9):656-668

Winn, W.D., & Windschitl, M. (2000). Learning science in virtual environments: The interplay of theory and experience. Themes in Education, 1, 373-389.

Winn, W.D., Learning in Interactive and Immersive Environments. Paper Presented at the Conference Media and Higher Education, National Institute for Multimedia Education, Chiba, Japan, November, 1995.

Winn, W., (2002)Current Trends in Educational Technology Research: The Study of Learning Environments Educational Psychology Review, Vol. 14, No. 3, September 2002