Usage of Computerized Simulations and Interfaces vs. Analog Testing in Visual Imagery, Representation, and Mental Rotation Batteries
by John Richardson
Emerson College Game Developers Association and
Dept of Communication Sciences and Disorders
Draft One, March 2008
Though its question goes back to the beginnings of psychology, the phenomenon of mental representation and visual imagery in cognition has seen a hotbed of scientific research over the past several decades. As with many other psychological topics of the same time period, it has answered many questions as to how we represent our world, though it remains one of the most fiercely debated. Indeed, as with any other field of recent study within cognitive psychology or neurology, the visual and mental construction of imagery is one that directly impacts the everyday lives of all intelligent forms, and has a high correlation to other psychological occurrences, issues, and methods. Although the large empirical body of work combined with a better working model of the brain has helped to solidify how visual representation functions, it has “presented a challenge for mainstream cognitive psychology by generating new kinds of theory concerning [representations] and new methods for investigating those representations”, as John T.E. Richardson (1989) purports. Its study has also revealed tangential analysis into the mechanics of gender differences, learning, sensation, and even speech, making its continued study important not just for its own field but also for a larger body of cognition research. For this specific analysis, an emphasis will be placed on the evolution of this field in terms of the advent of germane technologies, particularly computer simulations, imaging devices, and human computer interfaces. Such modalities have had a very recent impact on the analysis of and interaction with imagery’s foundations, shown several forms of results, and most importantly, have many implications for conventionally practiced analog testing and application.
What makes mental representation and imagery so interesting, and so complex, is not just that it has ramifications for many of the fundamentals of psychology and cognition, but that its effects reach deep into human philosophy. By determining how a person views, encodes, and then interprets an image, we unlock valuable answers as to how we comprehend the world around us from a practical standpoint. As a result of this, we could perhaps understand what makes us “tick”, from not just a humanistic standpoint, but in terms of ethics, vocation, relationships, politics, and even consumption. One of the difficulties in doing this, of course, is that “image” is not only difficult to define, but also a subjective concept to begin with; representation varies from person to person and community to community, and manifests itself in an infinite number of ways. From a scientific standpoint, a mental image is, as of this writing, still impossible to physically capture, and can be measured only through verbal or written tests or neuroimaging batteries such as Functional Magnetic Resonance Imaging (fMRI) or Positron Emission Topography (PET), all of which have their own flaws and limitations, such as not being able to account for human error or measuring only lateral brain activity. What has been attempted in the last few decades, however, are methods that allow psychologists not necessarily to see these images as they are stored, but recognize how we dynamically process, visualize, and manipulate them. One of the most cited and earliest developments of these methodologies was conducted by Jacqueline Metzler and Roger Shepard, who determined how the process of mental rotation occurs, and were particularly successful, along with subsequent studies, in recording “eye fixation patterns, in conjunction with reaction-time data, in interpreting the nature of a variety of cognitive processes, including mental rotation” (Shepard & Cooper, p. 171). Such salient methodologies led to a multitude of widely varying examinations of related mental transformations, but also are the drive of an ongoing debate as to the exact processes and substrates involved in such activity.
The debate over mental imagery has nothing to do with if we develop images in our minds. The mechanisms that activate, as observable via neuroimaging (PET, etc), in the visual cortex and cortical “interconnectors” such as the thalamus and hippocampus, demonstrate that we indeed do (though there is still debate as to role assignment). What is centrally contested is how exactly we take these purely visual images and store, manipulate, and then recall them, particularly in terms of representation and its treatment as a cognitive entity. As Psychologist Stephen Michael Kosslyn professes, the “anti-image” thrusts of this specific debate, fueled by such figures as Zenon Pylyshyn, includes (as aforementioned) the idea that image in itself is a flawed logic, and that its domain is no different than those used for language. This argument falls under what is referred to as “propositional representation”, whereby images are merely abstract and relational syntactical arguments. The opposing argument to this, one would then assume, classifies mental representation as pictures in the classical sense. Rather, this argument is “depictive” or “quasi-pictorial”, where mental images are not pictures as we think of them in a real-world sense, but rather delicate abstractions of such, in how they lack many of the latent qualities of a picture. The latter argument is that which has been most widely adopted and accepted among those in the field, especially as the technology with which such topics are studied have weakened propositionalism’s qualities, such as its medium-disinclination and non-topography. As we now understand, we store images spatially and a portion of an image corresponds to an object in the sense of second-order isomorphism (Kosslyn, 1980 and Rollins, 1989). Further issues, such as the role of tacit knowledge, modes of interpretation, and types of images, cloud this debate even further, demonstrating its complexity, and in some ways, infancy.
Technology, insofar as testing and analysis, has played a large role throughout these topics. Recently, updated methods for such assessment have been developed, and there is a growing body of research that demonstrates that active exploration of a scene or object has a marked benefit in the learning and interpretation of such constructs. These developments, while not all utilizing virtual worlds and computer-generated images in tandem with human computer interfaces, are essentially centered on the theory that motor output in concurrency with visual input leads to a more accurate mental representation of the matter being studied (James, Humphrey, & Goodale, 2001). In many ways, this is not a new concept; the simple practicality that is real-world exploration and its benefits for comprehension is something that has been practiced throughout education and occupations for much of modern history. In this sense, the adaption of these technologies from analog techniques for psychological testing has overreaching possibilities for more everyday uses, such as classroom-based instructional objectives with the use of virtualized environments or interactive games. On a more empirical basis, updated versions of classic tests such as Shepard & Metzler’s (1971) classic rotation task are now allowing such imaging techniques as fMRI to be levied in better concert with mental representation and visual complex tasks (Voyer, Butler, Cordero, Silbersweig, Stern, 2006). Since it is pointless to discuss advantages and disadvantages of such interfaces if they do not reflect the same accuracy that analog tests have shown in recent history, it makes the most sense to first analyze the validity of such standard computerized tasks.
Voyer et al. devised such a trial based of long-prescribed methods, in which hundred and fifty-seven undergraduate students were asked to complete a computerized MRT task along with the Spatial Relations subtest of the Primary Mental Abilities and the MRT as was given by Shepard & Metzler. Their discovery was that both the computerized and analog versions of such tests measure the same aspects of spatial behavior, with the results providing “clear preliminary evidence that the computerized mental rotation task is a valid measure of underlying abilities”, and that it is safe to assume such tests can be used safely within fMRI, given the close reflection of the performance that would be outside such neuroimaging. Procedurally, the computerized tests were successfully validated on the same basis that analog ones have been, this time on the basis of response time and accuracy. A small enough correlation in response time between the forced intervals on the computer and subject-driven time constraints on the paper tasks was found for it to show discriminant validity, and a high correlation between the latter test’s accuracy and the subject’s mathematical skills demonstrated concurrent validity, making its measure legitimate by established rubric. Perhaps most importantly, though, was the implication that because paper-and-pencil tests mimic scholastic measures themselves (such as grading, assignments, etc), a computerized test may be a better indicator of native spatial abilities that are not obscured by task fluency. In this case, it seems, there is now a valid replacement, or – at the very least – an alternative to the abovementioned analog tests in using such an interface, ideally if the administrator needs far better control and accuracy over task execution.
Usage of such computer interfaces throughout the study of mental imagery and representation is not at all exclusive to testing mechanisms alone. Clearly, one can assume that the seamless, everyday or disruptive usage of computer and digital interfaces has a prolonged effect on a plethora of aptitudes, as has been brought up throughout the development of our current modalities, though perhaps mostly from sociology or politics, and not cognitive psychology. They have also shown solvency in pre and post-test factor usage and analysis, demonstrating the magnitude of spatial function similar to quasi-pictorial cognitive theory, and in the methodology of examining imagery strategy development, as demonstrated by a few very recent studies.
De Lisi & Wolford (2002) gathered forty-seven third-graders and measured mental rotation ability much as was done above using the French Kit Card Rotation Test, except that instead of showing a series of objects to the right of a target referent, they used just one per reference, asking the children to decide whether the additional figure was different or identical and merely rotated. The test was administered twice – once before the experimental and control groups played “Tetris” and “Carmen Sandiego”, respectively, for eleven sessions, and once after. As with any normative Mental Rotation Test, the gender differences were significant even at such a young age, but far more important was that the “Tetris” players showed a significant increase in their MR scores proportionate to the final game score, implying that they were applying the same solutions used to achieve success in the game to spatial reasoning. These results were not achieved by the control group, which brings up the question of if the application of any type of computer game or application will have positive benefits for specific representational and spatial functions, or if such devices must target respective abilities in order to show any measurable corollary. Surely, a game such as “Carmen Sandiego”, which revolves around trivia games in very specific scholastic subjects and deduction, does not engage the same specialized cortical functions; the researchers do not engage this discrepancy. Less pressing questions remain such as how this sort of study would result in less formative age groups with more mature abilities, or how long these positive effects last after playing has ended.
Though testing using computer media brings with it many distinct advantages and revelations in terms of both methodologies and outcomes, the actual application of it in improving mental representation, spatial orientation, visualization, and “imagery reasoning” is where we uncover its most broad utility. Virtualized and computer environments that exercise and improve educational computing, spatial, or visual motor ability, or expand the cognitive map of a subject with limited mobility, to name only a few examples, have been specially developed in recent years (Smith, Morey, & Tjoe, 2007). As is the case in the argument for comprehension-targeted testing (abovementioned), allowing a subject to explore an environment in media-based learning or classrooms, it would be assumed, allows them to construct a mental representation based on the idea that passive observation is never as effective as unrestrictive methodology. Somewhat contrary to this assumption, however, such tactics using three-dimensional models, for example, show no discernable advantages or disadvantages over alike two-dimensional tasks, though this may be less a result of the model itself and more the user’s native spatial ability. Consequently, while three-dimensional variants prove useful in the subject’s learning ability, exactly the practicality of this is best extended by those with high spatial ability, since the resources they must dedicate to this kind of visual construction is less than those with relatively low such skills (Huk, 2006). This is an indication that any virtual modality for learning systems across a variety of disciplines has some benefit, though a per-user discrepancy must be considered when deploying such utilities across a broad demographic. It is impossible to control or ensure the same result for every user of any analog model, and the same goes for computerized versions of such.
Studying figurative and sometimes statistical dimensions of mental imagery, while certainly important for determining its neurological basis, tends to rely heavily on reducing error variance and does not account for the high inconsistencies that are important to the basis of human expression and thought. On a far less operational level than depicted in some of the aforementioned studies, these models lack recognition of the subjectivity and inherent creativity of the human mind, the latter being a cognition that can be measured – at least on an abstract level – through novel models of image emergence. Clearly, the computer interface-focused examinations of the likes of De Lisi et. al and Smith et. al showed some success in applying new media to the evaluation of mental representation, but little was said of how subjects interpreted the images they were encountering and encoding. Thus far, all that has been shown through the use and assessment of computerized simulations is that they likely develop the exploratory patterns of subjects for reference images, and therefore improve subject accuracy and performance in tasks. Finke, Ward, & Smith (1992) contend that such phenomenon is a result of a multitude of disparate processes, known as “creative cognition”, and cannot be chronometrically assessed. Though this is an overstatement of the flaws of mental testing, the message is that the field needs to better explore the causes of changes in imagery visualization in different modes, rather than simply marking the results.
The concept of image emergence, whereby humans are constantly creatively changing and interpreting the world around them in ways not measurable through traditional metrics, is a relatively new one and still somewhat stigmatized as a scientific field. Though some of this preliminary trepidation has and will continue to naturally wear down, it is important for researchers to now realize that much of what they are studying may not be best described through trial. Such observation intrinsically must contain boundaries, while the process of mental representation primarily is made up of highly personal construal dictated not just by a subject’s background or culture but by indescribable transparent heuristics that color their experiences. In essence, the forceful formulization of what is essentially a mental abstraction seems irrational. A compelling way to study mental imagery may be by asking a subject to evaluate their own visual cognition. Computerized tasks are inherently attuned to this sort of method. Using a variety of emergent, dynamically generated computer images, for example, researchers could exercise such creative cognition at a per-user level. The examiner would not be trying to create consensus or averages, but indeed would be encouraging mental formulaic deviation, while attempting to understand the patterns by which such divergence occurs. The capability of a computer to mimic mental transformational and generative processes in real-time provides further reason to carry out such mediated tasks. At the conceptual level, this seems somewhat generalized and not easily broken down, and would require the procurement of a variety of new scientific analytical methods. Given the lack of certainty over mental cognition related to imaging, which appears to be linked to a field of unexplored constituencies, it is exactly the kind of methodological transformation that the study needs.
References
De Lisi, R., & Wolford, J. (2002, September). Improving children's mental rotation accuracy with computer game playing. Journal of Genetic Psychology, 163(3), 272-282. Retrieved February 26, 2008, from PsycINFO database.
Finke, Ronald A., Ward, Thomas B., Smith, Steven M. (1992). Creative Cognition: Theory, Research, and Applications. Cambridge, MA: MIT Press.
Huk, T. (2006, October). Who benefits from learning with 3D models? The case of spatial ability. Journal of Computer Assisted Learning, 22(6), 392-404. Retrieved March 1, 2008, from PsycINFO database.
James, K., Humphrey, G., & Goodale, M. (2001, June). Manipulating and recognizing virtual objects: Where the action is. Canadian Journal of Experimental Psychology, 55(2), 111-120. Retrieved March 2, 2008, from PsycARTICLES database.
Kosslyn, Stephen Michael (1980). Image and Mind. Cambridge, MA. Harvard University Press.
Richardson, John T.E. (1999). Cognitive Psychology, a Modular Course: Imagery. East Sussex, UK: Psychology Press Ltd.
Rollins, Mark (1989). On the Limits of Cognitive Science: Mental Imagery. New Haven, CT: Yale University Press.
Shepard, Roger N. & Cooper, Lynn A. (1982). Mental Images and Their Transformations. Cambridge, MA: MIT Press.
Smith, G., Morey, J., & Tjoe, E. (2007). Feature masking in computer game promotes visual imagery. Journal of Educational Computing Research, 36(3), 351-372. Retrieved February 26, 2008, from PsycINFO database.
Voyer, D., Butler, T., Cordero, J., Brake, B., Silbersweig, D., Stern, E., et al. (2006, August). The Relation between Computerized and Paper-and-Pencil Mental Rotation Tasks: A Validation Study. Journal of Clinical and Experimental Neuropsychology, 28(6), 928-939. Retrieved February 26, 2008, from PsycINFO database.