visual research

Working with research images in a time of AI and big data

My ongoing doctoral research into multimodal assessment includes diverse data, of which a substantial amount is visual. That is, the data includes still or moving images taking into account colours, design elements as well as embodied modes (such as gaze and gestures).

If extracts from this kind of data should be included in any form of research publication (of course with consent from participants) images need to be processed to maintain the confidentiality of research participant. If people are present in an image that should be used for evidencing or illustrating some argument in the research, this person or these people need to be represented in a way that does not reveal their identity but still visualises the example or argument being made.

With the increasing use and capabilities of AI how we use and publish images need careful consideration. To reduce the risk of data being extracted from images I am now ‘doctoring’ images and illustrations using offline programs. Depending on what feature I want to highlight in an image I use different strategies. Word comes with a set of cartoon characters (full, half body, heads, faces) that can be superimposed on a photo for example. I then take a screenshot of this ‘new’ image’ where the person is now represented by a drawn character. By taking a screenshots I avoid metadata that is present in the original images (even if this has been manipulated). I some cases I want to keep the visual information from gestures, body language and placement in a room. Then cartoon elements to mask a person in my image are not a good choice, as the image would loose its relevance as an illustration of for example embodied modes. One solution is to manually draw from a photo, only keeping the relevant information. If I instead chose to use a digital method I now work with images in Inkscape. Sometimes in combination with cartoon elements in Word.

Below, I have used an old family photo (ca 1930), and processed it in Inkscape. The original is top left. The second image has been sharpened, the third is a filter that turns the photo into a line drawing and the fourth (top right) is another variation of line/ink drawing from photo. The bottom row of photos are examples with more or less use of further masking.

Original photo (by unknown) ca 1930, Boy with dachshund.

I have not spent a lot of time on processing these images, but they should offer some ideas of what you can do with just a bit of effort. There is much more you can do if your put your mind to it. It all depends on the image, your purpose and the level of integrity needed. In some situations the solution would be to not use images at all but to describe instead.

However, if you do need to work with images then Inkscape is a free software. It can be a bit tricky to start with. On the other hand there are lots of ‘how to do’ videos and instructions for Inkscape online. Easier to start with are using cartoon people in Word.

Multimodal Seminars and Networks

My research interests are focused on multimodality through a wide lens. I am interested in how people communicate and make meaning, and how people learn. In this post I have listed some of the networks and seminar series I follow. If you are new to multimodality, this list might be useful.

Research Centres and Networks

  • UCL Knowledge Lab;
    • “is a crucible for the study of technology in learning and teaching, and a vantage point from which to design digital media and artefacts that change the way people think and construct knowledge”
  • The Visual and Multimodal Research Forum;
    • “is a hub for researchers across the world who are interested in multimodality. It aims to provide a platform for dialogue and collaboration for advancing visual and multimodal research across disciplines, with a particular emphasis on communication and learning”
  • Research network in medical education (Karolinska Institutet);
  • “is a community for researchers within medical education, to share ideas and learn from each other”

Multimodal Seminar Series

  • The Online Multimodality Talks series; (UCL Institute of Education, University of Leeds, Stockholm University)
    • “a joint initiative for researchers across the world who are interested in multimodality. It aims to provide a platform for dialogue for advancing multimodal research across disciplines. Multimodality draws attention to how meaning is made through the combined use of semiotic resources such as gesture, speech, face expression, body movement and proxemics, (still and moving) image, objects, sound and music, writing, colour, layout, and the built environment”
  • Bremen-Groningen Online Workshops on Multimodality;
    • “a series of workshops organized around various topics and themes of recent multimodality research in order to discuss newest research questions, newly developed methods and frameworks, and/or latest results from empirical work”
  • Research Forum for Interaction and Learning (ReFIL)
    • “focuses on issues concerning analyses of interaction, primarily relevant to the field of Education”

Other Seminar Series and Networks

Multimodality on Social Media

In the Twitter flow you can find several accounts foused on multimodality.

More

Visual Culture

Visual culture: the study of the visual after the cultural turn was written by Margaret Dikovitskaya in 2005, i.e more than a decade ago. The book maps visual culture as an interdisciplinary field. The field visual studies is much more than history of art. In visual culture images are central to representing meaning, but the fields of study which are concerned with the visual are disparate. In the book the author gives an ‘archeology of visual culture’, and discusses theoretical frameworks and their application.

Studying the list of references in the book none of the well known scholars in multimodality are mentioned. However, points of contact could be works by Bourdieu och Foucault, but also (of course) that the book deals with working with the visual just as does Reading Images (Kress & van Leeuwen, 1996/2006) and that the cultural is central in multimodal perspectives as well. Other points in common are the focus on representation and on the social; “the way one sees the world is important, and the visual field is the place where social differences are inscribed” (Dikovitskaya, 2005, p. 57).


Reference

Dikovitskaya, M. (2005). Visual culture: the study of the visual after the cultural turn. Cambridge, Mass.: MIT Press.
KressG., & van LeeuwenT. (1996). Reading images: the grammar of visual designLondon, EnglandRoutledge.
KressG., & van LeeuwenT. (2006). Reading Images: The Grammar ofVisual Design (2nd ed.)London, EnglandRoutledge

 

Visual research literature (and learning)

This is a to-read list, which means this is a working document subject to continuous change.


Adami, E. (2009) ‘We/YouTube’: Exploring sign-making in video-interaction’. Visual Communication, 8 (4): 379-400.

Angelillo, C., Rogoff, B., and Chavajay, P. (2009) Examining shared endeavors by abstracting video coding schemes, in Goldman, R., Pea,R, Barron and Derry Video Research in the learning sciences Routledge: New York: 189-206

Banks, M. (2001) Visual Methods in Social Research. London: Sage. (available at HDB)

Barron, B. and ENGLE, R. (2007) Analyzing Data Derived From Video Records in Derry, S. (ed) Guidelines For Video Research In Education: Recommendations From An Expert Panel, Data Research and Development Center (NORC at the University of Chicago) http://drdc.uchicago.edu/what/video-research.html: 28-37

Bezemer, J. and Mavers, D. (2011) ‘Multimodal transcription as academic practice: a social semiotic perspective’, International Journal of Social Research Methodology, 14: 3, 191 — 206

Birdwhistell, R. (1970) Kinesics and Context: Essays on Body Motion Communication. London: Allen.

Buckingham, D. and Willet, R. (2009) Video Cultures: Media Technology and Everyday Creativity. London: Palgrave Macmillan.

Cope, B., & Kalantzis, M. (2015). The things you do to know: An introduction to the pedagogy of multiliteracies. In A Pedagogy of Multiliteracies (pp. 1-36). Palgrave Macmillan UK.

Derry, S. (ed) (2007) Guidelines For Video Research In Education: Recommendations From An Expert Panel, Data Research and Development Center (NORC at the University of Chicago) http://drdc.uchicago.edu/what/video-research.html.

Erickson, F. (2009) Ways of seeing video: Toward a phenomenology of viewing minimally edited footage, in Goldman, R., Pea,R, Barron and Derry Video Research in the learning sciences Routledge: New York: 145-158.

Flewitt, R. (2006) Using video to investigate preschool classroom interaction: education research assumptions and methodological practices, Visual Communication February vol. 5 no. 1 25-50.

Gilje, O. (2009) Mode, Mediation and Moving Images: An Inquiry of Digital Editing Practices in Media Education, Published PhD. Faculty of Education, University of Oslo, Norway.

Gjedde, L. and Ingemann, B. (2008) Researching Experiences; Exploring Processual and experimental Methods in Cultural Analsysis. Newcastle: Cambridge Scholars Publishing.

Goldman, R. (2009) Video representations and the perspectivity framework, in Goldman, R., Pea,R, Barron and Derry Video Research in the learning sciences Routledge: New York: 3-38.

Goldman, R., Erickson, F., Lemke, J. and Derry, S. (2007) Selection in video, in Derry, S. (ed) (2007) Guidelines For Video Research In Education: Recommendations From An Expert Panel, Data Research and Development Center (NORC at the University of Chicago) http://drdc.uchicago.edu/what/video-research.html: 19 – 27

Goldman, S. and McDermott, R. (2009) Staying the course with video analysis, in Goldman, R., Pea,R, Barron and Derry Video Research in the learning sciences Routledge: New York: 101-114.

Goodwin, C. (2000) Action and embodiment within situated human interaction, Journal of Pragmatics, 32: 1489 – 1522

Goodwin, C. and Goodwin,M. (1996) ‘Seeing as situated activity’. In Y.Engerstrom and D.Middleton (eds.) Cognition and Communication at Work (pp.61-95). Cambridge: Cambridge University Press.

Hall, R. (2007) Strategies For Video Recording: in Derry, S. (ed) (2007) Guidelines For Video Research In Education: Recommendations From An Expert Panel, Data Research and Development Center (NORC at the University of Chicago) http://drdc.uchicago.edu/what/video-research.html: 8-18

Han, C. (2015). How to do critical discourse analysis: A multimodal introduction.

Roth, W.M, (2009) Epistemic mediation: Video data as filters for the objectification of teaching by teachers, in Goldman, R., Pea,R, Barron and Derry (2006) Video Research in the learning sciences Routledge: New York: 367-382.

Hughes, J. (Ed.). (2012). SAGE visual methods. SAGE. (eBook available at SUB)

Jewitt, C., Bezemer, J., Jones, K. and Kress, G. (2009)Changing English? The impact of technology and policy on a school subject in the 21st century. English Teaching: Practice and critique 8(3): 21-40.

Jewitt, C. (2008) Technology, Literacy and Learning: A Multimodal Perspective. London:Routledge.

Jewitt, C (2012) An Introduction to Using Video for Research. NCRM Working Paper. NCRM E-prints.

Jewitt, C. (2011) (Guest editor) Video Based Social Research. Special issue of the International Journal of Social Research Methodology, 14, 3.

Kalantzis, M., & Cope, B. (2015). Learning and new media. The Sage handbook of learning, 373-387.

Kissmann, U. (ed.) (2009) Video Interaction Analysis, Frankfurt: Peter Lang.

Knoblauch, H., Schnettler, B., Raab, J., and Soeffner, H. (eds.) (2006) Video analysis– Methodology and Methods: Qualitative Audiovisual Data Analysis in Sociology. Frankfurt: Peter Lang.

Kress. G., Jewitt, C., Ogborn, J., and Tsatsarelis, C. (2001) Multimodal teaching and learning: The rhetorics of the science classroom. London, UK: Continuum.

Kress. G., Jewitt, Jones, K, Bourne, J., Franks, A and Hardcastle, J. (2005) English in Urban Classrooms. London, UK: Routledge.

Lahlou, S. (2011) How can we capture the subject’s perspective?: an evidence-based approach for the social scientist. Social science information, 50 (4). pp. 607-655.

Lemke, J. (2009) Video epistemology in and outside the box, in Goldman, R., Pea,R, Barron and Derry Video Research in the learning sciences Routledge: New York: 39 – 52.

Lomax, H. and Casey, N. (1998) ‘Recording Social Life: Reflexivity and Video Methodology’. Sociological Research Online, vol. 3, no. 2. http://www.socresonline.org.uk/3/2/1.html (Accessed: 28.01.11)

Mavrikis, M. and Geraniou, E. (2011) ‘Using Qualitative Data Analysis Software to analyse students’ computer-mediated interactions: the case of MiGen and Transana’, International Journal of Social Research Methodology, 14: 3, 245 — 252

Tochon, F. (2009) From video cases to video pedagogy, in Goldman, R., Pea,R, Barron and Derry (2006) Video Research in the learning sciences Routledge: New York: 53-66.

Ruby, J. (ed) (1992) A Crack in the mirror, University of Pennsylvania Press.

Scollon, R. and Wong-Scollon, S. (2010) ‘Multimodality and language: a retrospective and prospective view’ in C.Jewitt (ed.) Routledge Handbook of Multimodal Analysis (pp. 170- 180). London: Routledge.

Schubert (2006) Video analysis as practice and the practice of video analysis in Knoblauch, H., Schnettler, B., Raab, J., and Soeffner, H. (eds.) Video analysis: Methodology and Methods: Qualitative Audiovisual Data Analysis in Sociology. Frankfurt: Peter Lang:115-126.

Julia Snell (2011) ‘Interrogating video data: systematic quantitative analysis versus micro- ethnographic analysis’ International Journal Of Social Research Methodology,
14 (3), pp. 253-258.

Tobin, J. and Hsueh, Y (2009) The poetics and pleasures of video ethnography of education, in Goldman, R., Pea,R, Barron and Derry (2006) Video Research in the learning sciences Routledge: New York: 77-92.

White, S.A. (2003) Participatory Video: Images that Transform and Empower. Dehli, India: Sage.

Zhao, S., Djonov, E., & van Leeuwen, T. (2014). Semiotic technology and practice: a multimodal social semiotic approach to PowerPoint. Text & Talk, 34(3), 349-375.