Ray Ison, Peter Jones, Houda Kyayame, Ryan Murphy, Wendy Ross, Ruth Schmidt, and Katelyn Stenger
This set of hour-long systemic design conversations about reviewing concentrate on reviewing papers for conferences and journals. Bring your curiosity, stories, pragmatism, and the proverbial red pen.
These systemic design conversations are an opportunity to consider a range of topics, such as how reviewing is positioned in an academic career path and how it contributes to research and praxis. The sessions are recorded and aim to represent various perspectives and answer questions as a resource for people new to reviewing while offering fresh insights for established reviewers and editors.
An undercurrent for these sessions is acknowledging that academic norms are culturally and institutionally reinforced—and sometimes ripe for revisiting and questioning. The conversations start with points of view from some experienced and emerging reviewers on reviewing and then open to participants in an ask-me-anything format. For academics and practitioners, reviewing engagements are a way to offer authors professional experience and share domain knowledge through single- or double-anonymous review notes. The reviewing payback is the continuous improvement of writing and editing skills and expanded knowledge through exposure to new research. Topics related to reviewing extend to the (re)definition of scholarly practices, the role of metrics and researcher assessments, and barriers that limit the participation of early career researchers and practitioners. Although these issues will likely surface in the sessions, the primary goal is to throw the gate open and discuss what it means to review, who should review, and consider best practices.
For anyone new to reviewing or reviewing-curious, you are invited to think of these sessions as part of a developmental process. When you are getting started, you have the benefit of rules and procedural guidelines and can steadily advance these by applying them to a variety of cases. Reviewing reviewing this way can make it less daunting and help you connect with opportunities and participate in publishing communities.
Reviewing Reviewing #1: Considering experience and expertise, norms of practice, and exposure to good practice
A conversation with Peter Jones, Ruth Schmidt, and Katelyn Stenger
Things to think about
Reviewing Reviewing #2: Cybersystemic perspectives and identifiable moments of serendipity
A conversation with Ray Ison, Houda Khayame, and Wendy Ross
Things to think about
VIDEO & NOTES
Session Summary
This group challenged some of the norms of practice:
They discussed characteristics of high-quality and poor-quality peer reviews in academic publishing*
Factors that make someone a good reviewer, such as expertise, care, and providing constructive feedback were considered
Alternative approaches to peer review were explored, such as open reviews
The evolution of peer review from early scientific letters to the current journal-based system provides an overview
They expressed concerns about the potential impact of AI on academic publishing, such as overwhelming the system with low-quality articles
Discourse, perspectives, and relevant citations in academic writing and peer review are viewed as important to cultivate high-quality scholarship, as are setting up scholarly communities and infrastructures
Personal experiences with editing and guest-editing a special issue illuminated the opportunities and satisfaction these activities can provide
Insights from organizations seeking to reform research assessment, like DORA were shared
*Some characteristics given for high-quality reviews include:
- Being concise and constructive
- Informing gaps in the manuscript from an editorial or scholarly development perspective
- Helping authors through constructive feedback rather than criticism
- Providing clear and useful feedback to improve the work
Characteristics given for poor-quality reviews include:
- Being disrespectful, arrogant, or diminishing towards the authors
- Not giving useful insights or feedback for authors to develop their work
- Being overly concise without substantive critique
- Having a strong opinionated position that comes across as misleading rather than constructive
- Lacking care or consideration of the paper’s full context
Session Summary
This group discussed reviewing as a systemic co-inquiry, seeing it as a learning system where everyone learns through the process
They talked about moving towards more reflexivity in reviewing and considering different epistemologies and perspectives
Anonymized reviewing was discussed, with some panellists preferring to not be anonymized in order to take responsibility for their reviews
The tensions between different disciplines and ways of knowing in reviewing papers were explored
Ideas around developing reviewing philosophies and setting up the reviewing process to be more open and collaborative were proposed.*
*Some approaches to reviewing that were discussed:
- Being thorough, bringing evidence to support claims, and focusing on improving the work rather than personal attacks
- Considering the purpose of reviewing and whether it is to create knowledge, help researchers communicate, or have a learning conversation
- Accommodating different epistemologies and perspectives rather than just one’s own viewpoint
- Seeing reviewing as a systemic co-inquiry where everyone learns through engaging with multiple perspectives
Ray Ison: Reviewing needs to question the systems, rules and norms it is scaffolded by to avoid homogenization of knowledge. Reviewing can be seen as a practice that can be carried out reflexively and involves collaboration with authors.
Wendy Ross: Had a positive review experience early in her career that led to a long-term collaboration, showing the benefits of constructive feedback. Feels that open peer review led her to take more responsibility for her own submitted work.
Houda Khayame: Suggests framing reviewing as a systemic co-inquiry where multiple perspectives are considered. She wonders how early career researchers like herself are perceived and valued as reviewers, given hierarchies in academia.
the discussants
Ray Ison
Ray Ison was appointed Professor of Systems at the Open University in 1994, now Professor Emeritus. He is a member of the Systems Thinking in Practice (STiP) program involved in researching and teaching, which is developed and presented within the Open University’s MSc and Systems Thinking Practitioner Apprenticeship awards. He has also led and contributed to a range of major research, scholarship and teaching programs and projects as part of the Applied Systems Thinking in Practice (ASTiP) Group. His research and scholarship span the biophysical and social and are primarily interdisciplinary, international and collaborative. Ray is the current President (since 2019) of the International Federation for Systems Research. He has also served as President of the International Society for the Systems Sciences and as a Trustee of the American Society of Cybernetics. In 2022, he was given a Lifetime Achievement Award by the Systems Society of India for contributions to the transformation of society with a systems approach. https://en.wikipedia.org/wiki/Ray_Ison
Peter Jones
Peter Jones holds the Distinguished Professor in Systemic Design position at the School of Architecture, Art and Design, Tecnológico de Monterrey and Associate Professor, Faculty of Design, OCAD University (MDes). His research agenda is largely focused on the theory and application of systemic design in highly complex domains such as healthcare and public sector management. He is editor-in-chief of Contexts: The Systemic Design Journal and proceedings editorial advisor to RSDsymposium.org. Peter is also a reviewer for She Ji, Systems, Form Akademisk, Organization and Environment, Information, and other journals. He has organised special issue collections curated from the RSD Symposium series. He is a section editor for chapters submitted to the Systemic Design section of the Springer Handbook of Systems Sciences. Peter is a leading scholar in dialogic design and collaborative foresight for addressing complex future challenges and is widely published in the field. He has written four books about design and innovation, including Design Journeys through Complex Systems (BIS Publishers, 2022) and Design for Care: Innovating Healthcare Experience (Rosenfeld Media, 2013). https://tec.mx/en/our-faculty/eaad/peter-jones | designdialogues.com
Houda Khayame
Houda Khayame is a doctoral researcher at The Open University and an international development consultant with over 23 years of project leadership in the Middle East and North and West Africa, focusing on women and child health, gender equity, food systems and biodiversity. Her practice is transdisciplinary, using a repertoire of systems approaches and tools in the areas of summative and formative research, institutional reform, and social and behaviour change. A 2020 publication on soft systems approaches to improve public policy illustrates her practice of systemic development. Houda has carried her development interests into her PhD research, which focuses on institutional transformation towards systemic governance to improve complex social-ecological issues in the Anthropocene. As an Associate Lecturer with The Open University, she supports the learning journeys of post-graduate students to develop their systems thinking in practice capabilities. Houda is an editorial board member for Social Marketing Quarterly and has contributed peer reviews for The Journal of Social Marketing, Systems Research & Behavioural Science, and Environmental Science and Policy. LinkedIn profile
Ryan Murphy
Ryan Murphy is a PhD (candidate) in Management (Information Systems) at Memorial University. For over a decade, he has worked in Canadian K-12 and post-secondary education to teach and enhance programming for leadership, innovation, and changemaking skills. In his emerging research practice, Ryan studies changemakers and their tools. He explores how technology can support systemic change through the modelling and analysis of complex social systems and through the design of systemic strategies. Ryan designs and tests tools for augmenting intelligence. He also examines how leadership and innovation skills are practised and learned. He is an SDA board member serving 2022–2023 (Co-Secretary/Treasurer) and 2023–2024 (Secretary/Co-Treasurer). https://systemic-design.org/board-of-directors/ | fulcra.design
Wendy Ross
Wendy Ross is a Senior Lecturer in Psychology at London Metropolitan University. Her main areas of research are serendipity, possibility and creativity from an embodied and extended cognition perspective. Her work is firmly interdisciplinary, and she has worked with anthropologists, archaeologists, and experimental psychologists, drawing on a range of methods from eye-tracking to ethnography. She was a Senior Editor for the Palgrave Encyclopedia of the Possible (2023), overseeing some 30 entries as well as working on two smaller edited collections, the Art of Serendipity (2023) and Serendipity Science (2024). She currently serves on the editorial board of The Journal of Creative Behaviour, is Associate Editor of Creativity Research Journal, and Associate Managing Editor of Possibility Studies and Society. https://www.londonmet.ac.uk/profiles/staff/wendy-ross/ | www.possibilitystudies.net
Ruth Schmidt
Ruth Schmidt is an Associate Professor at the Institute of Design focused on behavioural design. Her teaching and research integrate humanity-centred design, systems design, and behavioural science to address behavioural challenges more systematically and equitably. Since 2009, Ruth has developed and taught courses that combine academic theory, practical application of existing methods, and the development of new methods and frameworks to tackle humanity-centered design challenges. In 2021–22, she contributed her expertise in organisational behavioural design to the first set of courses delivered through ID Academy, aimed at introducing mid-career and executive-level audiences to new and emerging areas of design outside of traditional degree programs. Ruth also has an established partnership with DORA (Declaration on Research Assessment), an international advocacy and open research initiative focused on promoting and supporting new, equitable practices for defining and assessing academic research quality and scholarly impact. Her work has appeared in academic journals and other outlets such as Behavioural Public Policy, The Journal of Design Strategies, Strategic Design Research Journal, Policy Design and Practice, Nature Sustainability, Rotman Management Magazine, and behavioralscientist.org.
Katelyn Stenger
Katelyn is the Guest Editor of The Nexus of Policy, Social and Behavioral Science, and a Just Energy Transition, the Energy and Climate Change special issue on just energy transitions and their interactions with social systems. [call for papers closes September 2024] Katelyn has extensive experience designing, engineering, and researching the built and social environment and approaches solving for complexity through collaboration and co-designing with passionate people who bring different ways of knowing to the table—from lived experience or learned expertise. Katelyn researches behavioural and systemic change for a clean energy future at the National Renewable Energy Laboratory by partnering with communities, utilities, and policymakers. Katelyn received a PhD from the University of Virginia and a Bachelor of Science from Rose-Hulman Institute of Technology. Her work has been published in the National Academies of Sciences, Engineering, and Medicine, Behavioural Public Policy, Elementa: Science of the Anthropocene, and behavioralscientist.org. Google Scholar Profile | LinkedIn