Scientific Performance Assessments Through a Gender Lens

a Case Study on Evaluation and Selection Practices in Academia

  • Matthias Wullum Nielsen Aarhus University

Abstract

The focus on excellence and quality assurance in the academy has spawned a significant increase in the use of bibliometric measures in performance assessments of individual researchers. This article investigates the organizational consequences of this development through a gender lens. Based on a qualitative case study of evaluation and selection practices at a Danish university, a number of potential gender biases related to the use of bibliometric performance measures are identified. By taking as default the research preferences, approaches and career paths of a succesful group of predominantly male scholars, evaluators using bibliometrics risk disadvantaging candidates diverging from the norm with implications for gender stratification. Despite these potential biases, bibliometric measures come to function as technologies supporting a managerial narrative of the gender-blind organization. They adhere to the prevailing ethos of the academic meritocracy by standardizing the criteria for organizational advancement and ensuring transparency and accountability in the selection process. While bibliometric tools in this sense may lead to the recruitment of scientists with a strong CV and track record, they may at the same time prevent many talented researchers diverging from the norm from being recognized and succeed as academics.

Downloads

Download data is not yet available.

Author Biography

Matthias Wullum Nielsen, Aarhus University

Danish Centre for Studies in Research and Research Policy, Department of Political Science and Government

References

Arhus University (2013) Criteria for evaluating candidates for positions as full professor, associate professor, senior scientist, or senior advisor. Retrieved from http://scitech.medarbejdere.au.dk/fileadmin/site_files/scitech.medarbejdere.au.dk/files/Evaluation_Criteria.pdf

Abramo, G., C. A. D'Angelo, and A. Caprasecca (2009) 'The Contribution of Star Scientists to Overall Sex Differences in Research Productivity', Scientometrics, 84 (3): 821–33.
https://doi.org/10.1007/s11192-010-0200-1

Acker, J. (1990) 'Hierarchies, Jobs, Bodies: A Theory of Gendered Organizations', Gender & Society 4 (2): 139–58.
https://doi.org/10.1177/089124390004002002

Acker, J. (1992) Gendering organizational theory. In A. J. Mills & P. Tancred (Eds.), Gendering Organizational Analysis (pp. 248-260). Newbury Park, London and New Delhi: Sage.

Adcock, R. and D. Collier (2001) 'Measurement validity: A shared standard for qualitative and quantitative research', The American Political Science Review 95(3): 529-546.
https://doi.org/10.1017/S0003055401003100

Addis, E. and M. Brouns (2004) Synthesis Report on the Workshop in Gender and Excellence in the making (Brussels: European Commission).

Åkerlind, G. S. (2005) Postdoctoral researchers: Roles, functions and career prospects. Higher Education Research & Development, 24(1), 21-40.
https://doi.org/10.1080/0729436052000318550

Aksnes, D. W., K. Rorstad, F. Piro, and G. Sivertsen (2011) 'Are female researchers less cited? A large-scale study of Norwegian scientists', Journal of the American Society for Information Science and Technology 62(4): 628-636.
https://doi.org/10.1002/asi.21486

Alvesson, M., and Billing, Y. D. (2009) Understanding gender and organizations. Sage.
https://doi.org/10.4135/9781446280133

Archambault, E. and V. Gagné (2004) 'The use of bibliometrics in the social sciences and humanities'. Prepared for the Social Sciences and Humanities Research Council of Canada.

Bailyn, L. (2004) 'Time in careers: Careers in time', Human Relations 57(12): 1507-1521.
https://doi.org/10.1177/0018726704049987

Bailyn, L. (2011) 'Redesigning work for gender equity and work-personal life integration', Community, Work & family 14(1): 97-112.
https://doi.org/10.1080/13668803.2010.532660

Bennett, A., Barth, A., and K. R. Rutherford (2003) 'Do we preach what we practice? A survey of methods in political science journals and curricula', Political Science and Politics, 36(03): 373-378.

Bordons, M., F. Morillo, M. T. Fernandez and I. Gomez, (2003) 'One step further in the production of bibliometric indicators at the micro level: Differences by gender and professional category of scientists', Scientometrics 57(2): 159-173.
https://doi.org/10.1023/A:1024181400646

Bornmann, L., W. Marx, A. Yuri Gasparyan and G. D. Kitas (2012) 'Diversity, value and limitations of the journal impact factor and alternative metrics', Rheumatology International, 32(7): 1861-1867.
https://doi.org/10.1007/s00296-011-2276-1

Brooks, C., Fenton, E. M., and T. J. Walker (2014) 'Gender and the evaluation of research', Research Policy, 43(6): 990-1001.
https://doi.org/10.1016/j.respol.2013.12.005

Bozeman, B., and E. Corley. (2004) 'Scientists' Collaboration Strategies: Implications for Scientific and Technical Human Capital', Research Policy, 33: 599–616.
https://doi.org/10.1016/j.respol.2004.01.008

Christenson, J. A., & Sigelman, L. (1985). Accrediting knowledge: Journal stature and citation impact in social science. Social Science Quarterly, 66(4), 964.

Cole, J. R. and H. Zuckerman (1991) 'Marriage, motherhood, and research performance in science', in H. Zuckerman, J. R. Cole and J. T. Bruer (eds), The Outer Circle: Women in the Scientific Community (New York: W.W. Norton): 157-170.

Davenport, E. And H. Snyder (1995) 'Who cites women? Whom do women cite? An exploration of gender and scholarly citation in sociology', Journal of Documentation, 51: 404–410.
https://doi.org/10.1108/eb026958

Dolado, J. J., F. Felgueroso and M. Almunia (2005) 'Do men and women-economists choose the same research fields? Evidence from top-50 departments', IZA Discussion Paper No. 1859. Available at SSRN: http://ssrn.com/abstract=866886

Donovan, C. (2007) 'The hidden perils of citation counting for Australasian political science', Australian Journal of Political Science, 42(4): 665-678.
https://doi.org/10.1080/10361140701595825

Espeland, W. and M. Sauder (2007) 'Rankings and reactivity: How public measures recreate social worlds', American Journal of Sociology, 113(1): 1-40.
https://doi.org/10.1086/517897

Espeland, W. and M. L. Stevens (1998) 'Commensuration as a social process', Annual Review of Sociology, 24: 313-343.
https://doi.org/10.1146/annurev.soc.24.1.313

Espeland, W. and M. L. Stevens (2008) 'The sociology of quantification', European Journal of Sociology, 49(3): 401-436.
https://doi.org/10.1017/S0003975609000150

European Commission (2000) Presidency conclusion: Lisbon European Council, March 23-24 (Brussels).

European Commission (2005) Communication to the Spring European Council. Working together for growth and jobs: A new start for the Lisbon Strategy. COM (2005) 24 (Brussels).

European Commission (2010) Europe 2020: A European strategy for smart, sustainable and inclusive growth. Communication from the commission (Brussels).

Evans, H. K. and E. P. Bucy (2010) 'The representation of women in publication: An analysis of political communication and the international journal of press/politics'. PS: Political Science and Politics, 43: 295-301.
https://doi.org/10.1017/s1049096510000168

Feller, I. (2003) 'Measurement of scientific performance and gender bias', in E. Addis and Margo Brouns (eds), Synthesis Report on the Workshop in Gender and Excellence in the making (Brussels: European Commission: 35-41).

Felt, U. and T. Stöckelová (2009) 'Modes of ordering and boundaries that matter in academic knowledge production', in U. Felt (ed.), Knowing and Living in Academic Research (Prague: Institute of Sociology of the Academy of Sciences of the Czech Republic): 41-124.

Fisher, R. L. (2005) The research productivity of scientists: how gender, organization culture, and the problem choice process influence the productivity of scientists. University Press of America.

Fleck C. (2013) 'The impact factor fetishism', European Journal of Sociology, 54(2): 327-356.
https://doi.org/10.1017/S0003975613000167

Fox, M. F., and C. A. Faver (1985) 'Men, women, and publication productivity: Patterns among social work academics', Sociological Quarterly, 26: 537-549.
https://doi.org/10.1111/j.1533-8525.1985.tb00243.x

Fox M. F. (2005) 'Gender, family characteristics, and publication productivity among scientists', Social Studies of Science, 35(1): 131-150.
https://doi.org/10.1177/0306312705046630

Fletcher, J. K. (2001) Disappearing acts: Gender, power, and relational practice at work. MA, Boston: MIT Press.

Fraser, N. (1995) False antithesis. In S. Benhabib, J. Butler, D. Cornell, & N. Fraser (Eds.), Feminist contentions (pp. 59-74). New York, NY: Routledge.

Fry, J. (2004) 'The cultural shaping of ICTs within academic fields: Corpus-based linguistics as a case study', Literary and Linguistic Computing 19(3): 303-319.
https://doi.org/10.1093/llc/19.3.303

Garcia-Pérez, M. A. (2009) 'A multidimensional extension to Hirsch's H-index', Scientometrics, 81(3): 779-785.
https://doi.org/10.1007/s11192-009-2290-1

Guena, A. and B. R. Martin (2003) 'University research evaluation and funding: An international comparison', Minerva 41(4): 277-304.
https://doi.org/10.1023/B:MINE.0000005155.70870.bd

Gunnarsson, L. (2011). 'A defence of the category "women"'. Feminist theory, 12(1): 23-37.
https://doi.org/10.1177/1464700110390604

Gläser, J. and G. Laudel (2007) 'The social construction of bibliometric evaluations', in R. Whitley and J. Gläser (eds), The Changing Governance of The Sciences: The Advent of Research Evaluation Systems. (Doodrecht: Springer): 101-123.
https://doi.org/10.1007/978-1-4020-6746-4_5

Haraway, D. J. (1989). Primate visions: Gender, race, and nature in the world of modern science. Hove: Psychology Press.

Haraway, D. J. (1997). Modest− Witness@ Second− Millennium. FemaleMan− Meets− OncoMouse: Feminism and Technoscience. Hove: Psychology Press.

Hicks, D., Wouters, P., Waltman, L., & de Rijcke, S., and I. Rafols (2015) 'Bibliometrics: The Leiden Manifesto for research metrics', Nature, 520: 429-31.
https://doi.org/10.1038/520429a

Hirsch, J. E. (2005) 'An index to quantify an individual's scientific research output', Proceedings of the National Academy of Sciences, 102: 16569-16572.
https://doi.org/10.1073/pnas.0507655102

Hunter, L., and E. Leahey (2010) 'Parenting and research productivity: New evidence and methods', Social Studies of Science, 40(3): 433–451
https://doi.org/10.1177/0306312709358472

Jessop, B. (2008) 'A cultural political economy of competitiveness and its implications for higher education', in B. Jessop et al. (eds), Education and the Knowledge-Based Econonomy in Europe (Rotterdam: Sense Publishers): 11-39
Jessop, B. (2008) 'A cultural political economy of competitiveness and its implications for higher education', in B. Jessop et al. (eds), Education and the Knowledge-Based Econonomy in Europe (Rotterdam: Sense Publishers): 11-39

Kelly, C. D. and M. D. Jennions (2006) 'The h index and career assessment by numbers', TRENDS in Ecology and Evolution, 21: 167–170.
https://doi.org/10.1016/j.tree.2006.01.005

Kalpazidou Schmidt, E. (2009) 'Nordic higher education systems in a comparative perspective: Recent reforms and impacts', The Journal of Finance and Management in Colleges and Universities, 6: 269–298.

Knights, D. and W. Richards (2003) 'Sex discrimination in UK academia', Gender, Work & Organization, 10(2): 213-238.
https://doi.org/10.1111/1468-0432.t01-1-00012

Kyvik, S. (1990) 'Motherhood and scientific productivity', Social Studies of Science, 20(1): 149-160.
https://doi.org/10.1177/030631290020001005

Kyvik, S. and M. Teigen (1996) 'Child care, research collaboration, and gender differences in scientific productivity', Science, Technology, & Human Values 21(1): 54-71.
https://doi.org/10.1177/016224399602100103

Lamont, M. (2009) How Professors Think: Inside the Curious World of Academic Judgment (Cambridge, MA: Harvard University Press).
https://doi.org/10.4159/9780674054158

Lariviere, V., Ni, C. Q., Gingras, Y., Cronin, B., and C. R. Sugimoto (2013) 'Global gender disparities in science', Nature, 504(7479): 211–213.
https://doi.org/10.1038/504211a

Latour, B. and S. Woolgar (1986) Laboratory Life: The Construction of Scientific Facts (Princeton, NJ: Princeton University Press).

Levin, M. (1988) 'Caring new world: Feminism and science', The American Scholar, 57: 100-106.

Light, R. (2013). 'Gender inequality and the structure of occupational Identity: The case of elite sociological publication'. Networks, Work, and Inequality, 24, 239-268.
https://doi.org/10.1108/S0277-2833(2013)0000024012

Long, J. S. (1992) "Measures of sex-differences in scientific productivity", Social Forces, 71(1): 159–178.
https://doi.org/10.2307/2579971

Luke, C. (1997) 'Quality assurance and women in higher education', Higher education, 33(4), 433-451.
https://doi.org/10.1023/A:1002905428032

Macdonald, S., & J. Kam (2007) 'Ring a Ring o'Roses: Quality Journals and Gamesmanship in Management Studies', Journal of Management Studies,44(4): 640-655.
https://doi.org/10.1111/j.1467-6486.2007.00704.x

Mairesse, J. and M. Pezzoni (2013). 'Does gender affect scientific productivity? A critical review of the empirical evidence and a panel data econometric analysis for French physicists'. Submitted to the AFSE Meeting, 26 June 2013, Aix en Provence.

Mallard, G., M. Lamont and J. Guetzkow (2009) 'Fairness as appropriateness: Managing epistemological differences in peer review', Science, Technology and Human Values, 34(5): 573-606.
https://doi.org/10.1177/0162243908329381

Maliniak, D., R. M. Powers and F. Barbara (2013) 'The gender citation gap in international relations'. International Organization, 67(4):889-922.
https://doi.org/10.1017/S0020818313000209

Mason, M. A., and M. Goulden (2004) 'Do babies matter (Part II)? Closing the baby gap', Academe, November–December, http://www.uri.edu/advance/files/pdf/Do_Babies_Matter(II).pdf

March, J. G., and H. A. Simon (1958) Organizations (New York: Wiley).

Mauleón, E., and M. Bordons (2006) 'Productivity, impact and publication habits by gender in the area of materials science', Scientometrics, 66(1): 199–218.
https://doi.org/10.1007/s11192-006-0014-3

McElhinny, B., Hols, M., Holtzkener, J., Unger, S., and C. Hicks (2003) 'Gender, publication and citation in sociolinguistics and linguistic anthropology: the construction of a scholarly canon', Language in Society, 32: 299–328.
https://doi.org/10.1017/S0047404503323012

Moed, H. F., M. Luwel and A. J. Nederhof (2002) 'Towards research performance in the humanities', Library Trends, 50(3): 498–520.

Morley, L. (2003) Power and quality in higher education (Berkshire: Open University Press).

Morley, L. (2005) 'The micropolitics of quality', Critical Quarterly, 47(1‐2): 83-95.
https://doi.org/10.1111/j.0011-1562.2005.00616.x

Müller, R. (2014) Postdoctoral life scientists and supervision work in the contemporary university: A case study of changes in the cultural norms of science. Minerva, 52(3), 329-349.
https://doi.org/10.1007/s11024-014-9257-y

Nielsen, M. W. (2015a) Gender inequality and research performance: moving beyond individual-meritocratic explanations of academic advancement. Studies in Higher Education, 1-17 (online version). http://dx.doi.org/10.1080/03075079.2015.1007945
https://doi.org/10.1080/03075079.2015.1007945

Nielsen, M. W. (2015b) Gender consequences of a national performance-based funding model: new pieces in an old puzzle. Studies in Higher Education, 1-23 (online version). http://dx.doi.org/10.1080/03075079.2015.1075197
https://doi.org/10.1080/03075079.2015.1075197

Nielsen, M. W. (2016). Limits to meritocracy? Gender in academic recruitment and selection processes. Science & Public Policy, 43 (3): 386-399.
https://doi.org/10.1093/scipol/scv052

Plowman, D. A. and Smith, A. D. (2011) 'The gendering of organizational research methods: Evidence of gender patterns in qualitative research', Qualitative Research in Organizations and Management: An International Journal, 6(1): 64-82.
https://doi.org/10.1108/17465641111129399

Prozesky, H (2008) 'A career-history analysis of gender differences in publication productivity among South African academics', Science Studies, 21(2): 47–67.

Prpic, K., (2002) 'Gender and productivity differentials in science', Scientometrics, 55(1): 27–58.
https://doi.org/10.1023/A:1016046819457

Rafols, I., L. Leydesdorff , A. O'Hare, P. Nightingale and A. Stirling (2011) 'How journal rankings can suppress interdisciplinary research: A comparison between innovation studies and business and management', DRUID Working Paper No. 11-05.

Ritzberger, K. (2008). A ranking of journals in economics and related fields.German Economic Review, 9(4), 402-430.
https://doi.org/10.1111/j.1468-0475.2008.00447.x

Salmi, J. (2009) The Challenge of Establishing World Class Universities (Washington DC: The World Bank).
https://doi.org/10.1596/978-0-8213-7865-6

Schiebinger, L. (1999) Has Feminism Changed Science? (Cambridge, MA: Harvard University Press).

Schneider, J. W. (2009) 'An outline of the bibliometric indicator used for performance-based funding of research institutions in Norway', European Political Science, 8: 364-378.
https://doi.org/10.1057/eps.2009.19

Seglen, P. (1997) 'Why the impact factor of journals should not be used for evaluating research', British Medical Journey, 314: 497-502.
https://doi.org/10.1136/bmj.314.7079.497

Seierstad, C. and G. Healy (2012) 'Women's equality in the Scandinavian academy: A distant dream?' Work Employment & Society, 26(2): 296-313.
https://doi.org/10.1177/0950017011432918

Simmel, G. (1903) 'Die Grosstädte und das Geistesleben, translated to 'The metropolis and modern life', by Kurt Wolff, in D. Weinstein (ed), 1950. The Sociology of Georg Simmel (New York: Free Press): 409-424.

Symonds, M. R. E., N. J. Gemell, T. L. Braisher, K. L. Gorringe, M. A. Elgar (2006) 'Gender differences in publication output: Towards an unbiased metric of research performance' in T. Tregenza (ed) PLoS ONE 1 (1): doi:10.1371/journal.pone.0000127
https://doi.org/10.1371/journal.pone.0000127

Sonnert, G. and G. Holton (1996) 'Career patterns of women and men in the sciences', American Scientist, 84(1): 63-71.

Svensson, G. (2006). 'The paradoxnoia of top journal (s) in marketing. European Journal of Marketing', 40(11/12): 1153-1168.
https://doi.org/10.1108/03090560610702740

Thomas, R. and A. Davies (2003) 'Gender and New Public Management: Reconstituting academic subjectivities', Gender, Work and Organization, 9(4): 372-397.
https://doi.org/10.1111/1468-0432.00165

Traweek, S. (2009) Beamtimes and lifetimes: The world of high energy physicists. MA, Boston: Harvard University Press.

Valian, V. (1999) 'The cognitive bases of gender bias', Brooklyn Law Review, 65:1037-1061.

Van Den Brink, M., B. Fruytier and M. Thunnissen (2013) 'Talent management in academia: Performance systems and HRM policies', Human Resource Management Journal, 23(2): 180-195.
https://doi.org/10.1111/j.1748-8583.2012.00196.x

Van Raan, A. F. J. (1998) 'The influence of international collaboration on the impact of research results: some simple mathematical considerations concerning the role of self-citations', Scientometrics, 42(3): 423-428.
https://doi.org/10.1007/BF02458380

Verner, M. (2008) 'Gender differences in rank within the academic profession: The case of Denmark', Paper presented at the IZA Workshop on Research in Economics: Rewards, Evaluation and Funding, Bonn, Germany.

Ward, K.B., Gast, J., and L. Grant (1992) 'Visibility and dissemination of women's and men's sociological scholarship', Social Problems, 39: 291–298.
https://doi.org/10.2307/3096963

Weingart, P. (2005) 'Impact of bibliometrics upon the science system: Inadvertent consequences', Scientometrics, 62(1): 117-131.
https://doi.org/10.1007/s11192-005-0007-7

Whitley, R. (1984) The Intellectual and Social Organization of the Sciences (Oxford: Oxford University Press).

Whitley, R. (2007) 'Changing governance of the public sciences: The consequences of establishing research evaluation systems for knowledge production in different countries and scientific fields' in R. Whitley, J. Gläser (eds), The Changing Governance of the Sciences (Doodrecht: Springer).
https://doi.org/10.1007/978-1-4020-6746-4_1

West, C., and Zimmerman, D. H. (1987) Doing gender. Gender & society, 1(2), 125-151.
https://doi.org/10.1177/0891243287001002002

Willmott, H. (2011) 'Journal list fetishism and the perversion of scholarship: reactivity and the ABS list', Organization, 18(4): 429-442.
https://doi.org/10.1177/1350508411403532

Xie, Y., & Shauman, K. A. (2003) Women in science: Career processes and outcomes. Cambridge, MA: Harvard University Press.
Published
2018-02-15
Section
Research Papers