Scientific Performance Assessments Through a Gender Lens

a Case Study on Evaluation and Selection Practices in Academia

  • Matthias Wullum Nielsen Aarhus University


The focus on excellence and quality assurance in the academy has spawned a significant increase in the use of bibliometric measures in performance assessments of individual researchers. This article investigates the organizational consequences of this development through a gender lens. Based on a qualitative case study of evaluation and selection practices at a Danish university, a number of potential gender biases related to the use of bibliometric performance measures are identified. By taking as default the research preferences, approaches and career paths of a succesful group of predominantly male scholars, evaluators using bibliometrics risk disadvantaging candidates diverging from the norm with implications for gender stratification. Despite these potential biases, bibliometric measures come to function as technologies supporting a managerial narrative of the gender-blind organization. They adhere to the prevailing ethos of the academic meritocracy by standardizing the criteria for organizational advancement and ensuring transparency and accountability in the selection process. While bibliometric tools in this sense may lead to the recruitment of scientists with a strong CV and track record, they may at the same time prevent many talented researchers diverging from the norm from being recognized and succeed as academics.


Download data is not yet available.

Author Biography

Matthias Wullum Nielsen, Aarhus University

Danish Centre for Studies in Research and Research Policy, Department of Political Science and Government


Arhus University (2013) Criteria for evaluating candidates for positions as full professor, associate professor, senior scientist, or senior advisor. Retrieved from

Abramo, G., C. A. D'Angelo, and A. Caprasecca (2009) 'The Contribution of Star Scientists to Overall Sex Differences in Research Productivity', Scientometrics, 84 (3): 821–33.

Acker, J. (1990) 'Hierarchies, Jobs, Bodies: A Theory of Gendered Organizations', Gender & Society 4 (2): 139–58.

Acker, J. (1992) Gendering organizational theory. In A. J. Mills & P. Tancred (Eds.), Gendering Organizational Analysis (pp. 248-260). Newbury Park, London and New Delhi: Sage.

Adcock, R. and D. Collier (2001) 'Measurement validity: A shared standard for qualitative and quantitative research', The American Political Science Review 95(3): 529-546.

Addis, E. and M. Brouns (2004) Synthesis Report on the Workshop in Gender and Excellence in the making (Brussels: European Commission).

Åkerlind, G. S. (2005) Postdoctoral researchers: Roles, functions and career prospects. Higher Education Research & Development, 24(1), 21-40.

Aksnes, D. W., K. Rorstad, F. Piro, and G. Sivertsen (2011) 'Are female researchers less cited? A large-scale study of Norwegian scientists', Journal of the American Society for Information Science and Technology 62(4): 628-636.

Alvesson, M., and Billing, Y. D. (2009) Understanding gender and organizations. Sage.

Archambault, E. and V. Gagné (2004) 'The use of bibliometrics in the social sciences and humanities'. Prepared for the Social Sciences and Humanities Research Council of Canada.

Bailyn, L. (2004) 'Time in careers: Careers in time', Human Relations 57(12): 1507-1521.

Bailyn, L. (2011) 'Redesigning work for gender equity and work-personal life integration', Community, Work & family 14(1): 97-112.

Bennett, A., Barth, A., and K. R. Rutherford (2003) 'Do we preach what we practice? A survey of methods in political science journals and curricula', Political Science and Politics, 36(03): 373-378.

Bordons, M., F. Morillo, M. T. Fernandez and I. Gomez, (2003) 'One step further in the production of bibliometric indicators at the micro level: Differences by gender and professional category of scientists', Scientometrics 57(2): 159-173.

Bornmann, L., W. Marx, A. Yuri Gasparyan and G. D. Kitas (2012) 'Diversity, value and limitations of the journal impact factor and alternative metrics', Rheumatology International, 32(7): 1861-1867.

Brooks, C., Fenton, E. M., and T. J. Walker (2014) 'Gender and the evaluation of research', Research Policy, 43(6): 990-1001.

Bozeman, B., and E. Corley. (2004) 'Scientists' Collaboration Strategies: Implications for Scientific and Technical Human Capital', Research Policy, 33: 599–616.

Christenson, J. A., & Sigelman, L. (1985). Accrediting knowledge: Journal stature and citation impact in social science. Social Science Quarterly, 66(4), 964.

Cole, J. R. and H. Zuckerman (1991) 'Marriage, motherhood, and research performance in science', in H. Zuckerman, J. R. Cole and J. T. Bruer (eds), The Outer Circle: Women in the Scientific Community (New York: W.W. Norton): 157-170.

Davenport, E. And H. Snyder (1995) 'Who cites women? Whom do women cite? An exploration of gender and scholarly citation in sociology', Journal of Documentation, 51: 404–410.

Dolado, J. J., F. Felgueroso and M. Almunia (2005) 'Do men and women-economists choose the same research fields? Evidence from top-50 departments', IZA Discussion Paper No. 1859. Available at SSRN:

Donovan, C. (2007) 'The hidden perils of citation counting for Australasian political science', Australian Journal of Political Science, 42(4): 665-678.

Espeland, W. and M. Sauder (2007) 'Rankings and reactivity: How public measures recreate social worlds', American Journal of Sociology, 113(1): 1-40.

Espeland, W. and M. L. Stevens (1998) 'Commensuration as a social process', Annual Review of Sociology, 24: 313-343.

Espeland, W. and M. L. Stevens (2008) 'The sociology of quantification', European Journal of Sociology, 49(3): 401-436.

European Commission (2000) Presidency conclusion: Lisbon European Council, March 23-24 (Brussels).

European Commission (2005) Communication to the Spring European Council. Working together for growth and jobs: A new start for the Lisbon Strategy. COM (2005) 24 (Brussels).

European Commission (2010) Europe 2020: A European strategy for smart, sustainable and inclusive growth. Communication from the commission (Brussels).

Evans, H. K. and E. P. Bucy (2010) 'The representation of women in publication: An analysis of political communication and the international journal of press/politics'. PS: Political Science and Politics, 43: 295-301.

Feller, I. (2003) 'Measurement of scientific performance and gender bias', in E. Addis and Margo Brouns (eds), Synthesis Report on the Workshop in Gender and Excellence in the making (Brussels: European Commission: 35-41).

Felt, U. and T. Stöckelová (2009) 'Modes of ordering and boundaries that matter in academic knowledge production', in U. Felt (ed.), Knowing and Living in Academic Research (Prague: Institute of Sociology of the Academy of Sciences of the Czech Republic): 41-124.

Fisher, R. L. (2005) The research productivity of scientists: how gender, organization culture, and the problem choice process influence the productivity of scientists. University Press of America.

Fleck C. (2013) 'The impact factor fetishism', European Journal of Sociology, 54(2): 327-356.

Fox, M. F., and C. A. Faver (1985) 'Men, women, and publication productivity: Patterns among social work academics', Sociological Quarterly, 26: 537-549.

Fox M. F. (2005) 'Gender, family characteristics, and publication productivity among scientists', Social Studies of Science, 35(1): 131-150.

Fletcher, J. K. (2001) Disappearing acts: Gender, power, and relational practice at work. MA, Boston: MIT Press.

Fraser, N. (1995) False antithesis. In S. Benhabib, J. Butler, D. Cornell, & N. Fraser (Eds.), Feminist contentions (pp. 59-74). New York, NY: Routledge.

Fry, J. (2004) 'The cultural shaping of ICTs within academic fields: Corpus-based linguistics as a case study', Literary and Linguistic Computing 19(3): 303-319.

Garcia-Pérez, M. A. (2009) 'A multidimensional extension to Hirsch's H-index', Scientometrics, 81(3): 779-785.

Guena, A. and B. R. Martin (2003) 'University research evaluation and funding: An international comparison', Minerva 41(4): 277-304.

Gunnarsson, L. (2011). 'A defence of the category "women"'. Feminist theory, 12(1): 23-37.

Gläser, J. and G. Laudel (2007) 'The social construction of bibliometric evaluations', in R. Whitley and J. Gläser (eds), The Changing Governance of The Sciences: The Advent of Research Evaluation Systems. (Doodrecht: Springer): 101-123.

Haraway, D. J. (1989). Primate visions: Gender, race, and nature in the world of modern science. Hove: Psychology Press.

Haraway, D. J. (1997). Modest− Witness@ Second− Millennium. FemaleMan− Meets− OncoMouse: Feminism and Technoscience. Hove: Psychology Press.

Hicks, D., Wouters, P., Waltman, L., & de Rijcke, S., and I. Rafols (2015) 'Bibliometrics: The Leiden Manifesto for research metrics', Nature, 520: 429-31.

Hirsch, J. E. (2005) 'An index to quantify an individual's scientific research output', Proceedings of the National Academy of Sciences, 102: 16569-16572.

Hunter, L., and E. Leahey (2010) 'Parenting and research productivity: New evidence and methods', Social Studies of Science, 40(3): 433–451

Jessop, B. (2008) 'A cultural political economy of competitiveness and its implications for higher education', in B. Jessop et al. (eds), Education and the Knowledge-Based Econonomy in Europe (Rotterdam: Sense Publishers): 11-39
Jessop, B. (2008) 'A cultural political economy of competitiveness and its implications for higher education', in B. Jessop et al. (eds), Education and the Knowledge-Based Econonomy in Europe (Rotterdam: Sense Publishers): 11-39

Kelly, C. D. and M. D. Jennions (2006) 'The h index and career assessment by numbers', TRENDS in Ecology and Evolution, 21: 167–170.

Kalpazidou Schmidt, E. (2009) 'Nordic higher education systems in a comparative perspective: Recent reforms and impacts', The Journal of Finance and Management in Colleges and Universities, 6: 269–298.

Knights, D. and W. Richards (2003) 'Sex discrimination in UK academia', Gender, Work & Organization, 10(2): 213-238.

Kyvik, S. (1990) 'Motherhood and scientific productivity', Social Studies of Science, 20(1): 149-160.

Kyvik, S. and M. Teigen (1996) 'Child care, research collaboration, and gender differences in scientific productivity', Science, Technology, & Human Values 21(1): 54-71.

Lamont, M. (2009) How Professors Think: Inside the Curious World of Academic Judgment (Cambridge, MA: Harvard University Press).

Lariviere, V., Ni, C. Q., Gingras, Y., Cronin, B., and C. R. Sugimoto (2013) 'Global gender disparities in science', Nature, 504(7479): 211–213.

Latour, B. and S. Woolgar (1986) Laboratory Life: The Construction of Scientific Facts (Princeton, NJ: Princeton University Press).

Levin, M. (1988) 'Caring new world: Feminism and science', The American Scholar, 57: 100-106.

Light, R. (2013). 'Gender inequality and the structure of occupational Identity: The case of elite sociological publication'. Networks, Work, and Inequality, 24, 239-268.

Long, J. S. (1992) "Measures of sex-differences in scientific productivity", Social Forces, 71(1): 159–178.

Luke, C. (1997) 'Quality assurance and women in higher education', Higher education, 33(4), 433-451.

Macdonald, S., & J. Kam (2007) 'Ring a Ring o'Roses: Quality Journals and Gamesmanship in Management Studies', Journal of Management Studies,44(4): 640-655.

Mairesse, J. and M. Pezzoni (2013). 'Does gender affect scientific productivity? A critical review of the empirical evidence and a panel data econometric analysis for French physicists'. Submitted to the AFSE Meeting, 26 June 2013, Aix en Provence.

Mallard, G., M. Lamont and J. Guetzkow (2009) 'Fairness as appropriateness: Managing epistemological differences in peer review', Science, Technology and Human Values, 34(5): 573-606.

Maliniak, D., R. M. Powers and F. Barbara (2013) 'The gender citation gap in international relations'. International Organization, 67(4):889-922.

Mason, M. A., and M. Goulden (2004) 'Do babies matter (Part II)? Closing the baby gap', Academe, November–December,

March, J. G., and H. A. Simon (1958) Organizations (New York: Wiley).

Mauleón, E., and M. Bordons (2006) 'Productivity, impact and publication habits by gender in the area of materials science', Scientometrics, 66(1): 199–218.

McElhinny, B., Hols, M., Holtzkener, J., Unger, S., and C. Hicks (2003) 'Gender, publication and citation in sociolinguistics and linguistic anthropology: the construction of a scholarly canon', Language in Society, 32: 299–328.

Moed, H. F., M. Luwel and A. J. Nederhof (2002) 'Towards research performance in the humanities', Library Trends, 50(3): 498–520.

Morley, L. (2003) Power and quality in higher education (Berkshire: Open University Press).

Morley, L. (2005) 'The micropolitics of quality', Critical Quarterly, 47(1‐2): 83-95.

Müller, R. (2014) Postdoctoral life scientists and supervision work in the contemporary university: A case study of changes in the cultural norms of science. Minerva, 52(3), 329-349.

Nielsen, M. W. (2015a) Gender inequality and research performance: moving beyond individual-meritocratic explanations of academic advancement. Studies in Higher Education, 1-17 (online version).

Nielsen, M. W. (2015b) Gender consequences of a national performance-based funding model: new pieces in an old puzzle. Studies in Higher Education, 1-23 (online version).

Nielsen, M. W. (2016). Limits to meritocracy? Gender in academic recruitment and selection processes. Science & Public Policy, 43 (3): 386-399.

Plowman, D. A. and Smith, A. D. (2011) 'The gendering of organizational research methods: Evidence of gender patterns in qualitative research', Qualitative Research in Organizations and Management: An International Journal, 6(1): 64-82.

Prozesky, H (2008) 'A career-history analysis of gender differences in publication productivity among South African academics', Science Studies, 21(2): 47–67.

Prpic, K., (2002) 'Gender and productivity differentials in science', Scientometrics, 55(1): 27–58.

Rafols, I., L. Leydesdorff , A. O'Hare, P. Nightingale and A. Stirling (2011) 'How journal rankings can suppress interdisciplinary research: A comparison between innovation studies and business and management', DRUID Working Paper No. 11-05.

Ritzberger, K. (2008). A ranking of journals in economics and related fields.German Economic Review, 9(4), 402-430.

Salmi, J. (2009) The Challenge of Establishing World Class Universities (Washington DC: The World Bank).

Schiebinger, L. (1999) Has Feminism Changed Science? (Cambridge, MA: Harvard University Press).

Schneider, J. W. (2009) 'An outline of the bibliometric indicator used for performance-based funding of research institutions in Norway', European Political Science, 8: 364-378.

Seglen, P. (1997) 'Why the impact factor of journals should not be used for evaluating research', British Medical Journey, 314: 497-502.

Seierstad, C. and G. Healy (2012) 'Women's equality in the Scandinavian academy: A distant dream?' Work Employment & Society, 26(2): 296-313.

Simmel, G. (1903) 'Die Grosstädte und das Geistesleben, translated to 'The metropolis and modern life', by Kurt Wolff, in D. Weinstein (ed), 1950. The Sociology of Georg Simmel (New York: Free Press): 409-424.

Symonds, M. R. E., N. J. Gemell, T. L. Braisher, K. L. Gorringe, M. A. Elgar (2006) 'Gender differences in publication output: Towards an unbiased metric of research performance' in T. Tregenza (ed) PLoS ONE 1 (1): doi:10.1371/journal.pone.0000127

Sonnert, G. and G. Holton (1996) 'Career patterns of women and men in the sciences', American Scientist, 84(1): 63-71.

Svensson, G. (2006). 'The paradoxnoia of top journal (s) in marketing. European Journal of Marketing', 40(11/12): 1153-1168.

Thomas, R. and A. Davies (2003) 'Gender and New Public Management: Reconstituting academic subjectivities', Gender, Work and Organization, 9(4): 372-397.

Traweek, S. (2009) Beamtimes and lifetimes: The world of high energy physicists. MA, Boston: Harvard University Press.

Valian, V. (1999) 'The cognitive bases of gender bias', Brooklyn Law Review, 65:1037-1061.

Van Den Brink, M., B. Fruytier and M. Thunnissen (2013) 'Talent management in academia: Performance systems and HRM policies', Human Resource Management Journal, 23(2): 180-195.

Van Raan, A. F. J. (1998) 'The influence of international collaboration on the impact of research results: some simple mathematical considerations concerning the role of self-citations', Scientometrics, 42(3): 423-428.

Verner, M. (2008) 'Gender differences in rank within the academic profession: The case of Denmark', Paper presented at the IZA Workshop on Research in Economics: Rewards, Evaluation and Funding, Bonn, Germany.

Ward, K.B., Gast, J., and L. Grant (1992) 'Visibility and dissemination of women's and men's sociological scholarship', Social Problems, 39: 291–298.

Weingart, P. (2005) 'Impact of bibliometrics upon the science system: Inadvertent consequences', Scientometrics, 62(1): 117-131.

Whitley, R. (1984) The Intellectual and Social Organization of the Sciences (Oxford: Oxford University Press).

Whitley, R. (2007) 'Changing governance of the public sciences: The consequences of establishing research evaluation systems for knowledge production in different countries and scientific fields' in R. Whitley, J. Gläser (eds), The Changing Governance of the Sciences (Doodrecht: Springer).

West, C., and Zimmerman, D. H. (1987) Doing gender. Gender & society, 1(2), 125-151.

Willmott, H. (2011) 'Journal list fetishism and the perversion of scholarship: reactivity and the ABS list', Organization, 18(4): 429-442.

Xie, Y., & Shauman, K. A. (2003) Women in science: Career processes and outcomes. Cambridge, MA: Harvard University Press.
Research Papers