different meanings for different people in many different contexts. Evaluation is a procedure that reviews a program critically. As part of this review, we aim to explore the following questions: What are the reasons behind trying to understand and evaluate research impact? Metrics have commonly been used as a measure of impact, for example, in terms of profit made, number of jobs provided, number of trained personnel recruited, number of visitors to an exhibition, number of items purchased, and so on. 1. Decker et al. 0000003495 00000 n Any tool for impact evaluation needs to be flexible, such that it enables access to impact data for a variety of purposes (Scoble et al. HEIs overview. Clearly there is the possibility that the potential new drug will fail at any one of these phases but each phase can be classed as an interim impact of the original discovery work on route to the delivery of health benefits, but the time at which an impact assessment takes place will influence the degree of impact that has taken place. Many times . It is desirable that the assignation of administrative tasks to researchers is limited, and therefore, to assist the tracking and collating of impact data, systems are being developed involving numerous projects and developments internationally, including Star Metrics in the USA, the ERC (European Research Council) Research Information System, and Lattes in Brazil (Lane 2010; Mugabushaka and Papazoglou 2012). Differences between these two assessments include the removal of indicators of esteem and the addition of assessment of socio-economic research impact. HEFCE indicated that impact should merit a 25% weighting within the REF (REF2014 2011b); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% (Russell Group 2009; Jump 2011) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments (REF2014 2010). The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes. Here is a sampling of the definitions you will see: Mirriam-Webster Dictionary Definition of Assessment: The action or an instance of assessing, appraisal . In the UK, there have been several Jisc-funded projects in recent years to develop systems capable of storing research information, for example, MICE (Measuring Impacts Under CERIF), UK Research Information Shared Service, and Integrated Research Input and Output System, all based on the CERIF standard. A university which fails in this respect has no reason for existence. Productive interactions, which can perhaps be viewed as instances of knowledge exchange, are widely valued and supported internationally as mechanisms for enabling impact and are often supported financially for example by Canadas Social Sciences and Humanities Research Council, which aims to support knowledge exchange (financially) with a view to enabling long-term impact. Evaluation of impact is becoming increasingly important, both within the UK and internationally, and research and development into impact evaluation continues, for example, researchers at Brunel have developed the concept of depth and spread further into the Brunel Impact Device for Evaluation, which also assesses the degree of separation between research and impact (Scoble et al. "Evaluation is a process of judging the value of something by certain appraisal." Characteristics of evaluation in Education Below are some of the characteristics of evaluation in education, Continuous Process Comprehensive Child-Centered Cooperative Process Common Practice Teaching Methods Multiple Aspects Continuous Process In line with its mandate to support better evaluation, EvalNet is committed to working with partners in the global evaluation community to address these concerns, and is currently exploring options for additional work. An empirical research report written in American Psychological Association (APA) style always includes a written . The following decisions may be made with the aid of evaluation. In the UK, more sophisticated assessments of impact incorporating wider socio-economic benefits were first investigated within the fields of Biomedical and Health Sciences (Grant 2006), an area of research that wanted to be able to justify the significant investment it received. Over the past year, there have been a number of new posts created within universities, such as writing impact case studies, and a number of companies are now offering this as a contract service. What is the Difference between Formative and Summative Evaluation through Example? One might consider that by funding excellent research, impacts (including those that are unforeseen) will follow, and traditionally, assessment of university research focused on academic quality and productivity. Capturing data, interactions, and indicators as they emerge increases the chance of capturing all relevant information and tools to enable researchers to capture much of this would be valuable. (2007) adapted the terminology of the Payback Framework, developed for the health and biomedical sciences from benefit to impact when modifying the framework for the social sciences, arguing that the positive or negative nature of a change was subjective and can also change with time, as has commonly been highlighted with the drug thalidomide, which was introduced in the 1950s to help with, among other things, morning sickness but due to teratogenic effects, which resulted in birth defects, was withdrawn in the early 1960s. Teresa Penfield, Matthew J. Baker, Rosa Scoble, Michael C. Wykes, Assessment, evaluations, and definitions of research impact: A review, Research Evaluation, Volume 23, Issue 1, January 2014, Pages 2132, https://doi.org/10.1093/reseval/rvt021. This framework is intended to be used as a learning tool to develop a better understanding of how research interactions lead to social impact rather than as an assessment tool for judging, showcasing, or even linking impact to a specific piece of research. The definition problem in evaluation has been around for decades (as early as Carter, 1971), and multiple definitions of evaluation have been offered throughout the years (see Table 1 for some examples). A collation of several indicators of impact may be enough to convince that an impact has taken place. 0000001883 00000 n Prague, Czech Republic, Health ResearchMaking an Impact. To allow comparisons between institutions, identifying a comprehensive taxonomy of impact, and the evidence for it, that can be used universally is seen to be very valuable. Definition of evaluation. Different authors have different notions of educational evaluation. In the UK, evaluation of academic and broader socio-economic impact takes place separately. Classroom Assessment -- (sometime referred to as Course-based Assessment) - is a process of gathering data on student learning during the educational experience, designed to help the instructor determine which concepts or skills the students are not learning well, so that steps may be taken to improve the students' learning while the course is Definitions of Performance Appraisal - By McGregor and Dale Beach . Evidence of academic impact may be derived through various bibliometric methods, one example of which is the H index, which has incorporated factors such as the number of publications and citations. only one author attempts to define evaluation. From an international perspective, this represents a step change in the comprehensive nature to which impact will be assessed within universities and research institutes, incorporating impact from across all research disciplines. One of the advantages of this method is that less input is required compared with capturing the full route from research to impact. What emerged on testing the MICE taxonomy (Cooke and Nadim 2011), by mapping impacts from case studies, was that detailed categorization of impact was found to be too prescriptive. Findings from a Research Impact Pilot, Institutional Strategies for Capturing Socio-Economic Impact of Research, Journal of Higher Education Policy and Management, Introducing Productive Interactions in Social Impact Assessment, Measuring the Impact of Publicly Funded Research, Department of Education, Science and Training, Statement on the Research Excellence Framework Proposals, Handbook on the Theory and Practice of Program Evaluation, Policy and Practice Impacts of Research Funded by the Economic Social Research Council. Any person who has made a significant . (2007), Nason et al. 2005; Wooding et al. Muffat says - "Evaluation is a continuous process and is concerned with than the formal academic achievement of pupils. One notable definition is provided by Scriven (1991) and later adopted by the American Evaluation Association (): "Evaluation is the systematic process to determine merit, worth, value, or . The definition of health is not just a theoretical issue, because it has many implications for practice, policy, and health services. The RQF was developed to demonstrate and justify public expenditure on research, and as part of this framework, a pilot assessment was undertaken by the Australian Technology Network. 0000008241 00000 n Merit refers to the intrinsic value of a program, for example, how effective it is in meeting the needs those it is intended help. Indicators were identified from documents produced for the REF, by Research Councils UK, in unpublished draft case studies undertaken at Kings College London or outlined in relevant publications (MICE Project n.d.). The term comes from the French word 'valuer', meaning "to find the value of". In undertaking excellent research, we anticipate that great things will come and as such one of the fundamental reasons for undertaking research is that we will generate and transform knowledge that will benefit society as a whole. It is acknowledged in the article by Mugabushaka and Papazoglou (2012) that it will take years to fully incorporate the impacts of ERC funding. One way in which change of opinion and user perceptions can be evidenced is by gathering of stakeholder and user testimonies or undertaking surveys. Figure 1, replicated from Hughes and Martin (2012), illustrates how the ease with which impact can be attributed decreases with time, whereas the impact, or effect of complementary assets, increases, highlighting the problem that it may take a considerable amount of time for the full impact of a piece of research to develop but because of this time and the increase in complexity of the networks involved in translating the research and interim impacts, it is more difficult to attribute and link back to a contributing piece of research. This article aims to explore what is understood by the term research impact and to provide a comprehensive assimilation of available literature and information, drawing on global experiences to understand the potential for methods and frameworks of impact assessment being implemented for UK impact assessment. The quality and reliability of impact indicators will vary according to the impact we are trying to describe and link to research. The book also explores how different aspects of citizenship, such as attitudes towards diverse population groups and concerns for social issues, relate to classical definitions of norm-based citizenship from the political sciences. Thalidomide has since been found to have beneficial effects in the treatment of certain types of cancer. 2007; Nason et al. Media coverage is a useful means of disseminating our research and ideas and may be considered alongside other evidence as contributing to or an indicator of impact. (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. 0000006922 00000 n The case study of the Research Information System of the European Research Council, E-Infrastructures for Research and Innovation: Linking Information Systems to Improve Scientific Knowledge, Proceedings of the 11th International Conference on Current Research Information Systems, (June 69, 2012), pp. The Oxford English Dictionary defines impact as a Marked effect or influence, this is clearly a very broad definition. 2. The criteria for assessment were also supported by a model developed by Brunel for measurement of impact that used similar measures defined as depth and spread. Times Higher Education, Assessing the Impact of Social Science Research: Conceptual, Methodological and Practical Issues, A Profile of Federal-Grant Administrative Burden Among Federal Demonstration Partnership Faculty, Department for Business, Innovation and Skills, The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental and cultural returns of publicly funded research, Reforming the Evaluation of Research. A very different approach known as Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions (SIAMPI) was developed from the Dutch project Evaluating Research in Context and has a central theme of capturing productive interactions between researchers and stakeholders by analysing the networks that evolve during research programmes (Spaapen and Drooge, 2011; Spaapen et al. In the UK, UK Department for Business, Innovation, and Skills provided funding of 150 million for knowledge exchange in 201112 to help universities and colleges support the economic recovery and growth, and contribute to wider society (Department for Business, Innovation and Skills 2012). Impact is assessed alongside research outputs and environment to provide an evaluation of research taking place within an institution. The traditional form of evaluation of university research in the UK was based on measuring academic impact and quality through a process of peer review (Grant 2006). It is important to emphasize that Not everyone within the higher education sector itself is convinced that evaluation of higher education activity is a worthwhile task (Kelly and McNicoll 2011). 4 0 obj Impact assessments raise concerns over the steer of research towards disciplines and topics in which impact is more easily evidenced and that provide economic impacts that could subsequently lead to a devaluation of blue skies research.
Brighton & Hove Systema club