European Research Council

Principles of proposal evaluation at ERC Established by the

European Commission

Jonas Bunikis, MD PhD

Ioannis Sagias, PhD
Research Programme Officer September 2012
ERC Executive Agency Varna – Sofia, Bulgaria
jonas.bunikis@ec.europa.eu
 European Research Council

In today’s talk
Established by the
European Commission

>> ERC evaluation panels and steps

>> Managing evaluations at ERC
>> ERC evaluations: Lithuanian experience
>> Evaluation of new schemes (Synergy and
PoC)
 European Research Council

ERC Panel structure (1)

Established by the
European Commission

Physical Sciences & Engineering

PE1 Mathematical foundations

PE2 Fundamental constituents of matter

PE3 Condensed matter physics

PE4 Physical & Analytical Chemical sciences

PE5 Materials & Synthesis

PE6 Computer science & informatics

PE7 Systems & communication engineering

PE8 Products & process engineering

PE9 Universe sciences

PE10 Earth system science
 European Research Council

ERC Panel structure (2)

Established by the
European Commission

Life Sciences

LS1 Molecular & Structural Biology & Biochemistry

LS2 Genetics, Genomics, Bioinformatics & Systems Biology

LS3 Cellular and Developmental Biology

LS4 Physiology, Pathophysiology & Endocrinology

LS5 Neurosciences & neural disorders

LS6 Immunity & infection

LS7 Diagnostic tools, therapies & public health

LS8 Evolutionary, population & environmental biology

LS9 Applied life sciences & biotechnology
 European Research Council

ERC Panel structure (3)

Established by the
European Commission

Social Sciences and Humanities

SH1 Individuals, institutions & markets

SH2 Institutions, values, beliefs and behaviour

SH3 Environment & society

SH4 The Human Mind and its complexity

SH5 Cultures & cultural production

SH6 The study of the human past
 Panel Panel Share European Research Council
Panel
European Research Council

NATIONALITY Chairs Members Total in % NATIONALITY Chairs

AT 1 35 36 2,4% IN
AU 5 5 0,3% IS Established by the
European Commission

BE
BG
53
2
53
2
3,6%
0,1% ERC panel IT
JP
CA 2 6 8 0,5% LT
CH
CR
3 39
3
42
3
2,8%
0,2%
experts MX
NE
CY 4 4 0,3% NL
CZ 28 28 1,9% NO
DE 17 171 188 12,6% PL
DK 5 31 36 2,4% PT
EE 2 2 0,1% RO
EL 1 25 26 1,7% RS
ES 3 64 67 4,5% RU
FI 10 31 41 2,8% SE
FR 14 138 152 10,2% SK
GI 1 1 0,1% SL
HK 3 3 0,2% SN
HR 3 3 0,2% TR
HU 39 39 2,6% TW
IE 13 13 0,9% UK
IL 27 27 1,8% US
Total
 European Research Council
Panel Panel Share
European Research Council

NATIONALITY Chairs Members Total in %

IN 3 3 0,2%
IS 1 1 0,1% Established by the
European Commission

IT 15 117 132 8,9%

JP 14 14 0,9%
LT
MX
3
2
3
2
0,2%
0,1% ERC panel
experts
NE 2 2 0,1%
NL 9 49 58 3,9%
NO 4 16 20 1,3%
PL
PT
4
2
36
23
40
25
2,7%
1,7%
(cont.)
RO 7 7 0,5%
RS 2 2 0,1%
RU 4 4 0,3%
SE 4 55 59 4,0%
SK 3 3 0,2%
SL 1 1 0,1%
SN 1 1 0,1%
TR 14 14 0,9%
TW 1 1 0,1%
UK 12 171 183 12,3%
US 12 122 134 9,0%
 European Research Council

Evaluation steps Established by the

European Commission

STEP 1 STEP 2

Remote evaluation of full

Remote evaluation of proposal’s
proposal (sections B1 & B2) by
synopsis (section B1) by panel
panel members and external
members
reviewers

Panel meeting
Panel meeting (+ interview for StG &
CoG)
Proposals retained
for Step 2
Ranked list of
proposals

Feedback to
applicants
 European Research Council

Panel’s role
Established by the
European Commission

Panel Chairs and Panel Members

1. Familiarize with all the proposals submitted to the panel.
2. Individually review a subset of the proposals (primarily as
generalists).
3. Participate in the panel meetings.
4. Ensure equal and fair treatment of all proposals

Panel Chairs
1.Chair panel meetings
2.Assign proposals to the panel members
3.Ensure the quality of reviews and procedures of the panel
 European Research Council

Remote reviewers
Established by the
European Commission

1. Are nominated by the panel at the end of Step 1 meeting.

2. Individually review 1-2 proposals as specialists.
3. Do NOT participate in the panel meetings.
 European Research Council

Evaluation criteria
Established by the
European Commission

Principal investigator
Research output
Intellectual capacity and creativity
Time commitment (an evaluation, not eligibility criterion)
Research project
Ground-breaking nature
Potential impact
Methodology
High risk/high gain balance (breakthrough vs incremental
research)
Host Institution (not a criterion any more)
HI contribution to the project (in Resources)
Multiple HIs- justification of the scientific added value
 European Research Council

The Criterion: Excellence

Established by the
European Commission

Proposal scoring range by ERC panel

4: outstanding
3: excellent
2: very good
1: non-competitive
 European Research Council

Managing ERC evaluation: Initial preparations

Established by the
European Commission

• Call Coordination team; Evaluation Roadmap

• Panel recruitment
Active & shadow panels
Roles of the ERC Scientific Council & Scientific
Officers
Sources
• Panel Chair meeting
Goal
Chair’s letter to the panels
 European Research Council

Processing of submitted proposals

Established by the
European Commission

Pre-assignment processing
Eligibility (StG and CoG: eligibility window;
resubmissions)
CoG vs StG change requests
Cross-panel transfers
The roles of ERC staff and panel Chairs
 European Research Council

ERC Evaluations: Assignment

Established by the
European Commission

• Proposal Assignment
Conflict of Interest verification; automatic/manual
Draft assignments
Lead reviewers
Pluridisciplinary proposals; Cross-panel reviews
• Chair’s approvals
 European Research Council

ERC Evaluations: Remote reviewing

Established by the
European Commission

• Guide for Peer Reviewers

• Chair’s letter to the panel
Conflict of Interest
Review quality
Scoring
• Remote reviewing
Reporting deadline
Distribution of compiled individual reviews
 European Research Council

ERC Evaluations: Step 1 Panel Meeting

Established by the
European Commission

• Meeting agenda/Panel briefing & rules

• Meeting input/output
materials (compiled reviews & IT support)
deliverables
ranking list
CoI list
remote reviewers’ nomination list
panel comments
panel report
• Modus operandi
(co)chairing & ad hoc situations
discussion/consensus/scoring practices
quality threshold
 European Research Council

ERC Evaluations: Post-step 1

Established by the
European Commission

• Feedback to the applicants; Evaluation Summary Report

• Recruitment of remote reviewers
• Full proposal assignments for panellists and remote
reviewers
• StG and CoG: Interview scheduling and approval
 European Research Council

ERC Evaluations: Step 2 Panel Meeting

Established by the
European Commission

• AdG as usual; Interviews for StG and CoG

• deliverables
ranking list; incl. reserve list
CoI list
panel comments
panel report
 European Research Council

ERC Evaluations: Post-step 2

Established by the
European Commission

• Feedback to the applicants; ESRs

• Funded projects: Ethical issues screening & review
 European Research Council

ERC Evaluations: Redress

Established by the
European Commission

• Post-step 1 & Post-step 2

• Redress reasons; procedure
• Scientific Officer’s role
• Committee
 European Research Council

EU-12 on IDEAS learning curve

Established by the
European Commission

Study of participation of Lithuanian applicants in

ERC IDEAS programme 2007-2011

Commissioned by the Ministry of Education and Science of Lithuania

Performed by Vaida Bankauskaite and Jonas Bunikis, ERC Executive Agency
 European Research Council

Bulgaria – Lithuania connection Established by the

European Commission

улица във Варна

1851-1927
Д-р Йо:нас Басана:вичюс
литовски учен, политик, деец на литовското
национално възраждане, български общественик
 European Research Council

Applications to ERC calls by researchers from

Lithuanian institutions (2007-2011) Established by the
European Commission

Note: PE- physics & engineering; LS- life sciences; SH- social sciences & humanities. QT- quality threshold;
n.a.- not applicable.
 European Research Council

Track record of Lithuanian applicants with

different Step 1 evaluation outcomes. Established by the
European Commission

Starting grants

Advanced grants

Note: Mean value of each metric in a group is presented. Only applicants to LS and PE domains are included in the analysis.
QT- quality threshold. Data for each applicant retrieved from SCOPUS. * n=11 evaluated proposals (one applicant has
submitted two proposals to PE and LS panels in different calls)
 European Research Council

Reviewer comments on Lithuanian StG proposals

with different Step 1 evaluation outcomes. Established by the
European Commission

Note: N=11 and N=4 evaluated proposals in correspondingly <QT and >QT groups. QT- quality threshold.
 European Research Council

Net-balance between reviewer positive or negative

commenting on various aspects of StG Lithuanian
proposals with different Step 1 evaluation outcome.
Established by the
European Commission

Note: N=11 and N=4 evaluated proposals in correspondingly <QT and >QT groups. QT- quality threshold.
* subtraction value between average no. (per review) of positive and negative comments within each outcome group.
 European Research Council

Concluding/putative extrapolations Established by the

European Commission

- Peer-review procedures and practices at ERC ensures

unbiased evaluation of the proposals, with no evidence of
discrimination against the proposals from EU-12 countries.

- Limited competitiveness of EU-12 proposals is at least in part

determined by sub-excellent track record of the applicants.

- Scientific merit of EU-12 proposals suffers primarily from

conceptual immaturity, lack of innovation, uncertain feasibility
and insufficient clarity of objectives.
ERC Synergy Grants: European Research Council

New pilot scheme in

2012 Established by the
European Commission

• Small groups of 2-4 researchers (not networks!)

• No thematic priorities
• Researchers can be in same country, even same
institution
• Unique configuration of scientists with
complementary skills
• High risk/high gain research
• Possible requested amount:
• max. € 15,000,000 for up to 6 years
• Same profile for applicants mainly as AdG
 European Research Council

SyG 2012: Evaluation procedure

Established by the
European Commission

Step 1a (single panel) Step 2a (1 new Panel)

Remote evaluation of short Panel members reassess the
proposals by panel members retained proposals based on
their step 1 reports

Panel chairs meet: preselect

proposals for full review Panel meeting: Select
Up to 7 times available budget proposals for interview
Step 1b (formation of 5 panels) Up to 1.5 times available budget

Remote evaluation of full Step 2b (Same panel as 2a)

proposals by panel members and
Panel meeting and interviews
external evaluators

Select proposals up to
Panel meeting: proposals available budget
retained for Step 2a
Proposals funded in priority order
Up to 2.5 times available budget based on their rank
 European Research Council

ERC Synergy 2012 – Overview

at call closure - step 1a Established by the
European Commission

710 proposals submitted

2255 PIs in total

1727 partners

Average duration: 69 months ( 5 y. 5 m.)

Average budget requested: € 8.3 million

Average number of PIs per proposal: 3.1

Average number of partners per proposal: 2.4
 European Research Council

SyG2012: Step 1a results overview

Established by the
European Commission

• 99 proposals passing to Step 1b (33 to Step 2a)

• 332 PIs (99 Corresponding PI + 233 PIs)

• 243 beneficiaries

• 14,2% success rate

• Average budget requested 11.7 million €

│ 33
 European Research Council

New instrument (since 2011):

ERC Proof of concept Established by the
European Commission

─ Initiated to help ERC grant-holder to bridge the gap between their

research and the earliest stage of a marketable innovation
─ Supporting grant-holders during the pre-demonstration
─ Up to 150.000 Euro per grant
─ One step evaluation
─ First call in 2011 with 2 deadlines
─ Next call in 2012 (deadlines: 3 May and 3 October 2012)
 European Research Council

ERC Proof of Concept – Reviewers

As for other ERC evaluation panels:

─ About 15 members
─ Nominated by the Scientific Council
─ Appointment procedure
─ Experts in assessing the evaluation criteria

Unlike other ERC evaluation panels:

─ No scientists (Tech Transfer, IPR, start-ups…)
─ Preferably from EU or ACs

│ 35
 European Research Council

ERC Proof of Concept – Procedure

As for other ERC evaluations:

─ Remote evaluation (WebEva online tool)
─ CoI rule strictly applied
─ Comments to feedback the applicant

Unlike other ERC evaluation panels:

─ No ‘panel meeting’ (unless necessary due to tie)
─ PASS/FAIL mark on each of the 3 criteria
─ “…ranked according to the number of pass marks which they
received from peer reviewers.”

│ 36
 European Research Council

Established by the
European Commission

благодаря за вниманието