About Cohesion

DD3001 - Research: Theory, Method, Practice
Motivation
Hope DD3001 helps you avoid this situation
The Significance of Research
9-Oct-10
Am assuming you already avoid this

Obsolete Scientific Method
Computer Scientific Method
Hypothesis
Hunch
Sequence of experiments.
1 experiment &
change all parameters
Change 1 parameter per

experiment.
Discard if it doesnt support

hunch
Prove/Disprove Hypothesis.
Why waste time?
Document for others to reproduce results.
We know this.
Source: How to have a bad career in research/academia by David Patterson, Feb 2002.
Remember Science/Research is Hard
Why is science hard?

Intrinsic - Our knowledge is imperfect. Makes it difficult to
correctly update this knowledge.

Personal - Scientists have biases and flaws in their approach.
Community - The way we organize ourselves can limit
progress.
Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.
Reason for Difficulty: Intrinsic
All possible theories
True theories
Theories we think are true

True theories
Much of what we consider today is probably wrong

Our successors will be amazed by the
amount of scientific rubbish discussed
today that is, if they have the patience
to trawl through the electronic archives of
obsolete journals.
Sydney Brenner, (1927)
Sydney Brenner is a South African biologist and a 2002 Nobel prize in
Physiology or Medicine laureate.
Theories we
think we have
tested well
True theories
Truth perhaps beyond our current imagination

..my own suspicion is that the Universe
is not only queerer than we suppose, but
queerer than we can suppose
J.B.S. Haldane, (1892-1964)
J.B.S. Haldane was a British-born geneticist and evolutionary biologist.
Theories considered
Theories we
think we have
tested well

True theories
Facts are simple in retrospect

Somebody once observed to the eminent philosopher Wittgenstein
how stupid medieval Europeans living before the time of
Copernicus must have been that they could have looked at the sky
and thought that the sun was circling the earth. Surely a modicum
of astronomical good sense would have told them that the reverse
was true. Wittgenstein is said to have replied:
I agree. But I wonder what it would

have looked like if the sun had been
circling the earth.
Source: James Burke. The Day the Universe Changed. TV Series 1985
Why we need help doing this?

Isnt science just common sense?
Yes, but it is advanced common sense.
Both lay observers and experts tend to...
See patterns in random data.
Generalize from incomplete samples.
See confirmation for favored ideas in ambiguous data.
We tend to believe...
What we already think we know.
What we hear second-hand.
What our community believes.
Source: T. Gilovich. How We Know What Isnt So. 1991.
Reason for Difficulty: Personal
Personal failings: the human condition

Slop - Presenting and doing research so it is not possible to
know what was done or observed

Confused or unclear procedures
Confused or unclear data-collection
Imprecise theorizing
Unexpressed assumptions
Informal derivation of predictions
Sloth - Doing too little!
Important data not obtained or recorded
Partial analysis of the data
Source: Donald D. Jensen. Unpublished lecture notes, University of Nebraska-Lincoln. 1995.
Personal Failings: Self-deception

Precipitance - Jumping to a conclusion
? Premature decision on an issue

? Accepting as established something that deserves further
investigation
Propaganda - Biased presentation of a theory or data
? Special pleading
? Salesmanship rather than science.
Prejudice - Biased evaluation of theory and data
? Expect more of other theories than of ones own

? Tilt the playing field in favour of ones own theory
Preservation - Clinging to a theory despite clear evidence
that it is false.
Not a new idea (Francis Bacon 1600s)

Idols of the Tribe We see evidence for our favoured
hypotheses in random data.

Idols of the Cave Our personal likes and dislikes affect our
professional judgments.
Idols of the Marketplace Common language is too limited
for accurate scientific communication.

Idols of the Theatre Poor methods and philosophical
systems can doom scientific investigations.
Personal failings: Willful deception

Finagle - Adjusting data so that it fits a favoured theory.
? Minor fraud.
Filch - Stealing ideas or data without giving appropriate credit
? Plagiarism or other unauthorized use of the work of others.

Fraud - Falsifying data and the reports of investigations
performed.
Reason for Difficulty: Community
Community failings
Science is a human community
Like any other community, it has...
7 Fads and fashions

7 Rigid structures and traditions that may interfere with progress
7 Disagreements over goals and direction that consume resources
7 Outsize personalities who may detrimentally dominate the field
Science progresses in spite of these problems.
Community benefits
Science is a human community
Like any other community, it can...
X Share data, results and theories

X Quickly explore fads and fashions
X Give independent evaluation of ones own theories and work
X Provide competition and deadlines
Science progresses because of these benefits.
Havent scientists overcome these challenges?

X Yes
With practice
When using tools developed over the past few hundred
years.
7 No when unaided and having little practice.
Most undergraduate education trains you how to learn
what is already known, not how to learn what is currently

unknown or misunderstood.
DD3001 - Research: Theory, Method, Practice
Course Overview
Answer to your questions

1. What is this course about?
We will review how research gets done and the important
components of your research career. More details...
2. Do I have to attend all the lectures?

No, but at least 4. But you do have to do all the assigned readings and
responses. Responses for non-attended lectures will be longer as you will
have missed the chance to participate in discussions during the lecture.
3. How can I access the reading material?

I have password protected the website containing the reading
material. I have put the password on the board.
Goals of DD3001
Inform you about the research process help your
burgeoning research career.

Raise your awareness that progress in research is very
non-linear. And requires hard work and perseverance!

Acquaint you with basic strategies for your research career.
? Identify important and solvable research topics, questions and
hypotheses.
? Select between alternative research directions.
? Importance of communication via reading and writing papers.
Make you consider some of the responsibilities of being a
member of the research community.

Teach you some strategies for clear and effective writing.
Outline of the lecture topics

1. Choosing a research topic.
? Strategy and Tactics for Discovery
2. You and your code; Reproducible Research.

3. Communication with other scientists:
Reading & Reviewing papers.
4. Communication with other scientists:
Paper writing.
5. Ethics.
6. Last lecture may be devoted to editing of a paper!
In most of the lectures will devote some time to writing tips.
Papers and texts

Papers & Essays
? General advice to young researchers

? Historical perspective on discoveries
? Stimulating papers for discussion
Selected chapters from these books
? Advice to a Young Scientist by Peter Medawar.

? The Incomplete Guide to the Art of Discovery by Jack E.
Oliver.
? Style - Lessons in Clarity and Grace by Joseph M.
Williams.
Course structure
Standard lectures interspersed with
? discussion of the pre-assigned readings,

? discussion of issues raised during the lectures,
? jotting down instant responses to questions.
What you will do in this course!

Read some interesting articles and essays about science
Think about their content.
Reflect on their relevance to your thoughts about scientific
research.
Summarize and write down these thoughts and reflections.
Write 2 essays to investigate an issue in more depth.
Details of assessment
Course is pass/fail.
To pass you must
Read the Medawar book and write a 400 word response.
Complete and submit the reading responses for the pre-lecture
reading assignments to an acceptable level.

Complete and submit two essay assignment of sufficient
merit. See course website for details.
Details of assessment
Course is pass/fail.
To pass you must
Read the Medawar book and write a 400 word response.
Complete and submit the reading responses for the pre-lecture
reading assignments to an acceptable level.

Complete and submit two essay assignment of sufficient
merit. See course website for details.
Remember you have to pass DD3001 to get your PhD.
Reading responses
At least 3 paragraphs
Upload response to bilda.
Content of response
? One paragraph summarizing main points in assigned reading.
? Make two or more key points that reveal the relevancy of the
paper to you or doing research.
Devote at least one short paragraph to each point.
Focus does not have to be the main point of the paper. But
the ones that are most relevant to you.
This course is a work in progress

Second time Im running this course and it is evolving. So
please be patient !
However, hopefully together we can have an interesting
journey exploring the issues raised.

Dont expect the final word on how to do research.
Expect ideas, partial and potentially conflicting answers
Please engage in discussions. Question what I say. By correct
probing we should closer to the truth!
Selecting Research Projects & Making

Discoveries
DD3001
October 14, 2013
Choose your research problem carefully

What problem to work on?
Most important decision you are going to make as a researcher
Dont choose a problem that is
?
trivial
trivial results
too hard
years of frustration & little accomplishment
in a crowded field
hard to get noticed or be first
Choose your research problem carefully

What problem to work on?
Most important decision you are going to make as a researcher
Dont choose a problem that is
?
trivial
trivial results
too hard
years of frustration & little accomplishment
in a crowded field
hard to get noticed or be first
Are you working on a problem because...
It was the first one that came to mind when you started.
Suggestion made by a professor.
It is related to what colleagues are working on.
Are you working on a problem because...
It was the first one that came to mind when you started.
Suggestion made by a professor.
It is related to what colleagues are working on.
A good basis for making the choice on how youll spend your
time for the next n years?
Careful deliberation is necessary
Select a problem
? whose solution will be important to the flow of science

? that is capable of being solved by you given your skills,
experience and talents.
As a young scientist
? this does not mean you have to limit yourself to problems of

modest significance.
? Remember
Significance not necessarily correlated with difficulty.
Tips for selecting a basic research project
1. Spend time and think very hard about the matter.

2. Do not settle for what happens to be at hand.
3. Choose the most fundamental significant and most
fundamental problem that may yield to your capabilities and
endeavour.
Tips for selecting a basic research project
1. Spend time and think very hard about the matter.

2. Do not settle for what happens to be at hand.
3. Choose the most fundamental significant and most
fundamental problem that may yield to your capabilities and
endeavour.
Do you agree? How did you make your choice?
!"#$%#&'(')$*+,#
63@"B("4
!#%%&AA
CDE&%5&8(F3-#&
!"#$%&'()*+,-)'.++(&$)/&,01(23(
6$"=3=?-?5@(
"4(!#%%&AA
!"#$%&'()*(+"&,-&(./0012(345&$(6*(7&8393$(.:;,&(<$5("4(5,&(!"-#=-&>2
Source: C. Loehle (1990) after P. Medawar (The Art of the Soluble)
!"#$%#&'(')$*+,#
63@"B("4
!#%%&AA
CDE&%5&8(F3-#&
!"#$%&'()*+,-)'.++(&$)/&,01(23(
6$"=3=?-?5@(
"4(!#%%&AA
How would you draw this curve?!"#$%&'()*(+"&,-&(./0012(345&$(6*(7&8393$(.:;,&(<$5("4(5,&(!"-#=-&>2

Source: C. Loehle (1990) after P. Medawar (The Art of the Soluble)
Remember
Competent research results can be ignored.
Why? 7 Worked on a trivial problem.

Modest work can produce a paper of great consequence.
Why? X Brilliance is in the choice of problem and the

approach of the solution.
A successful person isnt necessarily better than her less

successful peers at solving problems; her
pattern-recognition facilities have just learned what
problems are worth solving.
Ray Kurzweil
Interesting CS research problems

Explore or explain the behavior of algorithms, systems,
protocols, and other computational artifacts

Typical form of research questions:
? Why...
? How...
? Under what circumstances...
? What are the necessary and sufficient conditions for...
Answers
? 7 not yes/no
? Xparagraphs, mathematical or statistical models, simulations,
etc.
Factors to consider
Importance
? How important is the research topic within the larger research

and application community?
State of Knowledge
? What do we know already?

? What is the position of the research with respect to the
frontier?
Unique Competence
? Are you uniquely qualified to address this research?

? What is your secret weapon?
Interest
? How much does this research problem interest you personally?

? Do you have a passion for this problem?
Assessing importance
Audience
Who will care about the answer?

Impact
Will different answers change...

? ...what research gets done next?
? ...what is done by practitioners?
Longevity
How long will the answer be relevant and important?
It is fundamentally the confusion between effectiveness

and efficiency that stands between doing the right things
and doing things right. There is surely nothing quite so
useless as doing with great efficiency what should not be
done at all.
Peter Drucker
How many CS talks begin

Graph of rapid growth in...
? Processing power
? Heat generation
? Relative size or cost of caches
? Size of the Web
? Installed base of specific devices
? ....
Why?
The Frontier
Important work typically takes place at the frontier of a field.
You want your research findings to be relevant when they are
published, not just now.

The frontier in CS has two components:
? Changing face of computer science knowledge - a problem

common to all fields of science
? Changing environment of computer technology and

applications - a particularly challenging element of computer
science
I skate to where the puck is going to be, not where it has

been.
Wayne Gretzky
Methods for identifying frontiers

New observations or findings that need explanation.
New opportunities provided by instruments, methods of study,
or theoretical frameworks
Converging lines of research that combine to provide new
opportunities.
Emerging needs from outside the field (e.g., Web search,
bioinformatics, ecological monitoring, autonomous vehicles)

New questions or conjectures by researchers with good track
records.
Is the frontier special in CS?

Research groups often focus on very specific algorithms or
systems.
Frequently one system per research group.
But...
? ....Underlying concepts and fundamentals are the goal rather

than incidental algorithmic features.
? ...Exploratory work often converges.

? ...Core algorithms and principles emerge.
? ...Leaders in the field often agree on what work is central and
at the frontier .
The frontier is always difficult to identify early on.

systems.
But...


at the frontier .

systems.
But...


at the frontier .
Good research is done with a shovel, not with tweezers...

You should find an area where you can get a lot out of it
fast.
Roger Needham
Unique competence
What is it that makes you better able to address this research
question than others?

What is your secret weapon?
Poor answers
? I am smarter than other people

It is rarely true.
? I will work harder, longer, or faster

There is always someone who can do those things better than
you, and that weapon can cause unacceptable losses (harm
to family, friends, and personal happiness).
Unique competence
What is it that makes you better able to address this research
question than others?

What is your secret weapon?
Poor answers
? I am smarter than other people

It is rarely true.
? I will work harder, longer, or faster

There is always someone who can do those things better than
you, and that weapon can cause unacceptable losses (harm
to family, friends, and personal happiness).
Always have a secret weapon the biggest computer, a

problem imported from another field that others havent
heard of yet, a fact you stumbled on by being curious
about everything, a friend who is smarter than you are.
Anything, in short, that will give you an unfair advantage
in getting there first.
Herbert Simon
Write down what your or your groups secret weapon?
Always have a secret weapon the biggest computer, a

problem imported from another field that others havent
heard of yet, a fact you stumbled on by being curious
about everything, a friend who is smarter than you are.
Anything, in short, that will give you an unfair advantage
in getting there first.
Herbert Simon
Write down what your or your groups secret weapon?
Personal interest
Focus your work on areas of personal interest.
You will work harder, smarter, and more creatively.
Others will sense your strong interest and want to work with
you, support you work, and credit you with innovations.

Personal interest can be fostered and destroyed.
Track what affects your interest and use that knowledge

One widely shared factor: Speed of progress.
Review & Discussion of Tips in

The Incomplete Guide to the Art of Discovery,
Jack E. Oliver
Before we start
Book assumes that researchers want to make discoveries.
What are your motivations for completing a PhD?
? want to learn how to do research?

? want to make discoveries?
? want to understand something?
? want to tackle a certain problem?
? a fun job ?
? .....
Whatever they are, if you remain in research then youll have
a more successful and satisfying career if you are making

discoveries.
Strategies for Discovery

1. Dont Follow the Crowd
2. Rebel, but Wisely
3. Strive to Enhance Serendipity
4. Avoid Science Eddies
5. Study the Earth, and the Science of Geology
6. Seek the Non-questions
7. See Your Era in Long-Term Perspective
8. Go with Intuition
9. Avoid Sidetracking to Trivia
10. Be Competitive, Be a Winner, Be First
11. Argue by Analogy
12. Vision, Hypotheses, and Objective Testing
13. The Strategy of Exploration for Understanding
Strategies for Discovery

Pick and write down what your top 5
1. Dont Follow the Crowd
2. Rebel, but Wisely
3. Strive to Enhance Serendipity
4. Avoid Science Eddies
5. Study the Earth, and the Science of Geology
6. Seek the Non-questions
7. See Your Era in Long-Term Perspective
8. Go with Intuition
9. Avoid Sidetracking to Trivia
10. Be Competitive, Be a Winner, Be First
11. Argue by Analogy
12. Vision, Hypotheses, and Objective Testing
13. The Strategy of Exploration for Understanding
Questions
What strategies are missing from this list?
Which ones would you remove?
Why?
Do they apply to you as a PhD student? Why?
Questions
Have you seen these strategies in action by senior scientists in
your research field?

Which ones?
Questions
Have you got a long-term perspective of your field?
What were people doing a decade or 2 decades ago in your
field?
What was the stimulus for a big discovery in your field?
Tactics for Discovery

1. Adapt and Adopt Instruments and Technique
2. Skim the Cream
3. Minimize Jargon
4. Speak (Listen) to the Earth, and It Shall Teach Thee
5. Go for the Spatial Pattern
6. Skim the Rest of the Volume, Any Volume
7. Do It Yourself
8. The Knack of the Fresh Perspective
9. Choose Your Problem Very Carefully
10. The Curve of Discovery
11. Overcoming the Terminal Paper
Tactics for Discovery

Which of these tactics do you follow?
1. Adapt and Adopt Instruments and Technique

2. Skim the Cream
3. Minimize Jargon
4. Speak (Listen) to the Earth, and It Shall Teach Thee
5. Go for the Spatial Pattern
6. Skim the Rest of the Volume, Any Volume
7. Do It Yourself
8. The Knack of the Fresh Perspective
9. Choose Your Problem Very Carefully
10. The Curve of Discovery
11. Overcoming the Terminal Paper
Questions
What tactics do you or your research group follow that are
not on the list?

Write down examples where Instruments and Technique have
been adapted and adopted by those in your field.

What are the roots of the techniques you use in your research
field?
flattened. Judging whether a given set of observations is significant or important is not always easy. Sometimes, however, the
decision is obvious simply because of the large scale or broadQuestion
scope of the feature under study or because of its role in a larger
context.
t
Q
IME
Where do you think your research topic is on this curve?
Overcoming the 'Terminal Paper"
Questions
How do you keep a fresh perspective on your research?
Do you think it is beneficial to read broadly?
Do It Yourself . In your opinion, what does this mean w.r.t
your field.

About Cohesion

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

About Cohesion

Hochgeladen von

Copyright:

Verfügbare Formate

DD3001 - Research: Theory, Method, Practice

Hope DD3001 helps you avoid this situation

The Significance of Research

Am assuming you already avoid this

Computer Scientific Method

Change 1 parameter per

Discard if it doesnt support

Remember Science/Research is Hard

Why is science hard?

correctly update this knowledge.

Reason for Difficulty: Intrinsic

All possible theories

All possible theories

Theories we think are true

Much of what we consider today is probably wrong

All possible theories

Truth perhaps beyond our current imagination

All possible theories

Theories we think are true

Facts are simple in retrospect

I agree. But I wonder what it would

Why we need help doing this?

Reason for Difficulty: Personal

Personal failings: the human condition

know what was done or observed

Personal Failings: Self-deception

? Premature decision on an issue

? Expect more of other theories than of ones own

Not a new idea (Francis Bacon 1600s)

hypotheses in random data.

for accurate scientific communication.

systems can doom scientific investigations.

Personal failings: Willful deception

? Plagiarism or other unauthorized use of the work of others.

Source: Donald D. Jensen. Unpublished lecture notes, University of Nebraska-Lincoln. 1995.

Reason for Difficulty: Community

7 Fads and fashions

X Share data, results and theories

Havent scientists overcome these challenges?

what is already known, not how to learn what is currently

DD3001 - Research: Theory, Method, Practice

Answer to your questions

2. Do I have to attend all the lectures?

3. How can I access the reading material?

burgeoning research career.

non-linear. And requires hard work and perseverance!

Make you consider some of the responsibilities of being a

member of the research community.

Outline of the lecture topics

2. You and your code; Reproducible Research.

Papers and texts

? General advice to young researchers

? Advice to a Young Scientist by Peter Medawar.

? discussion of the pre-assigned readings,

What you will do in this course!

reading assignments to an acceptable level.

merit. See course website for details.

reading assignments to an acceptable level.

merit. See course website for details.

Remember you have to pass DD3001 to get your PhD.

the ones that are most relevant to you.

This course is a work in progress

journey exploring the issues raised.

How would you draw this curve?!"#$%&'()(+"&,-&(./0012(345&$(6(7&8393$(.:;,&(<$5("4(5,&(!"-#=-&>2