Outline

Learn from reviewers

• Part I: How to review a paper (Boi Faltings, 04)

• Part II: How to write a paper

Jia ZENG
Department of Computer Science
Hong Kong Baptist University • Part III: Case study

Dept. of Computer Science Dept. of Computer Science

1. How to review a paper Three reviewers

• Peer-reviewing: usually three reviewers judge • After submission, the associate editor (AE) will
– novelty assign your paper to three reviewers
– correctness
– significance
of your research results.
• At least in this world five guys will read your
paper: you, your boss, and three reviewers

• Currently peer-review is the only evaluation

mechanism. • Most of reviewers are your competitors (students,
postdoctoral fellows, etc.)

Dept. of Computer Science Dept. of Computer Science
 Types of reviews Conference reviewing
• Conferences: • Typical review load: 10 papers/reviewer
– one-shot
– accept/reject • Reading and understanding 10 papers takes 10
– few modifications hours of quality time

• Journals:
• Most reviewers do not have this time, so they
– iteration (major revision – minor revision – accept)
apply filtering and do not waste time on papers
– significant rewrites to improve quality
that are not acceptable anyway

Dept. of Computer Science Dept. of Computer Science

Filtering Hidden weaknesses of a paper

• Every paper must state (checklist):
– the problem addressed (novelty)
– the solution proposed (correctness)
– an example that shows how it works
– an evaluation, ideally a comparison with existing
techniques (completeness of experiments)
• Easy to check for initial filtering
• Many papers fail this test /
• Many authors are good salesman
– hiding assumptions
– using unrealistic examples
– comparing with old or wrong versions of existing work
– providing incorrect summaries of experimental results
• After initial filtering, the reviewers will focus on
finding hidden weaknesses of your paper

Dept. of Computer Science Dept. of Computer Science
 Reject Accept
• Yes/No questions: • Matter of degree:
– Is the paper complete (checklist) ? – Is the work novel? Are these just someone else’s
– Is the result correct? ideas in a different notion?
– Did you learn something from it? (difficult to satisfy) – Is the problem important?
– Is the work significant and difficult to obtain?
• If any of these is no, reject
• Yes, useful for ranking (weak/strong accept)

Dept. of Computer Science Dept. of Computer Science

Importance of comments Helpful comments

• Worst scenario for author: paper rejected, but • Example 1:
not clear why – This problem has been solved by many people years
ago
– This problem has been solved by A. Smith (AI journal,
• Helpful comments must justify 1992), with improvements by C. Miller (ECAI, 1999)
– why reject/accept the paper
• Example 2:
– How could the author improve it
– I do not think this solution works
– Listing typos, but not secondary
– On the following example, the method produces the
wrong results …; The proof of Theorem 3 is wrong,
and here is a counterexample …

Dept. of Computer Science Dept. of Computer Science
 Journal reviewing 2. How to write a paper
• Journals allow for iterations • Checklist
– Problem
• Same filters as for conferences, but important to – Solution
help author improve the paper – Example
– Evaluation
• Can expect significant rewrites/additional work
• Title, abstract, introduction, methodology, results,
• Most journal submissions are eventually discussion, conclusion
published somewhere

Dept. of Computer Science Dept. of Computer Science

Title Abstract
• Title (cover the most important information): • Abstract:
– New problem (new perspective) – Problem?
– New findings – Solution or methodology?
– New methodology – Results?
• Example:
• Example: – Higher-order structural dependencies such as
– Higher-order structural topic modeling in document networks coauthor relations exist widely in document networks.
(Zeng, 2010) However, their use in topic modeling problems is
– Physical signals for protein-DNA recognition (Zeng, 2009) significantly hampered in practice by the intractable
– Cascade Markov random fields for stroke extraction of Chinese complexity of modeling and optimizing the long-range
characters (Zeng, 2010) topic dependence. (Background)

Dept. of Computer Science Dept. of Computer Science
 Abstract
• Based on Markov random fields, we address the
higher-order structural topic modeling (HSTM)
problem with learned higher-order clique
potentials that capture complex topic structures.
We observe that higher-order topic structures
encountered in document networks are often
sparse, i.e., many topic configurations of a
higher-order clique are equally unlikely and thus
need not to be encouraged throughout the
network. (Problem)
Dept. of Computer Science
Abstract
• To achieve the approximate inference of the
best topic configuration, we develop a hybrid
belief propagation and Gibbs sampling algorithm
for HSTM. Experimental results on DBLP and
MEDLINE document datasets show that the
higher-order structural dependencies among
coauthors and documents can enhance the
overall topic summarization performance
compared with state-of-the-art topic models for
only pairwise relations. (Methodology + Results)
Dept. of Computer Science

Introduction Methodology
• Literature review • The key part of your paper:
– Why this problem is significant? – Theory
– Recent solutions? – Mathematical formulation
– What inspires this idea? – Algorithms
– Computational issues
• Deep understanding of your problem helps a lot – Implementation issues

• Present your idea • Try to package your method within a good

– Advantages?
theoretic framework (“why to do” is more
important than “how to do”)
– Disadvantages?

Dept. of Computer Science Dept. of Computer Science
 Results and discussion
• Data sets and performance measures
• Experimental settings
• Comparison with state-of-the-art methods
• When your method is better and when your
method does not work
Dept. of Computer Science
Reviewer 1
• Is there a particular reason that the author-topic
model is selected as the base model? How
would such a model behave on a more general
document network, such as a document citation
network and a document similarity network?
• How would this model scale when a large
dataset and a large number of topics in involved?
Dept. of Computer Science
3. Case study
• ICML 2010, Phase I, two reviewers, pass initial
filtering
• Reviewer 1:
– The author proposed a novel topic model that
incorporates higher-order relationship among
documents/authors. ……
– The author then proposed a new inference algorithm
which integrates belief propagation and Gibbs
sampling.
Dept. of Computer Science
Responses to reviewer 1
• This idea is also applicable to other types of
document networks. As stated in Sec 5, we think
one possible future work is to explore more
general citation networks by using LDA as base
model and the higher-order MRF for relational
modeling (Each document can be modeled by
LDA; and if A cites B and C, B also cites C, then
A, B, C forms a higher-order citation).
Dept. of Computer Science
 Responses to reviewer 1 (Continued)
• Generally, higher-order MRFs scale badly since
K^N topic configurations need to be considered.
But we find that in real document networks the
propagated higher-order topic patterns are
SPARSE in Fig 3. Sparsity not only reveals the
specificity of higher-order relations in topic
modeling, but also significantly reduces the
complexity of learning and inference (see Sec 3).
As a result, our model scales well with large
number of topics in practice.
Dept. of Computer Science
Reviewer 2 (Continued)
• I found the results slightly weak.
– The perplexity results (Fig 5) were indistinguihsable.
– For F-measure (Fig 6), I'm not sure if they compared
against state of the art of link prediction (e.g. Nallapati
KDD 2008).
– And for classification accuracy (Fig 7), one tends to
get much higher accuracies with, say, SVM, so SVM
results should appear as a baseline.
Dept. of Computer Science
Reviewer 2
• The model seemed quite similar to Daume's MRTF (both
in structure, and in results). Daume's model seemed to
have computational issues, so anything the current work
did to resolve that would be a plus.
• I didn't exactly follow the justification for the model in the
introduction. I would have been helped by a specific
example (in words) of where higher-order relations are
better than pair-wise relations.
Dept. of Computer Science
Responses to reviewer 2
• Our model fundamentally differs from MRTF in
three aspects:
– We formulate higher-order author relations with
learned arbitrary clique potentials from data rather
than "smoothness" priors, and show the effectiveness
of higher-order MRF in topic modeling
– We develop efficient inference and learning
algorithms based on BP and GS that reduces the
complexity in higher-order modeling by labeling
sparsity
– We design the LDA-AT model to handle the coauthor
document generating process
Dept. of Computer Science
 Responses to reviewer 2 (Continued)
• Fig 1 shows an example why higher-order
relations provide more specific constraints (since
simply using {y5,y1}, {y6,y1} to represent the
coauthor relation of {y1,y5,y6} cannot distinguish
with the situation of y5 and y6 separately
coauthored with y1 writing two different papers).
Dept. of Computer Science
Any questions?
Dept. of Computer Science
Responses to reviewer 2 (Continued)
• Fig 5 shows the log likelihood per word, where 0.1
corresponds to a significant improvement.
• Previous work (Chang 09) used the total log likelihood,
which scaled per word likelihood by the total number of
words. (Chang 09) shows an improved link prediction
than (Nallapati 08), so the comparison with (Chang 09) is
sufficient to demonstrate the improved link prediction.
• Because we compare the extracted topic features for
document classification with pairwise topic models, we
need not to show the SVM results using original word
vectors.
Dept. of Computer Science
Thank you!
Dept. of Computer Science