============================================================================ Adapt-NLP 2021 Reviews for Submission #22 ============================================================================ Title: An Empirical Study of Compound PCFGs Authors: Yanpeng Zhao ============================================================================ REVIEWER #1 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Appropriateness (1-5): 3 Clarity (1-5): 4 Originality / Innovativeness (1-5): 2 Soundness / Correctness (1-5): 3 Meaningful Comparison (1-5): 4 Thoroughness (1-5): 3 Impact of Ideas or Results (1-5): 2 Recommendation (1-5): 3 Reviewer Confidence (1-5): 4 Detailed Comments --------------------------------------------------------------------------- The paper analyzes the compound probabilistic context-free grammar (C-PCFG) model of Kim et al., 2019 on various experiments: reporting recall results on 6 different constituent labels, reporting F1 scores with varying maximum lengths of training sentences, conduct length generalization experiment by training on short spans, and testing on long spans, and finally conduct cross-lingual experiments from English to 9 different languages. The paper lacks innovation. The authors perform their experiments on an existing approach, the experiments and the findings are not really exciting, and lastly, there is only a slight connection to domain adaptation and generalization (the length generalization experiment and the cross-lingual setup). C-PFCG does not seem to generalize well to other languages (even though its average score is the highest among the existing baselines). I would expect the authors to suggest and show novel ways to improve C-PCFG ability to generalize to different languages and to consider other languages as the source domain. Besides that, the paper is well-written, the results are shown nicely, and the method is clearly described. --------------------------------------------------------------------------- Questions for Authors --------------------------------------------------------------------------- Can you think of additional analyses on the generalization abilities of C-PCFG to other languages that will shed light on its observed degradation? Can you think of novel ways to improve P-CFG's cross-lingual performance? --------------------------------------------------------------------------- ============================================================================ REVIEWER #2 ============================================================================ --------------------------------------------------------------------------- Reviewer's Scores --------------------------------------------------------------------------- Appropriateness (1-5): 4 Clarity (1-5): 5 Originality / Innovativeness (1-5): 3 Soundness / Correctness (1-5): 5 Meaningful Comparison (1-5): 5 Thoroughness (1-5): 4 Impact of Ideas or Results (1-5): 3 Recommendation (1-5): 4 Reviewer Confidence (1-5): 4 Detailed Comments --------------------------------------------------------------------------- This paper re-implements and presents in-depth analysis of compound probabilistic context-free grammars (C-PCFGs). In particular, - a break-down of recall on six frequent constituent labels (C-PCFs fall behind for VPs) - a study of whether C-PCFGs' performance is affected by sequence length (it is) - an exploration of C-PCFGs generalizability on unseen sentences and constituents (they generalize to sequences longer than 30 tokens if such instances are unseen) - a model ablation (sentence embeddings are most crucial for preterminal rules) - a multilingual evaluation (C-PCFGs seem to struggle with morphologically rich languages) Evidently, this paper presents a number of findings. Moreover, I don't see any problems with the evaluation methodology, so I recommend accepting the paper. ---------------------------------------------------------------------------