Results for English STS using BERT-mp

[zhujiatong628]

Hi, I'm reproducing your experiment recently, and I found when I evaluated BERT-mp for English STS. The results I got are much better than reported in the paper, so I also use your code provided at here to check. I found it's still better than paper shows. However, when I use your code to evaluate BERT-CLS for English STS, I can get the same results with the paper shows. So I was wondering is there something wrong with the results using BERT-mp for English STS?

[hardyqr]

Hi, if you check out

mirror-bert/evaluation/eval.py

Line 19 in 8b6b9de

parser.add_argument('--agg_mode', type=str, default="cls", help="{cls|mean|mean_std|...}")

, there are two options for mp: mean and mean_std where the first one considers padding tokens too. I used mean for reporting results which might have caused the discrepancy.

[zhujiatong628]

Thanks for your reply. I also used mean, so I'm so confused about the difference.😫