References

Our overview papers

2013 [Lab] Hanna Suominen, Sanna Salanterä, Sumithra Velupillai, Wendy W. Chapman, Guergana Savova, Noemie Elhadad, Danielle Mowery, Gareth J.F. Jones, Johannes Leveling, Liadh Kelly, Lorraine Goeuriot, David Martinez, Guido Zuccon (2013) – Overview of the ShARe/CLEF eHealth Evaluation Lab 2013. In Proceedings of CLEF 2013, Lecture Notes in Computer Science (LNCS), Springer. [Task] Sameer Pradhan, Noemie Elhadad, Brett R. South, David Martinez, Lee Christensen, Amy Vogel, Hanna Suominen, Wendy W. Chapman, Guergana Savova (2013) – Task 1: ShARe/CLEF eHealth Evaluation Lab 2013. In proceedings of CLEF 2013 (online working notes). [Task] Danielle L. Mowery, Brett R. South, Lee Christensen, Laura-Maria Murtola, Sanna Salantera, Hanna Suominen, David Martinez, Noemie Elhadad, Sameer Pradhan, Guergana Savova, Wendy W. Chapman (2013) – Task 2: ShARe/CLEF eHealth Evaluation Lab 2013. In proceedings of CLEF 2013 (online working notes). [Task] Lorraine Goeuriot, Gareth J. F. Jones, Liadh Kelly, Johannes Leveling, Allan Hanbury, Henning Müller, Sanna Salanterä, Hanna Suominen and Guido Zuccon (2013) – ShARe/CLEF eHealth Evaluation Lab 2013, Task 3: Information Retrieval to Address Patients? Questions when Reading Clinical Reports. In proceedings of CLEF 2013 (online working notes). [Special Issue] Suominen H. Guest editorial: Text mining and information analysis of health documents. Artificial Intelligence in Medicine 2014 61(3), 124-30.
2014 [Lab] Liadh Kelly, Lorraine Goeuriot, Hanna Suominen, Tobias Schreck, Gondy Leroy, Danielle L. Mowery, Sumithra Velupillai, Wendy W. Chapman, Guido Zuccon, and João Palotti (2014). Overview of the ShARe/CLEF eHealth Evaluation Lab 2014. In Proceedings of CLEF 2014. [Task] Hanna Suominen, Tobias Schreck, Gondy Leroy, Harry Hochheiser, Lorraine Goeuriot, Liadh Kelly, Danielle Mowery, Jaume Nualart, Gabriela Ferraro, Daniel Keim (2014), Task 1 of the CLEF eHealth Evaluation Lab 2014. In proceedings of CLEF 2014, p.1-30. [Task] Danielle Mowery, Sumithra Velupillai, Brett R South, Lee Christensen,David Martinez, Liadh Kelly, Lorraine Goeuriot, Noemie Elhadad, Sameer Pradhan, Guergana Savova, Wendy Chapman (20014). Task 2: ShARe/CLEF eHealth Evaluation Lab 2014. In proceedings of CLEF 2014, p.31-42. [Task] Lorraine Goeuriot, Liadh Kelly, Wei Li, Joao Palotti, Pavel Pecina, Guido Zuccon, Allan Hanbury, Gareth Jones, Henning Müller (2014). ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: User-centred Health Information Retrieval. In proceedings of CLEF 2014, p. 43-61
2015 [Lab] Lorraine Goeuriot, Liadh Kelly, Hanna Suominen, Leif Hanlen, Aurélie Névéol, Cyril Grouin, João Palotti, and Guido Zuccon (2015). Overview of the CLEF eHealth Evaluation Lab 2015. In Proceedings of CLEF 2015. [Task] Hanna Suominen, Leif Hanlen, Lorraine Goeuriot, Liadh Kelly, Gareth J. F. Jones (2015). Task 1a of the CLEF eHealth Evaluation Lab 2015: Clinical Speech Recognition. In CLEF 2015 Working notes. [Task] Aurélie Névéol, Cyril Grouin, Xavier Tannier, Thierry Hamon, Liadh Kelly, Lorraine Goeuriot, Pierre Zweigenbaum (2015). CLEF eHealth Evaluation Lab 2015 Task 1b: Clinical Named Entity Recognition. In CLEF 2015 Working notes. [Task] Joao Palotti, Guido Zuccon, Lorraine Goeuriot, Liadh Kelly, Allan Hanbury, Gareth Jones, Mihai Lupu, Pavel Pecina (2015). CLEF eHealth Evaluation Lab 2015, Task 2: Retrieving Information About Medical Symptoms. In CLEF 2015 Working notes.
2016 [Lab] L Kelly, L Goeuriot, H Suominen, A Névéol, J Palotti, G Zuccon (2016). Overview of the clef ehealth evaluation lab 2016. In Proceedings of CLEF 2016, Lecture Notes in Computer Science (LNCS), Springer. [Task] H Suominen, L Zhou, L Goeuriot, L Kelly (2016). Task 1 of the CLEF eHealth evaluation lab 2016: Handover information extraction. Proceedings of CLEF 2016 (online working notes). [Task] G Zuccon, J Palotti, L Goeuriot, L Kelly, M Lupu, P Pecina, H Mueller, J Budaher, A Deacon (2016). The ir task at the clef ehealth evaluation lab 2016: User-centred health information retrieval. Proceedings of CLEF 2016 (online working notes). [Task] A Névéol, L Goeuriot, L Kelly, K Cohen, C Grouin, T Hamon, T Lavergne, G Rey, A Robert, X Tannier, P Zweigenbaum (2016). Clinical information extraction at the clef ehealth evaluation lab 2016. Proceedings of CLEF 2016 (online working notes).
2017 [Lab] L Goeuriot, L Kelly, H Suominen, A Névéol, A Robert, E Kanoulas, R Spijker, J Palotti, G Zuccon (2017). CLEF 2017 eHealth Evaluation Lab Overview. 291-303 [Task] J Palotti, G Zuccon, Jimmy, P Pecina, M Lupu, L Goeuriot, L Kelly, A Hanbury (2017) CLEF 2017 Task Overview: The IR Task at the eHealth Evaluation Lab – Evaluating Retrieval Methods for Consumer Health Search. CLEF (Working Notes) 2017 [Task] A Névéol, A Robert, R Anderson, K Bretonnel Cohen, C Grouin, T Lavergne, G Rey, C Rondet, P Zweigenbaum: CLEF eHealth 2017 Multilingual Information Extraction task Overview: ICD10 Coding of Death Certificates in English and French. [Task] E Kanoulas, D Li, L Azzopardi, R Spijker: CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview.
2018 [Lab] H Suominen, L Kelly, L Goeuriot, A Névéol, L Ramadier, A Robert, E Kanoulas, R Spijker, L Azzopardi, D Li, Jimmy, J Palotti, G Zuccon (2018) Overview of the CLEF eHealth Evaluation Lab 2018. 286-301 [Task] Jimmy, G Zuccon, J Palotti, L Goeuriot, L Kelly: Overview of the CLEF 2018 Consumer Health Search Task. CLEF (Working Notes) 2018 [Task] A Névéol, A Robert, F Grippo, C Morgand, C Orsi, L Pelikan, L Ramadier, G Rey, P Zweigenbaum (2018) CLEF eHealth 2018 Multilingual Information Extraction Task Overview: ICD10 Coding of Death Certificates in French, Hungarian and Italian. [Task] E Kanoulas, D Li, L Azzopardi, R Spijker (2018) CLEF 2018 Technologically Assisted Reviews in Empirical Medicine Overview.

Participants paper

2013 Working notes: http://ceur-ws.org/Vol-1179/
2014 Working notes: http://ceur-ws.org/Vol-1180/
2015 Working notes: http://ceur-ws.org/Vol-1391/
2016 Working notes: http://ceur-ws.org/Vol-1609/ 
2017 Working notes: http://ceur-ws.org/Vol-1866/ 
2018 Working notes: http://ceur-ws.org/Vol-2125/ 

Papers published using our datasets

2014
  • Pradhan S, Elhadad N, South BR, Martinez D, Christensen L, Vogel A, Suominen H, Chapman WW, Savova G. Evaluating the state of the art in disorder recognition and normalization of clinical narrative. Journal of the American Medical Informatics Association (JAMIA) 2014 22(1), pp. 143–154. (PDF)
  • Lorraine Goeuriot, Liadh Kelly, Johannes Leveling (2014). An Analysis of Query Difficulty for Information Retrieval in the Medical Domain. In SIGIR 2014 short papers.
  • Kholghi, M., Sitbon, L., Zuccon, G., & Nguyen, A. (2014). Factors influencing robustness and effectiveness of conditional random fields in active learning frameworks.
  • Zuccon, G., & Koopman, B. (2014). Integrating understandability in the evaluation of consumer health search engines.
2015
  • Suominen H, Zhou L, Hanlen L, Ferraro G. Benchmarking clinical speech recognition and information extraction: New data, methods and evaluations. JMIR Medical Informatics 2015 3(2), e19. (PDF)
  • Zhou L and Suominen H. Information extraction to improve standard compliance: The case of clinical handover. Proceedings of the 28th Australasian Joint Conference on Artificial Intelligence 2015 (AI2015), Canberra, ACT, Australia. (PDF)
  • Zhou L, Suominen H, Hanlen L. Evaluation data and benchmarks for cascaded speech recognition and entity extraction. Proceedings of the ACM Multimedia 2015 Workshop on Speech, Language and Audio in Multimedia, Brisbane, QLD, Australia, October, 2015. (PDF)
  • Palotti, J. R. D. M., Zuccon, G., & Hanbury, A. (2015, October). The influence of pre-processing on the estimation of readability of web documents. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (pp. 1763-1766). ACM.
  • Kholghi, M., Sitbon, L., Zuccon, G., & Nguyen, A. (2015, October). External knowledge and query strategies in active learning: A study in clinical information extraction. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (pp. 143-152). ACM.
  • De Vine, L., Kholghi, M., Zuccon, G., Sitbon, L., & Nguyen, A. (2015). Analysis of word embeddings and sequence features for clinical information extraction.
  • Kholghi, M., Sitbon, L., Zuccon, G., & Nguyen, A. (2015). Active learning: a step towards automating medical concept extraction. Journal of the American Medical Informatics Association.
  • Zuccon, G., Koopman, B., & Palotti, J. (2015, March). Diagnose this if you can. In European Conference on Information Retrieval (pp. 562-567). Springer International Publishing.
2016
  • J Palotti, G Zuccon, J Bernhardt, A Hanbury, L Goeuriot (2016). Assessors Agreement: A Case Study Across Assessor Type, Payment Levels, Query Variations and Relevance Dimensions. Proceedings of CLEF 2016, Lecture Notes in Computer Science (LNCS), Springer.
  • J Palotti, L Goeuriot, G Zuccon, A Hanbury (2016). Ranking health web pages with relevance and understandability. In Proceedings of the 39th International ACM SIGIR conference.
  • A Beloborodov and L Goeuriot (2016). Improving Health Consumer Search with Contextual Information. In Proceedings of the 2nd SIGIR workshop on Medical Information Retrieval (MedIR).
  • L Goeuriot, L Kelly, G Zuccon and J Palotti (2016). Building Evaluation Datasets for Consumer-Oriented Information Retrieval. In Proceedings of Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016).
  • Rekabsaz, N., Lupu, M., Hanbury, A., & Zuccon, G. (2016). Generalizing translation models in the probabilistic relevance framework. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (pp. 711-720). ACM.
  • Zuccon, G., Palotti, J., & Hanbury, A. (2016). Query Variations and their Effect on Comparing Information Retrieval Systems. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (pp. 691-700). ACM.
  • Zuccon, G. (2016, March). Understandability biased evaluation for information retrieval. In European Conference on Information Retrieval (pp. 280-292). Springer International Publishing.

<this page is currently under construction – to be updated with content from: https://sites.google.com/site/clefehealth/references>