Artificial Intelligence (AI) systems using symptoms/signs to detect respiratory diseases may improve diagnosis especially in limited resource settings. Heterogeneity in such AI systems creates an ongoing need to analyse performance to inform future research. This systematic literature review aimed to investigate performance and reporting of diagnostic AI systems using machine learning (ML) for pneumonia detection based on symptoms and signs, and to provide recommendations on best practices for

more »

... esigning and implementing predictive ML algorithms. This article was conducted following the PRISMA protocol, 876 articles were identified by searching PubMed, Scopus, and Ovid SP databases (last search 5th May 2021). For inclusion, studies must have differentiated clinically diagnosed pneumonia from controls or other diseases using AI. Risk of Bias was evaluated using The STARD 2015 tool. Information was extracted from 16 included studies regarding study characteristics, MLmodel features, reference tests, study population, accuracy measures and ethical aspects. All included studies were highly heterogenous concerning the study design, setting of diagnosis, study population and ML algorithm. Study reporting quality in methodology and results was low. Ethical issues surrounding design and implementation of the AI algorithms were not well explored. Although no single performance measure was used in all studies, most reported an accuracy measure over 90%. There is strong evidence to support further investigations of ML to automatically detect pneumonia based on easily recognisable symptoms and signs. To help improve the efficacy of future research, recommendations for designing and implementing AI tools based on the findings of this study are provided.