May 22nd, 2020
This paper has been criticized for (a) not sufficiently adjusting for pre-treatment covariates, and (b) not giving enough information on data and methods. See here:
and here:
https://statmodeling.stat.columbia.edu/2020/05/25/hydroxychloroquine-update/
It would be good to see the authors' reply to these criticism and also to see the (de-identified) data. I say this in general terms without having studied this particular article in detail.
The Hypertension high value in control group is unexpected considering that the PSM group is N=3783 out of a N=81144 raw group.
How couldn’t the authors possibly achieve to match this very critical factor when they have a n=81144 group at their disposal?
This has the effect of worsening the control group, (as hypertension is a critical risk factor for severe COVID).
Had the control group been properly matched, the PSM control group would exhibit better outcomes and the discrepancy with the HCQ group would be even more striking.
The article reports 4,402 patients from Africa as at 14th April 2020, which looks rather high, given that by that date, according to the Africa CDC, there were a total of 15,738 cases reported among the 52 countries in Africa. The majority of cases are asymptomatic and not hospitalized, so the pool of patients available to join this study would have been even smaller than the 15k. Would be good to know which countries and hospitals contributed data but Appendix Table 1 does not list the countries from which they recruited.
Another confounding is the exclusion criteria. Fig. 1 shows that 1102 patients were excluded because they received CQ derivatives after intubation.
This doesn't look like a serious problem, but it is not a negligible number if we compare with the total number of patients who received HCQ before intubation (1243+616+814+403 = 3067). 26% intubated patients were excluded!
This may sound nothing, but it may significantly imbalance the baseline between treatment and control group, because no one is excluded in the control group. For example, the observation can be completely changed if:
selection bias towards those who are strong & young who can tolerate nasal feeding tube (intubation prohibits swallowing HCQ pills). --> treatment group is biased towards the elder after excluding youngsters.
selection bias towards those who looks hopeless, so doctor was hoping that HCQ can save his/her life. --> treatment group is biased towards the healthier after excluding the serious ones.
HCQ is a game-changer (or toxic drug) especially when a patient is intubated, compared to before intubation. The current exclusion criteria will worsen (or inflates) the observed survival, because it will selectively exclude the intubation-then-survive (or intubation-then-death) in the treatment group but not in the control group.
There is no mention of any minimal duration of treatment in the inclusion criteria.
If a patient in deteriorating condition is given a pill before right before being intubated, then he is included and the ventilation may falsely be accounted as an outcome of the treatment.
There seem to be two issues with the race/ethnicity data.
First, many hospitals do not collect such data. In France or Germany it would simply be illegal to even try.
Second, seems very unlikely that any hospital outside the US would use the categories "White", "Black", "Asian", and "Hispanic". (Even in the US, "Hispanic" is typically --- e.g., by the CDC --- regarded as an "ethnicity" within "White".)
Did the authors just assume that everyone in Europe and Australia was "White", everyone in South American "Hispanic", everyone in Asia "Asian", and everyone in Africa "Black"? Nothing is reported about this. I'm not sure how valid it is to class everyone in Brazil as "Hispanic", everyone in Africa as Black, and everyone from Ankara to Vladivostok to Tokyo to Mumbai as "Asian".
Also noteworthy that 1102 / (14888+1102) = 6,9% of patients who were given HCQ or CQ were intubated in the first 48h after diagnostic... since the 1102 and the 852 are mutually exclusive here (the total 276 + 1102 + 852 equals the total of excluded).
It's also very unlikely that no patient from the 852 was intubated. (CQ after 48h then intubated). So the number of intubated patient to consider in your calculation is probably a bit higher than 1102.
The article states that 73 patients died in 5 Australian hospitals (April 14th). At this date, government data states that there is 61 deceased for the whole country (and the majority in retirement houses) : https://www.health.gov.au/resources/publications/coronavirus-covid-19-at-a-glance-14-april-2020
If we look at this Australian government report (Table 5 of https://www1.health.gov.au/internet/main/publishing.nsf/Content/1D03BCB527F40C8BCA258503000302EB/$File/covid_19_australia_epidemiology_report_15_reporting_week_ending_23_59_aest_10_may_2020.pdf ), on May 10th, only 866 patients (in total) had been hospitalized in Australia, of which 7.9% died (68 patients)…
In this Lancet article, the data included patients hospitalized between December 20 and April 14. Five australian hospitals were included, according to its appendix (https://www.thelancet.com/cms/10.1016/S0140-6736(20)31180-6/attachment/84423d57-4cf8-41d0-99ca-0e921f2c80ce/mmc1.pdf) and mortality was recorded as 73 (12%), which, I suppose, means N = 73, or 12% of the 609 in Australia.
Can you explain this discrepancy ? Regards.
The mortality data for Australia was questionable, as per #11 Rhithrogena Tetrapunctigera's comment. The claim that they had waiver of ethics approval and the need to acknowledge sources of data was also questionable, if they were to publish Australian data.
The paper did, however, specified under Results that "96032 hospitalised patients from 671 hospitals were diagnosed with COVID-19 between Dec 20, 2019, and April 14, 2020 and met the inclusion criteria for this study (figure 1). All included patients completed their hospital course (discharged or died) by April 21, 2020". So the outcome data was up to April 21. This was pointed out by Carlos Ungil on # Andrew Gelman's comment's 2nd link.
Nevertheless, total death in Australia by 3pm Apr 21st was 71:28 in the state of NSW, 15 in Victoria, 8 in Tasmania, 7 in Western Australia, 6 in Queensland and 4 in South Australia. Then they have captured more than 100% of the mortality data we have in Australia. COVID-19 is a notifiable disease in Australia so it's very unlikely to have captured COVID-19 mortality data in a hospital in Australia that was not reflected on the official data. https://www.health.gov.au/sites/default/files/documents/2020/04/coronavirus-covid-19-at-a-glance-21-april-2020.pdf
If taking into account the time difference between US and Australia, as pointed out by my colleagues John and Tim, Apr 21st would be Apr 22nd in Australia. Total death in Australia by 3pm Apr 22nd was 74: 31 in the state of NSW, 15 in Victoria, 8 in Tasmania, 7 in Western Australia, 6 in Queensland and 4 in South Australia. https://www.health.gov.au/sites/default/files/documents/2020/04/coronavirus-covid-19-at-a-glance-22-april-2020.pdf
The paper reported in Australia there were 73 deaths (Table S3) across 5 Hospital (Table S1). This was merely impossible. They would have needed to have 1 hospital in each state to contribute the data to them, and each hospital in the state would have captured all of the mortality within that hospital. If you track the COVID-19 death in the state of NSW, from Mar 3rd (1st death reported) to Apr 22nd on the NSW Health website, , there were:
Australian deaths scattered around many hospitals and nursing home clusters. It's merely impossible to have captured 73 deaths within 5 hospitals.
If one includes Australia, New Zealand, West Papua and Papua New Guinea under "Australia" as a continent. There were 12 total deaths in New Zealand reported on Apr 20th, one more on Apr 21st, and one more on Apr 22nd: totalling 14 deaths by Apr 22nd. https://www.health.govt.nz/news-media/media-releases/9-new-cases-covid-19-0 https://www.health.govt.nz/news-media/media-releases/5-new-cases-covid-19-2 https://www.health.govt.nz/news-media/media-releases/6-new-cases-covid-19 There were 2 reported death as of May 29th in West Papua http://dinkes.papuabaratprov.go.id/ There were 0 reported death as of May 29th in Papua New Guinea. https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6
If one captured all the deaths in 1 hospital in New Zealand (also impossible). It's still impossible to capture 61 deaths in 4 hospitals in Australia. If one included 1 hospital in New Zealand and 1 in West Papua, it's still not possible to capture 59 deaths in 3 hospitals in Australia... etc.
The second issue is the claim of not needing ethics approval or the fact that they did not acknowledge the source of data in Australia. Under the Australian National Statement on Ethical Conduct in Human Research (section 3), secondary use and sharing of data or information is allowed. However, all data collections should have an identified custodian. There are requirements from the data custodian (presumably the 5 Australian hospitals) to the researchers wishing to share their data. Often this would involve asking for local ethics approval or waiver of ethics approval. If for any reason this was not required, the National Statement requires the researchers to demonstrate respect for participants by, amongst other things, "acknowledging the source of data or information in publications". The Australian data custodian would be bounded by this, which would translate to the researchers wishing to obtain data from the 5 Australian hospitals. https://www.nhmrc.gov.au/about-us/publications/national-statement-ethical-conduct-human-research-2007-updated-2018
The authors should address the questionable mortality data from Australia, and address their waiver of the need for acknowledging the source of data (at least for the Australian hospital part) and their waiver of the need for ethics approval. If these were the agreements between them and the 5 Australian hospitals, Lancet should consider assessing the evidence of that.
A major issue with the analyses is taht there are not stratified on countries, while the baseline risk of mortality may be quite different in the different countries, and maybe they compared mortality in treated patients in countries with high level of mortality to untreated patients in countries with low level of mortality. If that's the case it would introduce a huge difference. Running a stratified analysis is of upmost importance.
Newer version of the Open Letter: https://zenodo.org/record/3864691
Drs. Mehra, Desai, Ruschizka, and Patel,
Firstly, thank you in advance for fielding these questions. Best,
Regarding arrhythmias:
• Of the 502 patients who experienced denovo arrhythmias, how many died who were also in the HCQ with a macrolide group?
• How many of the 1479 nonsurvivors in the HCQ with a macrolide group (N= 6221), died from ventricular arrhythmia?
Regarding baseline disease severity: The best pulse oximetry threshold for detecting hypoxia is 92%. (see https://www.sciencedirect.com/science/article/abs/pii/S073567570051732X ) How many of the non-survivors (N=10,698) had a baseline oxygen saturation threshold of 92%? Adjusting the Pulse oximeter saturations (SpO2) to <93% would differentiate the sick from the very sick.
Regarding survival rates for HCQ with a macrolide:
• 76% of patients who took HCQ with a macrolide survived (4,742 out of 6,221)
• 5.6% of all survivors took HCQ with a macrolide (4,742 out of 85,334) Might these findings indicate further investigation into HCQ with a macrolide in an outpatient, ambulatory setting?
In the appendix the hazard ratio for diabetes by region is strange. For America and Europe, the Hazard ratio is bigger than one (1.305 and 1.151) but for South America, Africa and Australia under one (0.744, 0.769 and 0.897). This does not make any sense. This suggests mistakes in the model or in the input data.
An quick correction has been pushed. Basically, 546 patients were incorrectly registered to Australia therefore moved from Australia to Asia. The authors also redid the analysis correspondingly. https://www.thelancet.com/lancet/article/s0140-6736(20)31249-6
However, proportional hazards model for in-hospital mortality for Asia (Table S4e) has a big update.
Before correction
Asia: SpO2 HR=1.66 (CI 1.37-2.01) N=7555
Aus: SpO2 HR=1.63 (CI 1.36-1.95) N=609
After correction
Asia: SpO2 HR=0.92 (CI 0.73-1.17) N=8101
How can an important covariate like SpO2 get such a big change after merging two datasets in which SpO2 are both significant? Is this statistically sound? Can this indicate their Cox model is not robust enough to tolerate a small change (7555->8101, +8%)?
The correction made to the on-line paper does not correct the issues about comparing treated patients to untreated patients from different countries with potentially different background level of mortality. The observed results could be explained at least in part by this simple fact. Reviewers should have asked for a country stratified analysis, as usual in multinational studies. Providing such an analysis is needed
Open letter to MR Mehra, SS Desai, F Ruschitzka, and AN Patel, authors of “Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis”. Lancet. 2020 May 22:S0140-6736(20)31180-6. doi: 10.1016/S0140-6736(20)31180-6. PMID: 32450107
and to Richard Horton (editor of The Lancet).
Concerns regarding the statistical analysis and data integrity
The retrospective, observational study of 96,032 hospitalized COVID-19 patients from six continents reported substantially increased mortality (~30% excess deaths) and occurrence of cardiac arrhythmias associated with the use of the 4-aminoquinoline drugs hydroxychloroquine and chloroquine. These results have had a considerable impact on public health practice and research.
The WHO has paused recruitment to the hydroxychloroquine arm in their SOLIDARITY trial. The UK regulatory body, MHRA, requested the temporary pausing of recruitment into all hydroxychloroquine trials in the UK (treatment and prevention), and France has changed its national recommendation for the use of hydroxychloroquine in COVID-19 treatment and also halted trials.
The subsequent media headlines have caused considerable concern to participants and patients enrolled in randomized controlled trials (RCTs) seeking to characterize the potential benefits and risks of these drugs in the treatment and prevention of COVID-19 infections. There is uniform agreement that well conducted RCTs are needed to inform policies and practices.
This impact has led many researchers around the world to scrutinize in detail the publication in question. This scrutiny has raised both methodological and data integrity concerns. The main concerns are listed as follows:
The patient data were obtained through electronic health records, supply chain databases, and financial records. The data are held by the US company Surgisphere. In response to a request for the data Professor Mehra replied: “Our data sharing agreements with the various governments, countries and hospitals do not allow us to share data unfortunately.”
Given the enormous importance and influence of these results, we believe it is imperative that:
In the interests of transparency, we also ask The Lancet to make openly available the peer review comments that led to this manuscript being accepted for publication.
This open letter is signed by clinicians, medical researchers, statisticians, and ethicists from across the world. The full list of signatories and affiliations can be found below.
List of Signatories
Dr James Watson (Statistician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Amanda Adler (Triallist & Clinician, Director of the Diabetes Trials Unit, University of Oxford, UK)
Dr Ambrose Agweyu (Medical researcher, KEMRI-Wellcome Trust Research Programme, Kenya)
Professor Dani Prieto-Alhambra (Epidemiologist, University of Oxford, UK)
Dr Ravi Amaravadi (Researcher, University of Pennsylvania, USA)
Professor Juan-Manuel Anaya (Clinician, Universidad del Rosario, Colombia)
Professor Nicholas Anstey (Clinician, Menzies School of Health Research, Australia)
Professor Yaseen Arabi (Clinician and researcher, King Saud Bin Abdulaziz University for Health Sciences, Saudi Arabia)
Dr Elizabeth Ashley (Clinician, Director of the Lao-Oxford-Mahosot Hospital-Wellcome Trust Research Unit, Laos)
Professor Michael Avidan (Clinician, Washington University in St Louis, USA)
Professor Kevin Baird (Researcher, Head of the Eijkman-Oxford Clinical Research Unit, Indonesia)
Professor Francois Balloux (Researcher, Director of the UCL Genetics Institute, UK)
Dr Clifford George Banda (Clinician, University of Cape Town, South Africa)
Dr Edwine Barasa (Health economist, KEMRI-Wellcome Trust Research Programme, Kenya)
Dr Ruanne Barnabas (Physician Scientist, University of Washington, USA)
Professor Karen Barnes (Clinical Pharmacology, University of Cape Town, South Africa)
Professor Enrico Bucci (Systems Biologist, Temple University, USA)
Professor Buddha Basnyat (Clinician, Head of the Oxford University Clinical Research Unit - Nepal, Nepal)
Professor Philip Bejon (Medical researcher, Director of the KEMRI-Wellcome Trust Research Programme, Kenya)
Professor Mohammad Asim Beg (Clinician/Researcher, Aga Khan University, Pakistan)
Prof. Linda-Gail Bekker (Clinician, University of Cape Town, South Africa)
Professor Leïla Belkhir (Clinician, Université Catholique de Louvain, Belgium)
Mr Mostapha Benhenda (Data scientist, Melwy and COVIND Covid-19 Individual Patient Data Consortium, France)
Professor Marc Bonten (Clinician/Researcher, University Medical Center Utrecht, The Netherlands)
Professor Bjug Borgundvaag (Clinician, Director of the Schwartz/Reisman Emergency Medicine Institute, Canada)
Professor Emmanuel Bottieau (Clinician, Institute of Tropical Medicine, Antwerp, Belgium)
Professor David Boulware (Researcher & Triallist, University of Minnesota, USA)
Professor Anders Björkman (Clinician, Karolinska Insitutet, Sweden)
Dr Sabine Braat (Statistician, University of Melbourne, Australia)
Professor Frank Brunkhorst (Clinician & Director of the Center of Clinical Studies, Jena University Hospital, Germany)
Professor James Brophy (Clinician/Epidemiologist, McGill University, Canada)
Professor Caroline Buckee (Epidemiologist, Harvard TH Chan School of Public Health, USA)
Dr James Callery (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Todd Campbell Lee (Researcher, McGill University, Canada)
Professor Adrienne Chan, MD MPH FRCPC (Researcher, University of Toronto, Canada)
Professor John Carlin (Statistician, University of Melbourne & Murdoch Children’s Research Institute, Australia)
Dr Nomathemba Chandiwana (Research Clinician, University of the Witwatersrand, South Africa)
Dr Arjun Chandna (Clinician, Cambodia Oxford Medical Research Unit, Cambodia)
Professor Phaik Yeong Cheah (Ethicist/Pharmacist, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Allen Cheng (Clinician, Monash University, Australia)
Professor Ivy Cheng (Clinician/Researcher, University of Toronto, Canada)
Professor Kesinee Chotivanich (Researcher, Mahidol University, Thailand)
Professor Leonid Churilov (Statistician, University of Melbourne, Australia)
Professor Ben Cooper (Epidemiologist, University of Oxford, UK)
Dr Cintia Cruz (Paediatrician Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Bart Currie (Director, HOT NORTH, Menzies School of Health Research, Australia)
Professor Joshua Davis (Clinician, President of the Australasian Society for Infectious Diseases, Australia)
Professor Jeremy Day (Clinician, Oxford University Clinical Research Unit, Vietnam)
Professor Nicholas Day (Clinician, Director of the Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Hakim-Moulay Dehbi (Statistician, University College London, UK)
Professor Justin Denholm (Clinician, Researcher, Ethicist, Doherty Institute, Australia)
Dr Lennie Derde (Intensivist/Researcher, University Medical Center Utrecht, The Netherlands)
Professor Keertan Dheda (Clinician/Researcher, University of Cape Town, & Groote Schuur Hospital, South Africa)
Dr Mehul Dhorda (Clinical Researcher, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Annane Djillali (Dean of the School of Medicine, Simone Veil Université, France)
Professor Arjen Dondorp (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Joseph Doyle (Clinician, Monash University and Burnet Institute, Australia)
Professor Emeritus Jean Dupouy-Camet (Former president of the European Federation of Parasitologists, Paris University, France)
Dr Anthony Etyang (Medical Researcher, KEMRI-Wellcome Trust Research Programme, Kenya)
Dr Caterina Fanello (Epidemiologist, University of Oxford, UK)
Professor Neil Ferguson (Epidemiologist, Imperial College London, UK)
Dr Ricard Ferrer (Head of Department of Intensive Care, Hospital Universitari Vall d’Hebron, Spain)
Professor Andrew Forbes (Statistician, Monash University, Melbourne, Australia)
Professor Oumar Gaye (Clinical Researcher, University Cheikh Anta Diop, Senegal)
Dr Ronald Geskus (Head of Statistics at the Oxford University Clinical Research Unit, Vietnam)
Dr Mellie Gilder (Clinician/Researcher, Chiang Mai University, Thailand)
Professor Emeritus Richard Gill (Mathematician/Statistician, Former President of Dutch Statistical Society, The Netherlands)
Professor Dave Glidden (Biostatistics, University of California, USA)
Professor Azra Ghani (Epidemiologist, Imperial College London, UK)
Prof Philippe Guerin (Medical researcher, University of Oxford, UK)
Dr. Raph Hamers (Clinician/Triallist, Eijkman-Oxford Clinical Research Unit, Indonesia)
Dr Rashan Haniffa (Clinician/Researcher, NICST, Sri Lanka)
Professor Stephane Heritier (Statistician, Monash University, Australia)
Dr Thomas Hiemstra (Triallist, University of Cambridge, UK)
Dr. Risa Hoffman (Clinician/Clinical Researcher, University of California, USA)
Professor Peter Horby (Clinical Researcher, Centre for Tropical Medicine and Global Health, University of Oxford)
Dr Sybil Hosek (Clinical Researcher, Cook County Health, USA)
Dr Jens-Ulrik Jensen (Clinician/Triallist, University of Copenhagen, Denmark)
Dr Hilary Johnstone (Clinical Research Physician, Independent)
Professor Christine Johnston (Clinical Researcher, University of Washington School of Medicine, USA)
Professor Peter Jüni (Director of the Applied Health Research Centre, University of Toronto, Canada)
Professor Kevin Kain (Clinical Researcher, University of Toronto, Canada)
Dr Sharon Kaur (Ethicist, University of Malaya, Malaysia)
Dr Evelyne Kestelyn (Head of Clinical Trials, Oxford University Clinical Research Unit, Vietnam)
Dr Patricia Kissinger (Clinical Researcher, Tulane University, USA)
Professor Megan Landes (Clinician/Researcher, University of Toronto, Canada)
Dr Tan Le Van (Medical Researcher, Oxford University Clinical Research Unit, Vietnam)
Professor Katherine Lee (Statistician, University of Melbourne, Australia)
Professor Laurence Lovat (Clinical Director of Wellcome EPSRC Centre for Interventional & Surgical Sciences, UCL, UK)
Dr Nsobya Samuel Lubwama (Researcher, Makerere University College of Health Sciences, Uganda)
Professor Kathryn Maitland (Clinician, Imperial College London/KEMRI Wellcome Trust Programme, Kenya)
Dr Julie Marsh (Statistician, Telethon Kids Institute, Australia)
Professor John Marshall (Clinician/Researcher, University of Toronto, Canada)
Professor Gary Maartens (Clinician, University of Cape Town, South Africa)
DR Ignacio Martin-Loeches (Clinician, Trinity College Dublin, Ireland)
Professor Richard Maude (Clinician/Epidemiologist, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Mayfong Mayxay (Clinician/Researcher, Lao-Oxford-Mahosot Hospital-Wellcome Trust Research Unit, Laos)
Dr Colin McArthur (Clinician/Triallist, Auckland City Hospital and Monash University)
Dr Emily McDonald (Clinician/Researcher, McGill University Health Center, Canada)
Professor Rose McGready (Clinician/Researcher, Shoklo Malaria Research Unit, Thailand)
Dr Alistair McLean (Medical researcher, University of Oxford, UK)
Professor Shelley McLeod (Clinical Epidemiologist, University of Toronto, Canada)
Dr John McKinnon (Clinician/Researcher, Henry Ford Health System, USA)
Dr Bryan McVerry (Medical researcher, University of Pittsburgh, USA)
Laura Merson (Clinical researcher, University of Oxford, UK)
Professor Clara Menendez (Clinical Researcher, Barcelona University, Spain)
Professor William Meurer (Clinician/Medical researcher, University of Michigan, USA)
Professor Ramani Moonesinghe (Clinician researcher, University College London, UK)
Dr Kerryn Moore (Epidemiologist, London School of Hygiene and Tropical Medicine, UK)
Dr Rephaim Mpofu (Clinician, University of Cape Town, South Africa)
Dr Mavuto Mukaka (Statistician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Srinivas Murthy (Clinical Researcher, University of British Columbia, Canada)
Professor Kim Mulholland (Clinician, London School of Hygiene & Tropical Medicine, UK)
Professor Daniel Neafsey (Researcher, Harvard T.H. Chan School of Public Health, USA)
Professor Paul Newton (Clinician, University Oxford, UK)
Professor Alistair Nichol (Clinician Researcher, University College Dublin, Ireland, & Monash University, Australia)
Professor Francois Nosten (Clinician, Director of the Shoklo Malaria Research Unit, Thailand)
Dr Matthew O’Sullivan (Clinician, Westmead Hospital & University of Sydney, Australia)
Professor Piero Olliaro (Clinical Researcher, University of Oxford, UK)
Dr William O’Neill (Clinician/Researcher, Henry Ford Health System, USA)
Professor Marie Onyamboko (Clinical researcher, Kinshasa School of Public Health, DRC)
Dr Marcin Osuchowski (Medical researcher, Ludwig Boltzmann Institute, Austria)
Professor Catherine Orrell (Clinical Pharmacologist, University of Cape Town, South Africa)
Professor Jean Bosco Ouedraogo (Medical Researcher, WWARN, Burkina Faso)
Dr Temitope Oyedele (Clinician, Cook County Health, USA)
Dr Michael Paasche-Orlow (Clinical Researcher, Boston University, USA
Elaine Pascoe (Statistician, University of Queensland, Australia)
Professor Michael Parker (Director of The Wellcome Centre for Ethics and Humanities, The Ethox Centre, University of Oxford, UK)
Professor David Paterson (Clinician, Director, UQ Centre for Clinical Research, Australia)
Dr Kajaal Patel (Paediatrician, Cambodia Oxford Medical Research Unit, Cambodia)
Tom Parke (Statistician, Berry Consultants, UK)
Professor Philippe Parola (Researcher, Aix-Marseille University, France)
Professor Weerapong Phumratanaprapin (Deputy Dean of the Faculty of Tropical Medicine, Mahidol University, Thailand)
Professor Pedro Politi (Oncologist, University of Buenos Aires, Argentina)
Professor William Powderly (Director, Institute of Clinical and Translational Research, Washington University in St. Louis, USA)
Dr Christophe Pouzat (Mathematician, CNRS, France)
Dr David Price (Statistician, Doherty Institute & University of Melbourne, Australia)
Professor Richard Price (Clinician, Menzies School of Health Research, Australia)
Professor Sasithon Pukrittayakamee (Clinician, Mahidol University, Thailand)
Dr Aung Pyae Phyo (Clinician/Scientist, Myanmar Oxford Clinical Research Unit, Myanmar)
Dr Ben Saville (Statistician, Berry Consultants & Vanderbilt University)
Professor Jason Roberts (Pharmacist/Clinician, The University of Queensland, Australia)
Professor Frank Rockhold (Biostatistics/Bioinformatics, Duke University, USA)
Professor Stephen Rogerson (Clinician, University of Melbourne, Australia)
Professor Philip Rosenthal (Clinician, University of California, USA)
Professor Kathy Rowan (Researcher, Director of the ICNARC Clinical Trials Unit, UK)
Professor Ignacio Rubio (Researcher, University Hospital Jena, Germany)
Professor Fiona Russell (Paediatrician, The University of Melbourne, Australia)
Dr Sam Saidi (Clinician, University of Sydney, Australia)
Dr Makoto Saito (Clinician/Epidemiologist, University of Tokyo, Japan)
Dr William Schilling (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Anuraj Shankar (Clinician/Triallist, Eijkman-Oxford Clinical Research Unit, Indonesia)
Professor Sanjib Kumar Sharma (Clinician, Koirala Institute of Health Sciences, Nepal)
Professor André Scherag (Statistician, Jena University Hospital, Germany)
Professor Ilan Schwartz (Clinician/Researcher, University of Alberta, Canada)
Professor Julie Simpson (Statistician, University of Melbourne, Australia)
Dr Andre Siqueira (Medical researcher, Fundação Oswaldo Cruz, Brazil)
Professor Frank Smithuis (Clinical researcher, Director of the Myanmar Oxford Tropical Research Unit, Myanmar)
Dr Tim Spelman (Statistician, Karolinska Institute, Sweden)
Dr Kasia Stepniewska (Statistician, University of Oxford, UK)
Dr Nathalie Strub Wourgaft (Clinician, Drugs for Neglected Diseases initiative, Switzerland)
Professor Darrell Tan (Clinician-Scientist, University of Toronto, Canada)
Professor Christoph Thiemermann (Head of Centre for Translational Medicine & Therapeutics, Queen Mary University, UK)
Dr Aimee Taylor (Statistician, Harvard T.H. Chan School of Public Health, USA)
Dr Walter Taylor (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Antoni Torres (Clinician, University of Barcelona, Spain)
Professor Guy Thwaites (Clinician, Director of the Oxford University Clinical Research Unit, Vietnam)
Professor Tran Tinh Hien (Clinician, Oxford Clinical Research Unit, Vietnam)
Professor George Tomlinson (Biostatistician, Mt Sinai Hospital, Canada)
Professor Steven Tong (Clinician, University of Melbourne, Australia)
Professor Paul Turner (Clinician/Researcher, Director of Cambodia Oxford Medical Research Unit, Cambodia)
Professor Ross Upshur (Head of Division of Clinical Public Health, University of Toronto, Canada)
Professor Rogier van Doorn (Clinical Microbiologist, University of Oxford, UK)
Professor Lorenz von Seidlein (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Dee Dee Wang (Clinician/Researcher, Henry Ford Health System, USA)
Professor Sir Nicholas White (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Thomas Williams (Clinician, KEMRI-Wellcome Trust Research Programme, Kenya)
Dr Martin Sebastian Winkler (Clinician, University Medical Center Gottingen, Germany)
Professor Chris Woods (Researcher, Duke University, USA)
Dr Charlie Woodrow (Clinician, Oxford University Hospitals NHS Trust, UK)
Dr Sophie Yacoub (Clinician, Oxford University Clinical Research Unit, Vietnam)
Professor Marcus Zervos (Researcher, Henry Ford Health System, USA)
For clarification of data origin, I suggest that authors add the name of pre-existing database used to build the Surgisphere database used in the Lancet article, like written in the [Journal of New England Medicine article, https://www.nejm.org/doi/full/10.1056/NEJMoa2007621 mentioning the Surgical Outcomes Collaborative registry.
It is well explained in Mehra French interview in French http://www.francesoir.fr/opinions-entretiens-societe-sante/interview-exclusive-mandeep-mehra-lhydroxychloroquine-pas-efficacethat article dataset was built from pre-existing database:
"FS : Vous avez collecté 671 jeux de données de différents pays. Comment avez-vous harmonisé les données et corrigé des biais ?
MM : En fait, il s’agissait donc d’une base de données existante conçue pour l’évaluation des procédures cardiovasculaires et des pharmacothérapies. Cette base de données existe et est utilisée depuis un certain temps. Lorsque la crise du Covid-19 est apparue, nous avons redirigé et recentré l’ensemble de la collecte de données sur le Covid-19 afin de poser les questions critiques. Au début, nous avons étudié la question des risques cardiovasculaires, que nous avons publié dans notre premier article le 1er mai dans le NEJM (New England Journal of Medecine)".
Dear Dr Mehra and colleagues, I have some questions. Comparing your article published on May 1 2020 (New Engl J Med) and this published on May 22 2020 (Lancet), I note that the number of hospitals you analyzed has dramatically increased. But you failed to mention more details.
Could you give more precisions on the hospitals used for the Lancet paper? Second can you confirm that Turkey is considered as a European country? Or do you make difference between cities that are in Europe and cities that are in Asia?
I have also concerns about your ethnic groups. What are people you consider as Asian? Could you confirm that you have no crossbreeding in your population ?
Thank you for your answers
There are a number of strange things with this paper:
Data in Table S3 (appendix) seems incredibly homogeneous across continents, including in characteristics that one would expect to vary locally (e.g. proportion of smokers, comorbidities, treatments received). Note this is supposed to be the raw data, i.e. before selection of controls.
A significant proportion of patients, including in the treatment groups (around 15%) had cardiac antecedents, among which arrythmias (3.3 to 3.6%). It is hard to understand why doctors gave them chloroquine or hydroxychloroquine, which as far as I know, are usually not recommended in such cases. And supposing they did, a separate analysis should have been done excluding these patients.
As noted in previous comments, the remarkable variation in the hazard ratio for diabetes across continents looks suspect.
A high proportion of patients (40.5%), in both control and treatment groups, received antivirals, of which 17,5% were given two or more antivirals in combination. This raises the possibility of drug interactions. To control for this, it would have been recommended to run a a separate analysis without these patients.
Patients included in this study received the treatment < 48 hours after hospitalization, but the latter may happen at various times during the evolution of the disease (especially in a pandemic period when health services may be overcrowded). It would have been more accurate to consider the time from the onset of symptoms - though possibly this information was not available to the authors.
the authors report a very significant increase in the risk of arrythmia among the treatment groups, but whether arrythmia was systematically tested in the control group is unclear (I would expect since CQ/HCQ have known cardiac effects, patients taking these drugs are more likely to have done an electrocardiogram). It is clearly stated, p. 4 of the paper, that “for disease and drug variables, there were no codes to indicate that data were missing; if the patient’s electronic health record did not include information on a clinical characteristic, it was assumed that the characteristic was not present”.
the effect of the dose was not tested (this limitation has been acknowledged by the authors). This is not a trivial issue since doses of CQ/HCQ employed in trials agains COVID-19 vary at least from 400 to 1200 mg/day (according to literature I have seen) and are thus likely to result in different side effects.
the same team submitted a paper about the effect of ivermectin against covid-19, about which questions have been raised here concerning the data acquisition procedure (reported to be similar in both papers): https://www.trialsitenews.com/observational-study-usefulness-of-ivermectin-in-covid-19-illness-raises-some-questions/
https://marlin-prod.literatumonline.com/pb-assets/Lancet/pdfs/S0140673620312903.pdf
Expression of concern: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis
Important scientific questions have been raised about data reported in the paper by Mandeep Mehra et al— Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis1—published in The Lancet on May 22, 2020. Although an independent audit of the provenance and validity of the data has been commissioned by the authors not affiliated with Surgisphere and is ongoing, with results expected very shortly, we are issuing an Expression of Concern to alert readers to the fact that serious scientific questions have been brought to our attention. We will update this notice as soon as we have further information. The Lancet Editors
For people wanting to analyze the data themselves, I have converted all the tables into a spreadsheet. Available on Zenodo: doi.org/dxdp
June 4th, 2020 retraction at https://www.thelancet.com/lancet/article/s0140673620313246
"Today, three of the authors of the paper, "Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis", have retracted their study. They were unable to complete an independent audit of the data underpinning their analysis. As a result, they have concluded that they "can no longer vouch for the veracity of the primary data sources." The Lancet takes issues of scientific integrity extremely seriously, and there are many outstanding questions about Surgisphere and the data that were allegedly included in this study. Following guidelines from the Committee on Publication Ethics (COPE) and International Committee of Medical Journal Editors (ICMJE), institutional reviews of Surgisphere’s research collaborations are urgently needed. The retraction notice is published today, June 4, 2020. The article will be updated to reflect this retraction shortly."
Not many people know what constitutes a typically hyped Lancet publication. Such as this one. It is actually very good that three-quarters of the authors have chosen to quickly retract this paper.
Because, left to its own devices, this glamour UK medical journal, currently basking in the limelight as the media darling for COVID-19 guidance, does not do retractions. It took 12 long years for Wakefield.
Nothing is happening about this hyped up pair of 2008 and 2014 degenerative medicine papers
https://pubpeer.com/publications/2968ADAFF1170575BBB3D0B74FD6CD
https://pubpeer.com/publications/270762D58EFA307DE8A1CBAF0ED4CB
even though a Member of Parliament, who was chairing the Science and Technology Select Committee, had requested retraction because of the by then transparent fraud.
There is an interesting discussion on a blog that looks at some aspects of the company itself and a number of data wrangling and technical aspects:
http://freerangestats.info/blog/2020/05/30/implausible-health-data-firm
The author raises many issues about the data set.
Even assuming the data is mostly correct. The analysis is fundamentally flawed. The study is using only 2 indicators of disease severity and using them at a very low level of fidelity.
In a number of other studies on the topic of covid-19 and HCQ, we can see the inclusion of information such as CT imaging, paO2:FiO2 ratio, body temp, and SPO2 levels used as continues value. Evidence that the study failed to capture disease severity between control and treatment groups is the fact that the number of patients on a ventilator was more than double in HCQ group vs Control. Even if we were to assume that HCQ is causing sudden cardiac death due to QT prolongation that will not result in patients being intubated. We can see that despite their attempt control for comorbidities by not using appropriate indicators of disease severity, the study failed to control for indication bias – that is HCQ was given to significantly more severely ill patients.
"The University of Utah has “mutually agreed” to terminate the faculty appointment of Amit Patel":
Who still seems to have a ResearchGate account with the affiliation to the University of Miami:
Lancet chloroquine paper:
It's very disappointing to see that this paper, retracted in June, still receives a citation in Nature in November: https://www.nature.com/articles/d41586-020-03289-y An interesting case for a forthcoming study: "High incidence of sleeping sickness among reviewers"...
#34, this case and the risks of its post-retraction citations has already been discussed here:
https://link.springer.com/article/10.1007/s11019-020-09990-z
There's even worse than the citation in Nature: even the official WHO website (in French) is still referring uncritically to this retracted study!!! https://www.who.int/fr/emergencies/diseases/novel-coronavirus-2019/question-and-answers-hub/q-a-detail/q-a-hydroxychloroquine-and-covid-19 "À titre d’exemple, une étude observationnelle publiée le 22 mai dans la revue The Lancet a montré que, sur 100 000 patients de différents pays randomisés pour recevoir de l’hydroxychloroquine (seule ou en association avec un macrolide), le taux de mortalité était plus élevé et l’arythmie cardiaque plus fréquente." The text is dated 19 june 2020, i.e. no less than two weeks after the study retraction, and has not been corrected since. Incidentally, an observational study on randomised patients, that's a hugely interesting new concept (©WHO 2020) !!!
Attach files by dragging & dropping,
selecting them, or pasting
from the clipboard.
Uploading your files…
We don’t support that file type.
with
a PNG, GIF, or JPG.
Yowza, that’s a big file.
with
a file smaller than 1MB.
This file is empty.
with
a file that’s not empty.
Something went really wrong, and we can’t process that file.