top of page
  • Writer's pictureVibhu Agarwal

Beyond Clinical Trials: Building the Silver Standard of Medical Evidence

Background The demand for healthcare in India reached USD 372 billion in 2022 and driven by rising income, higher health awareness, chronic diseases and better access to health insurance it is expected to grow at 22% CAGR[1]. The availability of medical evidence to inform health policy, public health and patient care decisions is vital for balancing the objectives of efficacy, safety, patients’ values and treatment costs.

Randomized Clinical Trials (RCTs) constitute the gold standard in medical evidence. RCTs are considered superior to other methods of designing scientific studies since they prospectively define the study methodology including inclusion/exclusion criterion, exposures and endpoints. More importantly, RCTs minimize the effect of potential imbalance in the risk factors associated with the outcome that may exist in the patients that were exposed to the treatment versus others (confounding). However RCTs are extremely difficult to conduct. The US Department for Health and Human services has estimated the median cost of a phase 3 trial for a new drug approval to be USD 41 million. It is estimated that nearly 80% RCTs fail to enroll the required number of patients on time and nearly half fail to enroll or retain the number required to power the study. The most significant drawback of RCTs however is poor external validity. RCTs are designed with inclusion and exclusion criteria that facilitate patient enrollment and improve the chances of discovering a clinically significant treatment effect. As a consequence, the inferences drawn from RCTs do not generalize to the “real world” patient population all too frequently. A well known study on 8 year death rate data conducted by a team of researchers at the Johns Hopkins University, places medical interventions as the third leading cause of deaths in the US.

On the other hand, observational studies – in which the exposure and other independent variables are not set by design, may often be executed inexpensively to yield valid medical evidence. Specifically, retrospective observational studies can take advantage of historical patient data that gets generated during medical treatments. For instance in the 1980s, a retrospective case-control study led to the identification of a prone sleeping position in infants as a risk factor for sudden infant death syndrome[2]. Public health programs such as vaccinations require periodic reassessment of safety and efficacy and are another use case where observational studies offer a timely and pragmatic approach to generating medical evidence. Development of evidence-based guidelines for rare diseases – where severe sample size constraints may not permit RCTs to be conducted, is yet another area in which the evidence from observational studies may be more important than what is obtainable from RCTs[3]. Lastly, since the evidence from RCTs informs only a small proportion of medical interventions[4] (estimates vary between 10 - 20%), the case for using “big” patient data to advance evidenced-based medical practice seems obvious.

Opportunity Towards its goal of achieving universal health care as part of sustainable development goals, the Indian government introduced the Ayushman Bharat Pradhan Mantri Jan Arogya Yojana (PM-JAY) in 2018 with aim of providing healthcare insurance to 100 million families which translates into nearly 540 million beneficiaries obtaining healthcare cover of INR 0.5 million per year. The PM-JAY is the largest government sponsored healthcare program in the world. Between 2018 through 2022 a total of 3,67,45,368 hospitalizations worth INR 453,761.5 million were recorded under this program. The indications, treatments and the associated costs captured as longitudinal, computer-readable claims generated during these medical episodes represent an invaluable resource for medical research. To put this into perspective, the Nationwide Inpatient Sample (NIS)–the largest all-payer inpatient database in the United States that captures approximately 8 million hospitalizations every year, is the most commonly used ‘big-data’ repository supporting as many as 42% of medical studies within certain specialities[5]. By comparison, not only does the PM-JAY represent a population that is several times larger, but also one that may be engaging with the healthcare ecosystem for the very first time and has remained understudied and underserved thus far. The PM-JAY claims data therefore holds immense potential to advance medical research, along the lines of the NIS but with far greater impact.

Challenges A number of challenges need to be addressed before the scientific value of a large, nationwide medical claims repository can be unlocked. A frequent criticism of observational studies is that they may lead to biased conclusions on account of unrecognized factors as well as overestimated treatment effects. Advanced statistical methods such as multivariable analysis, matching, regression-based risk assessment, propensity score estimation and instrument variable analyses may mitigate the risk of bias in observational studies to some extent. However, applied incorrectly, these methods may themselves lead to incorrect and sometimes conflicting results. Working with patient data carries the risk of a data breach, leading to compromised patient confidentiality. The Digital Personal Data Protection bill (DPDP 2022)[6] – a legal framework for storing and processing patient data in India is currently being formulated. Legal obligations of data principals, fiduciaries and intermediaries have been defined but the mechanisms for compliance remain unclear. More importantly, effective data stewardship must follow from a patient-centric data management culture. SOPs that place the patient before the data and establish respect as a non-negotiable prerequisite for data access need to be reinforced through continuous education. Further, the IT infrastructure required for patient data capture is in early stages of adoption in many hospitals, resulting in low compliance. Many continue to use proprietary data models and terminologies which makes data sharing, aggregation and interpretation difficult. To address this, the digital backbone initiative taken by the National Health Authority of India’s Ayushman Bharat Digital Mission (ABDM) has published a consultation paper that defines the essential components needed for healthcare service delivery based on open standards[7]. Finally, the skills needed to implement the information safeguards, data management protocols, statistical methodology and legal frameworks will need to be developed through extensive cross-disciplinary training.

Conclusion RCTs are considered the gold standard in medical evidence, however they form the basis of only a small proportion of medical guidelines, leaving the majority of medical interventions to be guided by weaker sources of evidence such as case reports and expert opinion. Studies on the role of informal healthcare providers – individuals who offer medical consults but may lack medical accreditation, suggest that in locations that do not have adequate medical infrastructure 71% of patients may have received medical treatment from an informal provider[8] and therefore, likely based on anecdotal knowledge. A wide inference-gap characterizes the discordance between practice and evidence and has profound implications for healthcare policy and delivery. Observational studies offer an alternative means of generating medical evidence at scale if these are designed carefully and executed with the appropriate safeguards for patient privacy, information security, methods and legal sanctity. With a nearly pan-India coverage, PM-JAY generates longitudinal medical data that may hold the key to generating silver standard medical evidence at scale, thereby transforming medical care delivery to a large population with unmet medical needs.


  1. Sarwal R., Prasad U., Gopal KM., Kalal S., Kaur D., Kumar A., Regy P.V. and Sharma J., 2021. Investment opportunities in India's healthcare sector.

  2. Mitchell EA, Scragg R, Stewart AW, et al. Results from the first year of the New Zealand Cot Death Study. N Z Med J 1991; 104:71-6

  3. Benjamin RS. Observational studies: goldmines of information on rare diseases. BMC Med. 2017 May 12;15(1):100. doi: 10.1186/s12916-017-0868-7. PMID: 28494808; PMCID: PMC5427529.

  4. Hutchinson, N., Moyer, H., Zarin, D. A., & Kimmelman, J. (2022). The proportion of randomized controlled trials that inform clinical practice. eLife, 11, e79491.

  5. Tang OY, Pugacheva A, Bajaj AI, Rivera Perla KM, Weil RJ, Toms SA. The National Inpatient Sample: A Primer for Neurosurgical Big Data Research and Systematic Review. World Neurosurg. 2022 Jun;162:e198-e217. doi: 10.1016/j.wneu.2022.02.113. Epub 2022 Mar 3. PMID: 35247618.

  6. Ministry of Electronics and Information Technology (2022). The Digital Personal Data Protection Bill, 2022.

  7. National Health Authority. Consultation Paper on Operationalizing Unified Health Interface (UHI) in India, 2022.

  8. Mekoth N, Dalvi V. Does Quality of Healthcare Service Determine Patient Adherence? Evidence from the Primary Healthcare Sector in India. Hosp Top. 2015;93(3):60-8. doi: 10.1080/00185868.2015.1108141. PMID: 26652042

218 views0 comments


bottom of page