Using AI to Predict the Probability of Success in Drug Development
ALSO: Patented AI Platform Identifies Promising Early-Stage Biopharma Assets and Companies
Hi! I am Andrii Buvailo, and this is my weekly newsletter, ‘Where Tech Meets Bio,’ where I talk about technologies, breakthroughs, and great companies moving the biopharma industry forward.
If you've received it, then you either subscribed or someone forwarded it to you. If the latter is the case, subscribe by pressing this button:
This newsletter is sponsored by Intelligencia AI, a provider of AI-powered solutions that support data-driven decision-making in drug development.
They lead the way in leveraging proprietary data, biomedical expertise and artificial intelligence (AI) with its patented technology to address significant challenges in the pharmaceutical industry. Its suite of AI-powered solutions delivers actionable insights crucial in mitigating risks and enhancing decision-making associated with drug development by providing an accurate, unbiased assessment of a drug's probability of success. Founded in 2017, Intelligencia AI is headquartered in New York, NY, with offices in Athens, Greece, and employs 110 individuals globally.
Visit intelligencia.ai to discover more and request a demo.
Now, let’s get to this week’s topic!
Many of us know Moore’s Law, which, in short, says that the number of computer transistors (a measure of performance) doubles every two years while prices fall. So, over time, we get more computing power at a lower cost.
Few people know the opposite of Moore’s Law—aptly named Eroom’s Law—which states that drug discovery is becoming slower and exponentially more expensive over time despite technological improvements.
Significant effort and resources are being invested in reversing that trend. Artificial intelligence (AI) is one of the latest tools deployed to reduce costs and increase drug discovery and development productivity.
The reason why the hopes for AI are high lies in the ability of machine learning (ML) algorithms to detect patterns and learn similar to the way humans learn, just much faster and paired with the ability to ingest vast amounts of data in a short period of time. This is a critical feature as the amount of data available continues to multiply rapidly.
AI is already used at many stages in drug discovery and development, including target identification, virtual screening and optimization of compounds, prediction of drug response on cells and the efficacy of drug-target interaction, and after approval for the reporting of adverse effects, to name but a few.
AI is now also being applied to additional critical decision points in drug development, such as assessing the probability that a drug in development will successfully complete clinical trials and receive approval from regulatory authorities, known as the probability of technical and regulatory success (PTRS).
What is PTRS? How is it used, and why is it so important?
The probability of technical and regulatory success (PTRS) is a quantitative assessment of the inherent risk a drug development program bears. This metric estimates the likelihood that a pharmaceutical or biotech product will successfully meet the technical (scientific and clinical) and regulatory (health authorities) requirements from development to market approval.
PTRS assessments of clinical programs occur at various stages during drug development to support decision-making at critical stage gates. In particular, encoding a program’s inherent risk into a PTRS score helps biopharma companies decide whether to continue investing in a drug candidate or discontinue development. Consequently, this informs asset prioritization at the portfolio level, long-range planning and revenue forecasting, and resource allocation decisions.
Both decisions bear risks. Investing in a drug that ultimately fails wastes significant resources that could have been invested in another candidate. The earlier drug developers know a drug is unlikely to succeed, the better. “Failing early” saves potentially 10s or even 100s of millions of dollars in development costs. In addition, failing early also avoids subjecting volunteers and patients to unnecessary treatments.
The flip side of this risk is discontinuing a drug that would have been approved, which results in a missed opportunity to bring a novel therapy to market and provide patients with more treatment options.
The more accurately the probability of success can be assessed, the more informed drug developers’ go/no go and resource allocation decisions become. This improves the efficiency of the drug development process by enabling companies to prioritize programs that are more likely to succeed. Greater efficiency contributes to reversing Eroom’s Law and ultimately benefiting patients.
What are some of the common approaches to calculating PTRS in pharma?
PTRS assessments in pharma are an integral part of pharmaceutical decision-making. Companies have long been trying to evaluate the chances a drug will make it to the market. While the actual process varies from company to company, industry benchmarks are traditionally used as the starting point, adjusted by experts in their respective fields, and supplemented with statistical analysis.
This PTRS approach comes with challenges and limitations:
Industry benchmarks are inconsistent; some are based on just a few program characteristics, and others are based on many. Also, reliable, comprehensive industry-wide benchmarks are challenging to obtain and lack granularity.
Adjustments to these benchmarks are not often based on objective data and can be liable to overestimate the probability of success. Human bias comes into play, both cognitive and motivational.
Statistical analysis, such as regression analysis of large data sets, is challenging and often static because new data cannot be incorporated into the analysis in a timely fashion, impacting the conclusion one can draw from the non-updated data sets.
Given these shortcomings, drug development has long needed a more objective, data-driven approach capable of analyzing vast amounts of data and seamlessly ingesting new data.
How can artificial intelligence (AI) improve PTRS assessments?
Large amounts of continuously updated data, complex interdependencies and subtle patterns are the conditions under which AI surpasses all other approaches.
AI, specifically the machine learning (ML) sub-discipline, has demonstrated that it can outperform drug developers and physicians when assessing the probability of success.
Machine Learning is defined as the use and development of computer systems that can learn and adapt without following explicit instructions by using algorithms and statistical models to analyze and draw inferences from patterns in data.
ML has several significant advantages over traditional approaches:
The algorithms can ingest broader and much larger sets of input data
Sophisticated algorithms can identify inferences that analysts cannot spot by traditional means
Data from hundreds of disparate sources – both structured and unstructured datasets – can be integrated into structured ontologies for a single source of truth.
These rich data sources and superior analyses yield objective probability assessments throughout the drug development pipeline.
An additional feature, namely AI explainability, addresses the concern around the “Black Box” nature of these algorithms. AI explainability provides insights into the relative contributions of various features (e.g., drug biology, clinical trial design, regulatory designations, and outcomes) to the final PTRS prediction. It makes it easier for the user to trust AI-generated probabilities of success.
Benefits of AI-driven probability assessments in clinical development
PTRS predictions have long been used to help make consequential decisions in drug development.
AI is a powerful tool to augment current methods and base drug approval and phase transition predictions on a highly objective, data-driven foundation that aids informed decision-making at critical decision points.
The accurate assessment of the likelihood of clinical trials to succeed helps:
Improve the efficiency of drug development by giving companies an edge in identifying therapeutic programs that are more likely to succeed in clinical trials and should, therefore, be prioritized.
Companies fail drugs with little chance of success early on, potentially saving 10s of millions of dollars and years of effort. These resources can be redirected to more promising drug candidates.
Business development and licensing (BD&L) decisions allow acquirers to better establish the value of a drug candidate they are interested in buying and sellers to better understand the value of their assets.
We need to leave the detrimental dynamics of Eroom’s Law behind and enter into a new era of efficient and more affordable drug development. Artificial intelligence is a powerful tool in this process. Applying it to support decision-making at critical points will benefit pharmaceutical and biotech companies and, most importantly, patients.
Patented AI Platform Identifies Promising Early-Stage BioPharma Assets and Companies
Fierce competition. Thin pipelines. Patent cliffs. The stakes are sky-high for pharmaceutical companies and investors alike. With biotech firms often going public before generating revenue, valuations hinge on the potential for FDA approval of their assets. Herein lies the power of artificial intelligence (AI): by predicting the likelihood of success for these assets, we can uncover undervalued companies, offering a strategic edge not only to hedge funds and VCs but also to big pharma companies considering acquisitions. In this dynamic landscape, projected to see M&A activities reaching as high as $275B in 2024, the ability to discern and invest in promising biotech assets early is crucial.
At Intelligencia AI, our methodology offers a clear edge by delivering superior, real-time insights into the probability of success (PoS) for Phase 1 and Phase 2 assets gaining FDA approval. We used our predictions to create a virtual portfolio that generated 60% returns in one year and 3 out of the top 10 companies we highlighted were acquired.
This advanced predictive capability is pivotal for stakeholders to identify promising acquisition and in-licensing opportunities with a higher degree of accuracy and confidence. Our methodology has proven to be highly effective as a standalone strategy and can also complement established approaches.
Achieving Results: A Glimpse Behind the Curtain
At Intelligencia AI, we used our proprietary, patented probability of success (PoS) prediction technology to create our own virtual portfolio* of publicly traded biotech companies. Our portfolio was constructed with early-stage, pre-proof-of-concept biotech companies with oncology pipelines and a market capitalization between $100M and $1B.
Our investment strategy focused on ranking companies based on predicted overperformance against historic benchmarks and selecting the top percentile of ranked companies. The ranked list discussed here was produced algorithmically. Overall, results can be further qualified and prioritized by our team of expert analysts when further dimensions may be relevant to our partners.
Biotech companies ranked by their expected overperformance.
Intelligencia AI’s approach has been prospectively validated with time-stamped predictions from April 21, 2023. By March 21, 2024, Intelligencia AI’s portfolio of top 40 ranked biotechs achieved a 60% return and a Sharpe Ratio of 1.83. In comparison, similar market cap companies had a 25% return, while the biotech market benchmark index (XBI-ETF) saw a 17% return, underscoring the strategy's effectiveness. This approach has also been back-tested over a 5-year period.
Performance of Intelligencia AI portfolio of top 40 ranked biotechs compared to the rest of the biotech market.
The same methodology can be applied continuously incorporating the latest information to construct the most up-to-date portfolios. We continued to construct such time-stamped profiles. Below we highlight one constructed in late November 2023 displaying top ten picks from that portfolio. Highlights include multiple recently acquired early-stage biotechs.
Highlights from Intelligencia AI’s portfolio of top 10 ranked biotechs on November 28, 2023.
Harpoon Therapeutics (HARP), which has developed a portfolio of novel T-cell engagers, entered into a definitive agreement with Merck (MSD) on January 8, 2024.
We singled out Harpoon Therapeutics’ assets in clinical development as standing out from competitive products. Both the lead candidate, HPN328 targeting T-cell engager, and HPN217 targeting B-cell maturation antigen (BCMA), were included in the acquisition.
PoS distribution of ongoing assets similar to Harpoon’s candidate HPN217 targets B-cell maturation antigen (BCMA). Harpoon’s candidate had been outperforming most development programs within the same indication and stage of development, belonging to the top quartile.
Ambrx (AMAM), an early-stage biotech with proprietary Antibody Drug Conjugate (ADC) technology, entered into a definitive agreement with Johnson & Johnson on January 8, 2024.
We singled out two of the programs currently in early-stage clinical development as highly promising. Both novel ADC candidates, ARX788 and ARX517, were part of the acquisition.
Kinnate Biopharma Inc. (KNTE), a clinical-stage precision oncology company, entered into a definitive agreement with XOMA Corporation on February 16, 2024.
We highlighted two of their programs, Exarafenib and KIN-3248 treating melanoma and intrahepatic cholangiocarcinoma respectively, which had probabilities of success that far exceeded their expected historical benchmarks.
Better Predictions, Better Investment and Clinical Development Decisions With Validated PoS Assessments
The ability to create the above-described composite biotech scores on the basis of PoS is driven by highly accurate, prospectively validated PoS predictions for specific programs. Built on a unique, proprietary database, Intelligencia AI’s platform brings together expertly curated and harmonized clinical development, biological and regulatory data, which feeds our AI algorithms to assess drug approval and phase transition probabilities.
Our algorithms show an 83%** accuracy in predicting FDA approvals for Phase 2 oncology programs, reaching 90% retrospectively. We achieve this performance by using standard calibration techniques and continuously validating our predictions against real FDA outcomes.
In conducting a 4-year, ongoing prospective study, we demonstrate the high accuracy of our algorithms for early-stage development programs. In 2019, we collaborated with a leading pharmaceutical company to assess PoS for all FDA-track, industry-led, interventional Phase 1 and Phase 2 non-small cell lung cancer (NSCLC) and melanoma programs.
We have been monitoring these programs since then, with 140 failures and 6 approvals. Our platform accurately predicted approval for four drugs, including novel ones and three receiving accelerated approvals: Amivantamab, Tepotinib, Tebentafusp, and Selpercatinib. Our PoS predictions for these programs were significantly higher than for those of similar-stage peers, even with relatively limited information at the time. Also, by time stamping the PoS assessments, we show that the accuracy of the PoS predictions has further improved in later releases of our algorithms.
The graphs show the distribution of approved and failed programs, based on the PoS assigned by Intelligencia AI. Our algorithms have consistently assigned a significantly higher PoS to programs that moved on to approval and lower probability to programs that eventually failed. Importantly, our accuracy has improved over time, as the comparison between 2019 (left panel) and 2022 (right panel) shows.
Conclusion and Future Outlook
Our AI-driven approach in assessing biotech companies based on the risk of their development portfolio constitutes an innovative approach to identify assets for acquisition and in-licensing. The returns realized with our approach highlight its accuracy and demonstrate its capability to guide timely, informed and impactful decisions. These capabilities allow us to help our partners quickly pinpoint overlooked opportunities and realize superior returns.
As we continue optimizing the use of AI for risk assessment in drug development, we are committed to supporting life science companies in bringing groundbreaking therapies to the market and transforming patients’ lives.
Visit intelligencia.ai to discover more and request a demo.
* This material is provided for informational purposes only, and it is not, and may not be relied on in any manner as investment advice or as an offer to sell or a solicitation of an offer to buy an interest in any fund or investment vehicle managed by Intelligencia AI or any other Intelligencia entity. The information and data are as of the publication date unless otherwise noted, and Intelligencia has no obligation to update such information or data. Certain information contained herein has been obtained from third-party sources. Although such content is believed to be reliable, it has not been independently verified as to its accuracy or completeness and cannot be guaranteed.
** Based on Intelligencia AI predictions as of May 2022 for oncology programs with definite announcements in 2023.