Quantum deep reinforcement learning for clinical decision support in oncology: application to adaptive radiotherapy | Scientific Reports – Nature.com
Quantum deep reinforcement learning
Quantum deep reinforcement learning is a novel action value-based decision-making framework derived from QRL23 and deep q-learning10 framework. Like conventional RL9,31, our qDRL based CDSS framework is comprised of 5 main elements: clinical AI agent, ARTE, radiation dose decision-making policy, reward, and q-value function. Here, the AI agent is a clinical decision-maker that learns to make dose decisions for achieving clinically desirable outcomes within the ARTE. The learning takes place by the agent-environment interaction, which can be sequentially ordered as: the AI decides on a dose and executes it, and in response, a patient (part of the ARTE) transits from one state to the next. Each transition provides the AI with feedback for its decision in terms of RT outcome and associated reward value. The goal of RL is for the AI to learn a decision-making policy that maximizes the reward in the long run, defined in terms of a specified q-value function that assigns a value to every state-dose-decision pair obtained from the accumulation of rewards over time (returns).
Assuming Markovs property (i.e., an environments response at time (t + 1) depends only on the state and dose-decision at time (t)), the qDRL task can be mathematically described as a 5-tuple ((S, left| D rightrangle , TF, P, R)), where (S) is a finite set of patients states, (left| D rightrangle) is a superimposed quantum state representing the finite set of eigen-dose decision, (TF:S times D to S^{prime }) is the transition function that maps patients state (s_{t}) and eigen-dose (left| d rightrangle_{t}) to the next state (s_{t + 1}), (P_{LC|RP2} :S^{prime } to left[ {0,1} right]) is the RT outcome estimator that assigns probability values (p_{LC}) and (p_{RP2}) to the state (s_{t + 1}), and (R:left[ {0,1} right] times left[ {0,1} right] to {mathbb{R}}) is the reward function that assigns a reward (r_{t + 1}) to the state-decision pair (left( {s_{t} ,left| d rightrangle_{t} } right)) based on the outcome probability estimates.
Eigen-dose (left| d rightrangle) is a physically performable decision that is selected via quantum methods from the superimposed quantum state (left| D rightrangle) which simultaneously represents all possible eigen-doses at once. In simple words, (left| D rightrangle) is the collection of all possible dose options and (left| d rightrangle) is one of those options which is selected after a decision is made. Selecting dose decision (left| d rightrangle) is carried out in two steps: (1) amplifying the optimal eigen-dose (left| d rightrangle^{*}) from the superimposed state (left| D rightrangle) (i.e., (left| D rightrangle^{prime } = widehat{Amp}_{{left| d rightrangle^{*} }} left| D rightrangle)) and (2) measuring the amplified state (i.e., (left| d rightrangle = widehat{Measure}(left| {D^{prime } } rightrangle )).
The optimal eigen-dose (left| d rightrangle^{*}) is obtained from deep Q-net, which is the AIs memory. Deep Q-net, (DQN:S to {mathbb{R}}^{d}), is a neural network that takes patients state as input and then outputs q-value for each eigen-dose ((left{ {q_{left| d rightrangle } } right})). The optimal dose is then selected following greedy policy where the dose with the maximum q-value is selected (i.e., (left| d rightrangle^{*} = begin{array}{*{20}c} {argmax} \ {left| {d^{prime } } rightrangle } \ end{array} { q_{left| d rightrangle } })). We have applied a double Q-learning 32 algorithm in training the deep Q-net. The schematic of a training cycle is presented in Fig.2 and additional technical details are presented in the Supplementary Material.
We initially employed Grovers amplification procedure33,34 for the decision selection mechanism. While Grovers procedure works on a quantum simulator, it fails to correctly work in a quantum computer. The quantum circuit depth of Grovers procedure (for 4 or higher qubits) is much greater than the coherence length of the current quantum processor35. Whenever the quantum circuit length exceeds the coherence length, quantum state becomes significantly affected by the system noise and loses vital information. Therefore, we designed a quantum controller circuit that is shorter than the coherence length and is suitable for the task of decision selection. The merit of our design is its fixed length; since its length is fixed for any number of qubits, it is suitable for higher qubit systems, as much as permitted by the circuit width. Technical details regarding its implementation in quantum processor is presented in the Supplementary Materials.
An example of a controller circuit is given in Fig.5. Controller circuits use twice the number of qubits (n), which can be divided into control and main. Optimal eigen-states obtained from deep Q-net are created in the control by selecting the appropriate pre-control gates. Then the control is entangled with the qubits from the main via controlled NOT (CNOT) gates. CNOT gates are connected between a control qubit from the control and a target qubit from the main. CNOT gates flip the target qubit from (left| 1 rightrangle) to (left| 0 rightrangle) only when the control is in (left| 1 rightrangle) state and does not perform any operation otherwise. Because all the main qubits are prepared in (left| 0 rightrangle) state, we introduced the reverse gates (n X-gates in parallel) to flip them to (left| 1 rightrangle). X-gates flip (left| 0 rightrangle) to (left| 1 rightrangle), and vice-versa. The CNOT flips all the qubits whose controls are in (left| 1 rightrangle) state, creating a state that is element-wise opposite to the marked state. Finally, another set of reverse gates is applied to the main before making a measurement.
Quantum controller circuit for a 5 qubit (32 bit) system. (a) Quantum controller circuit for the selection of the state (left| {10101} rightrangle). The probability distribution corresponding to (b) failed Grovers amplification procedure for one iteration run in the 5-qubit IBMQ Santiago quantum processor and (c) successful quantum controller selection run in the 15-qubit IBMQ Melbourne quantum processor.
Another advantage of the controller circuit is controlled uncertainty level. The controller circuit has additional degrees of freedom that can control the level of uncertainty that might be needed to model a highly dubious clinical situation. By replacing the CNOT gate by a more general (CU3left( {theta ,phi ,lambda } right)) gate, we can control the level of additional stochasticity with the rotation angles (theta), (phi), and (lambda), which corresponds to the angles in the Bloch sphere. The angles can either be fixed or, for additional control, changed with training episode.
The patients state in the ARTE is defined by 5 biological features: cytokine (IP10), PET imaging feature (GLSZM-ZSV), radiation doses (Tumor gEUD and lung gEUD), and genetics (cxcr1- Rs2234671). Their descriptions are presented in Table 2. These 5 variables were selected from a multi-objective Bayesian Network study13, which considered over 297 various biological features and found the best features for predicting the joint LC and RP2 RT outcomes.
The training data analyzed in this study are obtained from the University of Michigan study UMCC 2007.123 (NCI clinical trial NCT01190527) and the validation data analyzed in this study are obtained from the RTOG-0617 study (NCI clinical trial NCT00533949). Both trials were conducted in accordance with relevant guidelines and regulations and informed consent was obtained from all subjects and/or legal guardians. Details on training and validation datasets, and necessary model imputation carried out to accommodate the differences in the datasets are presented in the Supplementary Materials.
Deep Neural Networks (DNN) were applied as transition functions for IP10 and GLSZM-ZSV features. They were trained with a longitudinal (time-series) dataset, with the pre-irradiation patient state and corresponding radiation dose as input features and post-irradiation state as output. For lung and tumor gEUD, we utilized prior knowledge and applied a monotonic relationship for the transition function since we know that gEUD should increase with increasing radiation dose. We assumed that the change in gEUD is proportional to the dose fractionation and tissue radiosensitivity,
$$frac{{gleft( {t_{n} } right) - gleft( {t_{n - 1} } right)}}{{t_{n} - t_{n - 1} }} propto d_{n} left( {1 + frac{{d_{n} }}{{frac{alpha }{beta }}}} right).$$
(1)
Here (gleft( {t_{n} } right)) is the gEUD at time point (t_{n}), (d_{n}) is the radiation dose fractionation given during the nth time period, and (alpha /beta) ratio is the radiosensitivity parameter which differs between tissue type. Note that we first applied constrained training42 to maintain monotonicity with DNN model, however the gEUD over time trend was flatter than anticipated, thus we opted for a process-driven approach in the final implementation. The technical details on the NNs and its training are presented in the Supplementary Material.
DNN classifiers were applied as the RT outcome estimator for LC and RP2 treatment outcomes. They were trained with post irradiation patient states as input and binary LC and RP2 outcomes as its labels.
RT outcome estimator must also satisfy a monotone condition between increasing radiation dose and increasing probability of local control as well as probability of radiation induced pneumonitis. To maintain this monotonic relationship, we used a generic logistic function,
$$p_{LC|RP2} = frac{1}{{1 + exp left( {frac{{gleft( {t_{6} } right) - mu }}{T}} right)}},$$
(2)
where (gleft( {t_{6} } right)) is the gEUD at week 6, and (mu) and (T) are two patient-specific parameters that are learned from training the DNN. Here, (mu) and (T) are the outputs of two neural networks that are fed into the logistic function and tuned one after the other, leaving the other fixed. The training details are presented in the Supplementary Materials.
The task of the agent is to determine the optimal dose that maximizes (p_{LC}) while minimizing (p_{RP2}). Accordingly, we built a reward function on the base function (P^{ + } = P_{LC} left( {1 - P_{RP2} } right)) as shown in Fig.6. The algebraic form is as follows,
$$R = left{ {begin{array}{*{20}l} {P^{ + } + 10 } hfill & { {text{if}} 70% < p_{Lc} < 100% ;{text{and}}; 0% < p_{RP2} < 17.2% } hfill \ {P^{ + } + 5} hfill & {{text{if}} 50% < p_{Lc} < 70% ;{text{and}}; 17.2% < p_{RP2} < 50% } hfill \ {P^{ + } - 1} hfill & {{text{if}} 0% < p_{Lc} < 50% ;{text{and}}; 50 < p_{RP2} < 100% } hfill \ end{array} } right.$$
(3)
Reward function for reinforcement learning. Contour plot of reward function as a function of the probability of local control (PLC) and radiation induced pneumonitis of grade 2 or higher (PRP2). Area enclosed by the blue line corresponds to the clinically desirable outcome, i.e., (P_{LC} > 70{%}) and ({P_{RP2}} <17.2{%}). Similarly, the area enclosed by the green lines corresponds to the computationally desirable outcome, i.e., (P_{LC} > 50{%}) and ({P_{RP2}} <50{%}). Along with (P_{LC} times (1-P_{RP2})) the AI agent receives+10 reward for achieving clinically desirable outcome,+5 for achieving computationally desirable outcome, and -1 when unable to achieve a desirable outcome.
Here the AI agent receives additional 10 points for achieving clinically desirable outcome (i.e., (p_{LC} > 70% quad {text{and}} quad p_{RP2} < 17.2%)), 5 points for achieving computationally desirable outcome (i.e., (p_{LC} > 50% quad {text{and}} quad p_{RP2} < 50%)), and -1 point for failing to achieve a desirable outcome altogether. The negative point motivates the AI agent to search for the optimal dose as soon as possible.
To compensate for low number of data points we employed WGAN-GP43, which learns the underlying data distribution and generates more data points. We generated 4000 additional data points for training qDRL models. Having a larger training dataset helps the reinforcement learning algorithm in accurately representing the state space. The training details are presented in the Supplementary Material.
See the rest here:
Quantum deep reinforcement learning for clinical decision support in oncology: application to adaptive radiotherapy | Scientific Reports - Nature.com
- What's Going On With IonQ Stock Today? - Benzinga - April 1st, 2025 [April 1st, 2025]
- Quantum computer solves optimization problem at Ford's assembly line - Interesting Engineering - April 1st, 2025 [April 1st, 2025]
- Finnish Quantum Startup IQM in Talks to Raise Over 200 Million - Bloomberg.com - April 1st, 2025 [April 1st, 2025]
- Quantum Computing Approach Generates First Ever Truly Random Number - Discover Magazine - April 1st, 2025 [April 1st, 2025]
- National Quantum Computing Centre Launches Insights Paper Exploring Quantum Computings Transformative Potential in Healthcare and Pharmaceuticals -... - April 1st, 2025 [April 1st, 2025]
- JPMorganChase, Quantinuum, Argonne National Laboratory, Oak Ridge National Laboratory and University of Texas at Austin advance the application of... - April 1st, 2025 [April 1st, 2025]
- Certified randomness using a trapped-ion quantum processor - Nature - April 1st, 2025 [April 1st, 2025]
- What's Going On With Quantum Computing Stock Today? - Benzinga - April 1st, 2025 [April 1st, 2025]
- D-Wave Pushes Back At Critics, Shows Off Aggressive Quantum Roadmap - The Next Platform - April 1st, 2025 [April 1st, 2025]
- Quantum Computing Inc. Secures Quantum Photonic Vibrometer Order with Delft University of Technology - Yahoo Finance - April 1st, 2025 [April 1st, 2025]
- How quantum cybersecurity changes the way you protect data - TechTarget - April 1st, 2025 [April 1st, 2025]
- Pasqal Selected for 140-Qubit Quantum Computer to Be Hosted at CINECA - insideHPC - April 1st, 2025 [April 1st, 2025]
- D-Wave and Japan Tobacco use quantum to build a better AI model for drug discovery - SiliconANGLE - April 1st, 2025 [April 1st, 2025]
- Quantum Computing is a cross industry revolution, and we want to be part of it - CTech - April 1st, 2025 [April 1st, 2025]
- Quantum Computing Stocks Fall. Here's A Look At Upcoming News Events. - Investor's Business Daily - April 1st, 2025 [April 1st, 2025]
- Honeywell May Take Quantinuum Public in Next 2 Years. Its a Quantum Thing. - Barron's - April 1st, 2025 [April 1st, 2025]
- The 6 different types of quantum computing technology - TechTarget - April 1st, 2025 [April 1st, 2025]
- Nvidia to Open Quantum Computing Research Center in Boston This Year in a Landmark for Regions Tech Sector - The Harvard Crimson - April 1st, 2025 [April 1st, 2025]
- Quantum Threats Are HereWhy the Next Cybersecurity Boom May Already Be Underway - Baystreet.ca - April 1st, 2025 [April 1st, 2025]
- D-Wave and Japan Tobacco Validate Quantum and AI Workflow Towards Generative Drug Discovery - The Quantum Insider - April 1st, 2025 [April 1st, 2025]
- The High Cost of Quantum Randomness Is Dropping - Quanta Magazine - April 1st, 2025 [April 1st, 2025]
- Beyond encryption: Why quantum computing might be more of a science boom than a cybersecurity bust - oodaloop.com - April 1st, 2025 [April 1st, 2025]
- NVIDIA (NVDA): One of the Best Quantum Computing Stocks to Buy Right Now? - Yahoo Finance - March 18th, 2025 [March 18th, 2025]
- I work at a leading quantum lab: Here are the qualifications recruiters in the field are looking for - Business Insider - March 18th, 2025 [March 18th, 2025]
- 5 wild things quantum computing could unlock now that Big Tech believes a breakthrough is within reach - Yahoo - March 18th, 2025 [March 18th, 2025]
- Controversy erupts over claims Microsoft invented a new state of matter - Salon - March 18th, 2025 [March 18th, 2025]
- Chinese quantum processor is 1 quadrillion times faster than the best supercomputer and it rivals Google's breakthrough Willow chip - Livescience.com - March 18th, 2025 [March 18th, 2025]
- IQM Quantum wants to be the European answer to Google and IBM - Sifted - March 18th, 2025 [March 18th, 2025]
- Twisting atomically thin materials could advance quantum computers - University of Rochester - March 18th, 2025 [March 18th, 2025]
- D-Wave Quantum Stock Hits $11: Heres What This Top Analyst Predicts Ahead - TipRanks - March 18th, 2025 [March 18th, 2025]
- A Computer Has Achieved "Quantum Supremacy" On Real-World Problem For First Time, Company Claims - IFLScience - March 18th, 2025 [March 18th, 2025]
- INVESTOR ALERT: Pomerantz Law Firm Announces the Filing of a Class Action Against Quantum Computing Inc. and Certain Officers - QUBT - PR Newswire - March 18th, 2025 [March 18th, 2025]
- D-Wave Quantum Sets Benchmark with New Computing Advance - News and Statistics - IndexBox, Inc. - March 18th, 2025 [March 18th, 2025]
- Rigettis Rally Hits a Bump Are Insider Sales a Red Flag? - Wall Street Pit - March 18th, 2025 [March 18th, 2025]
- Quantum AI: What Is It and How Does It Work? - CNET - March 18th, 2025 [March 18th, 2025]
- D-Wave Shares Jump 46.9% on Friday - Should You Buy QBTS Stock? - TradingView - March 18th, 2025 [March 18th, 2025]
- 2 Top Quantum Computing Stocks to Buy in 2025 - The Motley Fool - March 13th, 2025 [March 13th, 2025]
- D-Wave Claims Breakthrough. Quantum Computing Stocks Gain. - Investor's Business Daily - March 13th, 2025 [March 13th, 2025]
- Physicists Just Witnessed a Quantum Phase Flip and Its More Mind-Bending Than Expected - SciTechDaily - March 13th, 2025 [March 13th, 2025]
- Beyond Classical: D-Wave First to Demonstrate Quantum Supremacy on Useful, Real-World Problem - Business Wire - March 13th, 2025 [March 13th, 2025]
- What is quantum computing and how it could change the tech world - Yahoo Finance - March 13th, 2025 [March 13th, 2025]
- Quantum Computing Giant IonQ Is Down More Than 60% From its All-Time High. Should You Buy The Dip? - The Motley Fool - March 13th, 2025 [March 13th, 2025]
- D-Wave Deep Dive: A Look at The Quantum Advantage Findings -- And The Questions That Remain - The Quantum Insider - March 13th, 2025 [March 13th, 2025]
- D-Wave claims to have achieved quantum supremacy at last, but others disagree - SiliconANGLE News - March 13th, 2025 [March 13th, 2025]
- D-Wave Claims It Achieves Quantum Supremacy. What the Breakthrough Means for Quantum Computing. - Barron's - March 13th, 2025 [March 13th, 2025]
- D-Wave Posts Wider-Than-Expected Loss. Why the Stock Is Rising After Earnings. - Barron's - March 13th, 2025 [March 13th, 2025]
- Nu Quantum Partners With The University of Sussex, Cisco, and Infineon to Scale Trapped Ion Quantum Computers - The Quantum Insider - March 13th, 2025 [March 13th, 2025]
- IonQ Could Be a Quantum Computing Powerhouse, but Is It a Buy Right Now? - The Motley Fool - March 13th, 2025 [March 13th, 2025]
- D-Wave Quantum Inc. Reports Fourth Quarter and Year-End 2024 Results - TradingView - March 13th, 2025 [March 13th, 2025]
- Hybrid Quantum Workflow Moves Toward Real-World Applications - IoT World Today - March 13th, 2025 [March 13th, 2025]
- As NVIDIAs Quantum Day Nears, Analysts Suggest Event is More Than a Gesture - The Quantum Insider - March 13th, 2025 [March 13th, 2025]
- D-Wave Posts Wider-Than-Expected Loss. Why the Stock Is Rising Anyway. - MSN - March 13th, 2025 [March 13th, 2025]
- Recent Breakthroughs Accelerate The Race For Quantum Computing - Forbes - March 13th, 2025 [March 13th, 2025]
- An operating system for executing applications on quantum network nodes - Nature.com - March 13th, 2025 [March 13th, 2025]
- D-Wave Reports Quantum Advantage in Materials Simulation Study - HPCwire - March 13th, 2025 [March 13th, 2025]
- Experts Weigh in on Microsofts Topological Qubit Claim - Physics - March 13th, 2025 [March 13th, 2025]
- Quantum Computing Inc. To Attend 37th Annual ROTH Conference - PR Newswire - March 13th, 2025 [March 13th, 2025]
- Quantum leap: Passwords in the new era of computing security - BleepingComputer - March 13th, 2025 [March 13th, 2025]
- Quantum computing will reach its inflection point in 2029: How investors should prepare - Finextra - March 13th, 2025 [March 13th, 2025]
- Quantum computing - Unlocking science, and maybe your bank account - Home Team Science and Technology Agency - March 13th, 2025 [March 13th, 2025]
- SXSW 2025 live coverage: The potential of quantum computing, Ireland's prime minister makes a splash, and a Metallica concert in Apple Vision Pro -... - March 13th, 2025 [March 13th, 2025]
- QuamCore Emerges From Stealth With $9 Million in Seed Funding to Build Worlds First Scalable 1 million Qubit Quantum Computer - The Quantum Insider - March 13th, 2025 [March 13th, 2025]
- QuamCore Emerges with $9M Seed Funding to Build Scalable Million-Qubit Quantum Computer - Quantum Computing Report - March 13th, 2025 [March 13th, 2025]
- QuamCore emerges from stealth with $9 million in Seed funding to build a 1 million qubit quantum computer - Scientific Computing World - March 13th, 2025 [March 13th, 2025]
- D-Wave says it achieved quantum supremacy using its computer - Fast Company - March 13th, 2025 [March 13th, 2025]
- D-Waves Annealing Quantum Computer Just Beat a Supercomputer Heres Why It Matters - Wall Street Pit - March 13th, 2025 [March 13th, 2025]
- Recently, a series of quantum computer-themed exchange-traded funds (ETFs) have been released in the.. - - March 13th, 2025 [March 13th, 2025]
- China unveils quantum computer thats one quadrillion times faster than existing supercomputers - Yahoo Finance UK - March 7th, 2025 [March 7th, 2025]
- China unveils quantum computer that could spell new era of processors - The Independent - March 5th, 2025 [March 5th, 2025]
- Startup PsiQuantum says it is making millions of quantum computing chips - Reuters - March 1st, 2025 [March 1st, 2025]
- A quantum computing startup says it is already making millions of light-powered chips - The Conversation - March 1st, 2025 [March 1st, 2025]
- Quantum Breakthrough: Microsoft and Purdue Unlock the Future of Topological Qubits - SciTechDaily - March 1st, 2025 [March 1st, 2025]
- Interested in Quantum Computing Investing? Here Are 4 Fantastic Picks to Maximize Your Odds of Picking a Winner - The Motley Fool - March 1st, 2025 [March 1st, 2025]
- If I Could Only Buy 1 Quantum Computing Stock, This Would Be It - The Motley Fool - March 1st, 2025 [March 1st, 2025]
- Amazon unveils quantum chip, aiming to shave years off development time - Reuters - March 1st, 2025 [March 1st, 2025]
- Quantum Computing Is Finally Here. But What Is It? - Bloomberg - March 1st, 2025 [March 1st, 2025]
- Microsoft makes quantum computing breakthrough - Drexel University The Triangle Online - March 1st, 2025 [March 1st, 2025]
- Google, Microsoft, and now Amazon: The quantum computing race is heating up - Quartz - March 1st, 2025 [March 1st, 2025]
- Groundbreaking qubit technology reduces errors in quantum computing - The Brighter Side of News - March 1st, 2025 [March 1st, 2025]
- Fortanix Tackles Quantum Computing Threats With New Algorithms - Dark Reading - March 1st, 2025 [March 1st, 2025]