Optimal Scheduling for Cross-Facility Workflows

XWF for CDT

Author

Mark Asch

Published

April 1, 2026

Welcome

This (online) user-guide contains a complete presentation of optimal scheduling applied to Exascale and post-Exascale workflows. These workflows combine data acquisition, data storage, data transfer and data analysis. The HPC components of such workflows can incorporate a diverse range of models, including partial differential equation solvers, algebraic solvers, AI/ML-based models, and data analytics components, all integrated into a single process (Ferreira da Silva et al. 2024), (Unat et al. 2025).

The objective here is to address the inherent cross-facility, multi-domain character of these workflows. This requires a very carefully-crafted theoretical foundation that can be readily generalized and extended to these contexts. The material covers both the theory and its application to practical use-cases, including real contexts with uncertainties emanating from different causes. To understand these well, numerous examples¹ are provided in the form of python code snippets and jupyter notebooks. All of these are intgerated in a digital twin² of the underlying cyberinfrastructure, the so-called digital continuum.

¹ These examples cover all aspects, often in a more general context, but their application to the CDT context is always presented.

² Known as the CDT

The Continuum Digital Twin (CDT) is defined and described in the series of forthcoming papers (Garénaux-Gruau, Martineau, et al. 2026; Garénaux-Gruau, Certenais, et al. 2026; Garénaux-Gruau, Bodin, et al. 2026). For optimization and scheduling, there are numerous excellent references. Among these we point out particularly (Birge and Louveaux 2011), (Pinedo 2022), and (Powell 2022). Most of the codes are based on the wonderful pyomo framework (Bynum et al. 2021), (Hart et al. 2011) and (Postek et al. 2025). General background on exascale workflows can be found in (Asch et al. 2018).

Finally, in (Asch 2022) there are basic explanations of optimization, uncertainty quantification, inverse problems and their use for digital twins.

This book explains all the tools needed for formulating and implementing digital twins.

Author

Mark Asch is Emeritus Professor of the Université de Picardie Jules Verne, Mathematics department.

https://markasch.github.io/DT-tbx-v1/

https://github.com/markasch/

http://masch.perso.math.cnrs.fr/

mark.asch@u-picardie.fr

Citation

Asch, Mark. Optimal Scheduling for Cross-Facility Workflows. Online (2026) https://markasch.github.io/RCP4CDT/

@book{Asch2026
    title = {Optimal {S}cheduling for {C}ross-{F}acility {W}orkflows},
    author = {Asch, Mark},
    url = {https://markasch.github.io/RCP4CDT/},
    year = {2026},
    publisher = {Online}
}

License

This online book is frequently updated and edited. It’s content is free to use, licensed under a Creative Commons licence, and the code can be found on GitHub. A physical copy of the book will be available at a later date.

License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

References

Asch, Mark. 2022. A Toolbox for Digital Twins: From Model-Based to Data-Driven. Society for Industrial; Applied Mathematics. https://doi.org/10.1137/1.9781611976977.

Asch, M, T Moore, R Badia, et al. 2018. “Big Data and Extreme-Scale Computing: Pathways to Convergence-Toward a Shaping Strategy for a Future Software and Data Ecosystem for Scientific Inquiry.” The International Journal of High Performance Computing Applications 32 (4): 435–79. https://doi.org/10.1177/1094342018778123.

Birge, John R., and François Louveaux. 2011. Introduction to Stochastic Programming. Second edition. Springer New York, NY. https://doi.org/10.1007/978-1-4614-0237-4.

Boyd, Stephen, and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press.

Bynum, Michael L., Gabriel A. Hackebeil, William E. Hart, et al. 2021. Pyomo–Optimization Modeling in Python. Third. Vol. 67. Springer Science & Business Media.

Ferreira da Silva, Rafael, Rosa M. Badia, Deborah Bard, Ian T. Foster, Shantenu Jha, and Frederic Suter. 2024. “Frontiers in Scientific Workflows: Pervasive Integration With High-Performance Computing .” Computer (Los Alamitos, CA, USA) 57 (08): 36–44. https://doi.org/10.1109/MC.2024.3401542.

Garénaux-Gruau, Marius, François Bodin, and Mark Asch. 2026. Continuum Digital Twin—Mathematical Models. https://arxiv.org/abs/26xx.xxxx.

Garénaux-Gruau, Marius, Mathis Certenais, Laurent Morin, François Bodin, and Mark Asch. 2026. Continuum Digital Twin Use Cases. https://arxiv.org/abs/26xx.xxxx.

Garénaux-Gruau, Marius, Olivier Martineau, François Bodin, and Mark Asch. 2026. Continuum Digital Twin Implementation. https://arxiv.org/abs/26xx.xxxx.

Hart, William E, Jean-Paul Watson, and David L Woodruff. 2011. “Pyomo: Modeling and Solving Mathematical Programs in Python.” Mathematical Programming Computation 3 (3): 219–60.

IEA, Paris. 2025. Energy and AI. https://www.iea.org/reports/energy-and-ai.

Nocedal, Jorge, and Stephen J. Wright. 2006. Numerical Optimization. 2. ed. Springer Series in Operations Research and Financial Engineering. Springer.

Pinedo, Michael L. 2022. Scheduling: Theory, Algorithms, and Systems. 6th edition. Springer Cham.

Postek, Krzysztof, Alessandro Zocca, Joaquim Gromicho, and Jeffrey Kantor. 2025. Hands-On Mathematical Optimization with Python. Cambridge University Press. https://doi.org/10.1017/9781009493512.

Powell, Warren B. 2022. Reinforcement Learning and Stochastic Optimization: A Unified Framework for Sequential Decisions. John Wiley & Sons.

Qi, Qinglin, Fei Tao, Tianliang Hu, et al. 2021. “Enabling Technologies and Tools for Digital Twin.�� Journal of Manufacturing Systems 58: 3–21. https://doi.org/https://doi.org/10.1016/j.jmsy.2019.10.001.

Shapiro, Alexander, Darinka Dentcheva, and Andrzej Ruszczyński. 2009. Lectures on Stochastic Programming. SIAM, Society for Industrial; Applied Mathematics. https://doi.org/10.1137/1.9780898718751.

Unat, Didem, Anshu Dubey, Emmanuel Jeannot, and John Shalf. 2025. “The Persistent Challenge of Data Locality in the Post-Exascale Era.” Computing in Science & Engineering 27 (4): 19–27. https://doi.org/10.1109/MCSE.2025.3567586.