## High-Dimensional and Zero-Inflated: The Fundamental Computational Challenges of Microbiome Data Having engaged in water treatment research for over a decade—from early studies on nitrogen migration and transformation in China to currently leading a team in Singapore evaluating diverse water treatment facilities and aquaculture systems across both regions—I am frequently asked one question. Why do predictive models that perform almost flawlessly in laboratory beakers often exhibit significant fluctuations when applied to actual aquatic environments? Furthermore, why does the transition from laboratory prototype to real-world deployment invariably require years of refinement? The answer fundamentally resides within the underlying data. In microbiome data mining across medicine, aquaculture, and environmental engineering, the construction of stable predictive models is frequently constrained by three primary computational challenges: the curse of dimensionality, extreme sparsity, and the rigid constraints of compositional data. When attempting to establish a digital twin for a water treatment bioreactor, one may be surprised to observe that up to 90% of the data points in the input matrix manifest as "zeros" (Zhang et al., 2024). More problematically, the number of variables to be tracked often exceeds the practically obtainable sample size by several orders of magnitude. Whether investigating the ecological succession of natural water bodies or optimizing the nitrogen removal efficiency of a Recirculating Aquaculture System (RAS), overcoming these seemingly mundane mathematical barriers constitutes a prerequisite for achieving precise regulation in real-world applications. The most immediate constraint is the curse of dimensionality, typically denoted as $p \gg n$. In standard 16S rRNA amplicon or metagenomic sequencing analyses, datasets are inherently high-dimensional; the number of Operational Taxonomic Units (OTUs) or Amplicon Sequence Variants (ASVs) can readily reach hundreds or even thousands (Ragini et al., 2023). However, practical realities intercede. Constrained by substantial experimental costs and protracted sampling cycles, the independent sample sizes available typically remain at relatively modest levels (Ragini et al., 2023). For instance, in a monitoring project spanning several months, engineers may only acquire around a hundred time-series samples, yet each sample is associated with tens of thousands of dimensions of microbial features. This phenomenon, wherein feature dimensions increase exponentially relative to sample size, is termed "ultrahigh-dimensional data" in metagenomics (Ragini et al., 2023). If traditional black-box models, such as Support Vector Machines or Random Forests, are indiscriminately applied, it is highly prone to inducing severe dimensionality penalties and overfitting. Consequently, while the model may perform exceptionally well on the training set, its generalization capacity typically drops precipitously when introduced to unknown, real-world scenarios. An even more formidable challenge than high dimensionality is the extreme sparsity of the data. A salient characteristic of microbiome data is pronounced zero-inflation. In certain complex environments, dense zero counts can account for up to 90% of the data matrix (Zhang et al., 2024). This is not merely an artifact of equipment inaccuracy. The underlying mechanism is twofold. First are "structural zeros," indicating true absence—for instance, obligate anaerobic pathogens cannot survive in an aerobic basin. Second are "sampling-related zeros," where the microorganisms are genuinely present in the aquatic environment but remain undetected due to insufficient sequencing depth or sampling randomness (Lee et al., 2025). Conventional data cleaning algorithms are fundamentally incapable of distinguishing between these two profoundly different scenarios, which subsequently pushes predictive models into a severe signal-to-noise ratio impasse. Extreme zero-inflation renders many traditional statistical testing methods effectively obsolete. In an ideal scenario with no censoring of zero values, one could comfortably rely on non-parametric methods, such as the Wilcoxon rank-sum test or the Kruskal-Wallis test, to evaluate the P-values for differential abundance of individual taxonomic species (Zhang et al., 2024). In practical operations, however, massive volumes of zero values are mechanistically misidentified by underlying models as censored values. This causes these traditional non-parametric tests to frequently encounter severe failures when confronted with zero-inflated data (Zhang et al., 2024). Forcibly applying existing proportion-based analytical methods under these conditions directly exposes issues of low statistical power and significantly inflated Type I error rates (Zhang et al., 2024). Ultimately, the extracted components may merely represent false biomarkers masquerading as environmental noise.
