HNF-DDA: subgraph contrastive-driven transformer-style heterogeneous network embedding for drug–disease association prediction

Abstract

Background

Drug–disease association (DDA) prediction aims to identify potential links between drugs and diseases, facilitating the discovery of new therapeutic potentials and reducing the cost and time associated with traditional drug development. However, existing DDA prediction methods often overlook the global relational information provided by other biological entities, and the complex association structure between drug diseases, limiting the potential correlations of drug and disease embeddings.

Results

In this study, we propose HNF-DDA, a subgraph contrastive-driven transformer-style heterogeneous network embedding model for DDA prediction. Specifically, HNF-DDA adopts all-pairs message passing strategy to capture the global structure of the network, fully integrating multi-omics information. HNF-DDA also proposes the concept of subgraph contrastive learning to capture the local structure of drug-disease subgraphs, learning the high-order semantic information of nodes. Experimental results on two benchmark datasets demonstrate that HNF-DDA outperforms several state-of-the-art methods. Additionally, it shows superior performance across different dataset splitting schemes, indicating HNF-DDA’s capability to generalize to novel drug and disease categories. Case studies for breast cancer and prostate cancer reveal that 9 out of the top 10 predicted candidate drugs for breast cancer and 8 out of the top 10 for prostate cancer have documented therapeutic effects.

Conclusions

HNF-DDA incorporates all-pairs message passing and subgraph capture strategies into heterogeneous network embedding, enabling effective learning of drug and disease representations enriched with heterogeneous information, while also demonstrating significant potential for applications in drug repositioning.

Background

The development of a small molecule drug from design to market approval typically takes an average of 15 years and approximately $2 billion in investment [1]. In addition to the high costs of research and development, the clinical trial phase for new drugs has a failure rate as high as 90% [2]. Consequently, the number of newly approved drugs is insufficient to meet the needs of treating increasingly complex diseases. In 2022, the FDA approved only 37 novel drugs [3, 4]. New strategies are urgently needed to reduce costs and shorten the development cycle to enhance drug discovery efficiency. Compared to traditional drug discovery methods, drug repositioning identifies new indications for already approved clinical drugs, thereby avoiding the complex and costly drug design process and the high failure rates associated with clinical trials. Drug repositioning significantly improves drug discovery efficiency [5,6,7,8,9,10,11] and has been successfully applied in treating diseases such as COVID-19 [12] and Alzheimer’s disease [13]. With the advancement of computer technology and the massive accumulation of biomedical data, many computational methods have been applied in the field of biomedicine [14,15,16,17,18,19,20], which computational virtual screening for new drug indications has gradually gained attention [21, 22]. Utilizing machine learning models to predict reliable potential drug–disease associations (DDAs) can substantially reduce the human and material costs associated with traditional experiments [23,24,25]. Therefore, computational methods for predicting DDAs have become crucial for accelerating drug discovery.

Computational methods for predicting DDAs can be categorized into two types: drug-based and disease-based methods and multi-source heterogeneous data-based methods [26]. The first type predicts potential DDAs by constructing a drug-disease bipartite network and leveraging known drug–disease association patterns [27,28,29,30,31]. For instance, NCH-DDA [27] employs single-neighborhood and multi-neighborhood feature extraction modules to extract critical features of drugs and diseases from both the drug-disease bipartite network and drug/disease similarity networks in parallel, utilizing contrastive learning to obtain common features. DRAGNN [28] uses a graph attention mechanism to obtain dynamically allocated attention coefficients for nodes, enhancing the effectiveness of information gathering for target nodes. However, these methods have a limitation: the mechanisms of drug action and disease pathology involve multiple biomolecules and signaling pathways. By focusing solely on the direct associations between drugs and diseases, these methods neglect the biological mechanisms involving other entities, such as proteins, in DDAs.

The second category, multi-source heterogeneous data-based methods, integrates data from various biological entities to capture potential associations between drugs and diseases. These methods can be divided into three types based on data integration strategies: path-based, network embedding-based, and knowledge graph embedding-based. Path-based methods use walk strategies, such as random walks, to generate node sequences that capture relationships between different types of nodes and edges, thereby learning the representations of drug and disease nodes [32,33,34]. For example, DREAMwalk [32] proposed a “semantic multilayer association induction” method, which uses random walks guided by semantic information to generate node sequences populated by drugs and diseases. FuHLDR [25] obtains low-order features based on graph convolutional networks and high-order features based on meta-paths, then integrates these high-order and low-order representations to determine a comprehensive representation of drugs and diseases. However, these meta-path-based features often rely on local information and have limited ability to extract higher-order structures, making it difficult to capture the complex interaction mechanisms between drugs and diseases. Network embedding-based methods construct a heterogeneous network containing various biological entities and then use graph representation learning techniques to capture the network structure and learn node feature representations [35,36,37,38,39]. For example, PSGCN [35] proposed an end-to-end specific partner drug repositioning method based on graph convolutional networks. DDAGDL [24] incorporates complex biological information into the topology of heterogeneous networks, effectively learning smooth representations of drugs and diseases through an attention mechanism. These methods use graph convolutional networks (GCN) or graph attention networks (GAT) to integrate information from neighboring nodes but overlooks the all-pairs message passing between nodes [40]. Knowledge graph embedding-based methods view associations in the knowledge graph as transformations from source entities to target entities [41,42,43]. For example, RotatE [41] models relationships between entities as rotations in the complex plane. Although knowledge graph embedding techniques can map entities and relationships in the graph into a low-dimensional vector space, this representation method may lose some structural and semantic information.

Considering the limitations of the existing methods, we propose a subgraph contrastive-driven transformer-style heterogeneous network embedding model (HNF-DDA) for DDA prediction (Fig. 1). First, we construct a heterogeneous network encompassing various biological entities and employ the attribute information of these entities to obtain initial node embeddings using a biological large language model. Second, to learn the embeddings of drugs and diseases, HNF-DDA employs an all-pairs message passing heterogeneous network embedding model to capture global signal transmission between any nodes. A subgraph capture strategy is proposed to extract high-order semantic structures within the heterogeneous network. Finally, an eXtreme Gradient Boosting (XGBoost) classifier [44, 45] is employed to predict the association probabilities between drugs and diseases. Experiments conducted on real-world datasets demonstrate that HNF-DDA outperforms existing methods in AUROC, AUPR, and Accuracy. Results from experiments with different dataset splitting schemes indicate that HNF-DDA has superior generalization capability for new drug and disease categories. Therefore, HNF-DDA not only effectively learns the representations of drugs and diseases that contain heterogeneous information but also shows greater potential for application in drug repositioning. This study makes the following contributions:

Fig. 1

The overview of the HNF-DDA framework. We input the SMILES of drugs, the sequences of proteins, and the textual descriptions of other biological entities into a biological language model to obtain the initial features of the nodes. The HNFormer module is then used to derive the embeddings of drugs and diseases. Next, we employ XGBoost for multiple independent training sessions. We average the predicted scores from these multiple runs and perform a ranking analysis based on the average scores

• To obtain multi-source biological entity information, we employ a large-scale biological language model to generate initial embeddings for drug structures, protein sequences, diseases, and other biological entity attributes.

• To achieve global information transmission in heterogeneous networks, we utilize an all-pairs message-passing Transformer-style network embedding model that simulates signal transmission between any nodes, enabling adaptive integration of various biological entity information.

• To better capture the complex association mechanisms between drugs and diseases, we propose a drug-disease subgraph contrastive strategy that ensures better connections between drugs and diseases in the embedding space.

• Experimental results demonstrate that HNF-DDA outperforms state-of-the-art methods. Additionally, split experiment results and case studies on breast cancer and prostate cancer confirm the model’s generalization and reliability, offering new insights for drug repositioning.

Results and discussions

Datasets

We evaluated our model on two benchmark datasets: KEGG [46] and HetioNet [34]. Both datasets contain drug, protein, disease, pathway entities and multi-type association information. The statistics of the two datasets are shown in Table 1.

Table 1 The statistics of KEGG and HetioNet

Baselines

In this study, we compared HNF-DDA with 10 state-of-the-art DDA prediction methods:

• RotatE [41]: This model introduces a new knowledge graph embedding method capable of modeling and inferring various relational patterns, including symmetric/antisymmetric, inversion, and composition, for learning drug and disease embeddings.

• QuatE [47]: This method introduces a more expressive hypercomplex representation to model entities and relationships in knowledge graph embeddings, learning node embeddings.

• WalkPool [48]: This algorithm combines the expressive power of topology-based heuristic algorithms with the feature learning capabilities of neural networks.

• SEAL [49]: The model proposes a novel decaying heuristic theory that unifies a broad range of heuristic algorithms within a single framework. It demonstrates that all these heuristic algorithms can be well-approximated from local subgraphs, which retain rich information about the existence of links.

• ComplEx [50]: This method demonstrates that using the asymmetric Hermitian product as a relational operation can automatically understand the structural knowledge of large knowledge bases and address the link prediction problem.

• DTi2vec [51]: The model constructs a heterogeneous network and employs node embedding techniques to automatically generate features for each drug and target, subsequently using ensemble learning techniques to identify drug-target interactions.

• NEWMIN [33]: This method proposes a network embedding framework within multiple networks to predict synergistic drug combinations.

• DDAGDL [24]: This method incorporates complex biological information into the topology of heterogeneous networks, effectively learning smooth representations of drugs and diseases through an attention mechanism.

• DREAMwalk [32]: This model proposes a “semantic multi-layer guilt-by-association” method, which predicts DDAs at the drug-gene-disease level using the relational guilt principle “similar genes share similar functions.”

• FuHLDR [25]: This methods propose a novel graph representation learning model for drug repositioning by fusing higher and lower-order biological information.

The baselines we selected can be categorized into random walk-based, graph neural network-based, and knowledge graph-based link prediction models. The random walk-based models include FuHLDR, DREAMwalk, NEWMIN, and DTi2vec; the graph neural network-based models include DDAGDL, WalkPool, and SEAL; the knowledge graph embedding models, including CompIEx, RotatE, and QuatE.

Experimental setting and evaluation metrics

We use known DDAs as positive samples and randomly sample an equal number of negative drug-disease pairs as negative samples. Then, we evaluate the performance of HNF-DDA and other methods on the two datasets using tenfold cross-validation repeated 10 times, with different dataset splits for each experiment. Since the dimensions of the initial features of different biological entities are different, we first convert the initial embedding to the same size of the input embedding, set the size of the input embedding to 512, and set the size of the hidden layer embedding to 32; the number of layers for the all-pair message-passing encoder is 2 for the KEGG dataset and 1 for the HetioNet dataset; the weights of the learning objectives, $\alpha$ and $\beta$, are both 0.01 for KEGG, and 1.0 and 0.1 for HetioNet, respectively.

The evaluation metrics include AUROC, AUPR, and Accuracy.

$$TPR=Recall=\frac{TP}{TP+FN}$$

(1)

$$FPR=\frac{FP}{FP+TN}$$

(2)

$$Precision=\frac{TP}{TP+FP}$$

(3)

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(4)

where TP is the number of samples correctly classified as positive, TN is the number of samples correctly classified as negative, FP is the number of samples incorrectly classified as positive, and FN is the number of samples incorrectly classified as negative. Accuracy is the proportion of all samples that are correctly predicted. AUROC is the Area Under the TPR-FPR Curve plotted at different thresholds, and AUPR is the Area Under the Precision-Recall Curve plotted at different thresholds. We comprehensively evaluate the performance of HNF-DDA using AUROC, AUPR, and Accuracy.

Performance comparison

To evaluate the performance of HNF-DDA, we compared it with baselines on two datasets. Figure 2 shows the results of HNF-DDA and other baselines using tenfold cross-validation 10 times. HNF-DDA achieved an average accuracy of 0.8897, AUROC of 0.9507, and AUPR of 0.9491 on both biomedical heterogeneous network datasets, outperforming the best-performing baselines, DREAMwalk (average accuracy of 0.8704, AUROC of 0.9382, AUPR of 0.9353), and FuHLDR (average accuracy of 0.8823, AUROC of 0.9462, AUPR of 0.9445). Therefore, HNF-DDA outperformed all compared baselines, achieving higher performance across all evaluation metrics. It is noteworthy that the KEGG dataset contains more drug, disease, and DDA data than the HetioNet dataset. HNF-DDA’s performance improvement over the DREAMwalk and FuHLDR model in KEGG (accuracy by 3.5% and 0.13%, AUROC by 2.2% and 0.185%, AUPR by 2.0% and 0.195%) and in HetioNet (accuracy by 0.8% and 1.59%, AUROC by 1.1% and 0.14%, AUPR by 0.9% and 0.13%).

Fig. 2

The drug–disease association prediction performances of each model on the KEGG and HetioNet. A DDA prediction performance on KEGG dataset. B DDA prediction performance on HetioNet dataset. All methods are repeated 10 times, and the black short lines in the figures represent error bars. The specific experimental data sets can be found in Additional file 2

The best baselines DREAMwalk and FuHLDR both generate meta-paths based on the idea of random walks, thereby capturing the topological information of nodes in the network. However, since random walks tend to frequently visit nodes that are close to each other, while the probability of visiting distant nodes is low, the captured topological structure is more biased towards locality. Moreover, random walks only rely on the topological structure of the network and cannot directly capture high-order semantics. The all-pair message-passing mechanism proposed by HNF-DDA can capture the potential relationship between any nodes and learn the global information of nodes in the network; the subgraph contrastive learning module proposed by HNF-DDA can capture the high-order semantic information of the drug-disease subgraph and learn the local information of nodes in the network; in addition, the individual attribute information of nodes is learned using a biological large language model. Therefore, HNF-DDA integrates multi-source heterogeneous information from multiple perspectives, captures the potential association between drugs and diseases, and improves the prediction performance of DDA.

Predictive potential for unknown drug/disease classes

To evaluate the effectiveness of the model in real-world drug repositioning scenarios, we compare HNF-DDA with DDAGDL, DREAMwalk, and FuHLDR model, the best-performing baselines, through DDA split prediction experiments on the KEGG dataset (as shown in Fig. 3). First, we classified all drug and disease entities: drugs are classified according to their ATC codes, and diseases are classified according to the highest MeSH (Medical Subject Headings) term category. Then, based on the categories of drugs or diseases, we divide the DDAs into train, validation, and test sets in an approximate ratio of 8:1:1. This forced the model to predict the DDA probability for unknown drug or disease categories. We repeated the division 10 times to ensure that the data sets differed in each split.

Fig. 3

Performance of DDAGDL, DREAMwalk, FuHLDR, and HNF-DDA based on split experiments in the dataset KEGG. A Performance of drug split experiment. B Performance of disease split experiment. Note: The bold numbers in the figure represent the average results of 10 experiments. The specific experimental data sets can be found in Additional file 2

As shown in Fig. 3, DREAMwalk outperforms the DDAGDL and FuHLDR in the split experiment. DREAMwalk with average accuracy of 0.7818, AUROC of 0.8868, and AUPR of 0.8976. HNF-DDA with average accuracy of 0.7961, AUROC of 0.9014, and AUPR of 0.8935. In the disease split experiment, DREAMwalk with average accuracy of 0.6190, AUROC of 0.6900, and AUPR of 0.7009. HNF-DDA with average accuracy of 0.6648, AUROC of 0.7955, and AUPR of 0.7829. These results demonstrate that HNF-DDA has greater potential to accurately predict unknown drug or disease categories in real-world scenarios compared to DREAMwalk. Additionally, as shown in Fig. 3, the distribution of prediction results across 10 experiments indicates that HNF-DDA has better stability in prediction performance. Due to the insufficient number of DDAs in the HetioNet dataset, which makes it difficult to perform category-based split experiments, no such experiments are conducted.

Ablation experiments

To comprehensively validate the predictive performance of the HNF-DDA model, we conducted ablation experiments. We created the following variants targeting the heterogeneous network encoder and learning objective modules of the HNF-DDA model:

• w/o link: Remove the edge-level learning objective.

• w/o sub: Remove the subgraph-level learning objective.

• w/o class: Remove the node-level learning objective.

• w/o init_feat: Replace the initial embeddings learned by the large language model with one-hot encoding.

• GAT: Replace the all-pair message passing encoder with a GAT.

• GCN: Replace the all-pair message passing encoder with a GCN.

From the results in Fig. 4, HNF-DDA performs the best on the KEGG and HetioNet datasets. From the overall trend shown in Fig. 4, the changes in KEGG are relatively minor. As seen in Table 1, this is because the KEGG dataset contains a larger number of known DDA samples for training, which allows the model to fully learn the drug–disease association patterns. Therefore, the effects of different experimental conditions are minimal, and the performance metrics are higher than those of HetioNet.

Fig. 4

Performance of HNF-DDA and different variants on KEGG and HetioNet datasets. The specific experimental data sets can be found in Additional file 2

Comparing the results of w/o link, w/o sub, w/o class, and w/o init_feat, it can be observed that the results of w/o sub and w/o init_feat are relatively worse. The w/o sub results indicate that the subgraph capture module proposed by HNF-DDA effectively mines the potential associations between drugs and diseases. The w/o init_feat results suggest that HNF-DDA effectively integrates semantic information of biological entities and heterogeneous network structure, and replacing the semantic features learned by the large language model may degrade the predictive performance. The comparison of GAT and GCN with HNF-DDA indicates that the all-pair message passing encoder used by HNF-DDA can effectively capture signals between any pair of nodes in the heterogeneous network, integrating multiple sources of heterogeneous information comprehensively and enhance the prediction performance. These experimental results demonstrate the effectiveness of the all-pair message passing and subgraph capture modules proposed by HNF-DDA.

Visualization of embeddings

We visualize the learned heterogeneous network node embeddings using T-sne [52]. Figure 5A and B show the visualization results of node embeddings on KEGG and HetioNet, respectively. Figure 5C and D show the visualization results of node embeddings on KEGG and HetioNet after removing the subgraph capture module. In Fig. 5A, drug clusters (red) are relatively close to disease clusters (blue) and relatively far from pathway clusters (yellow), in Fig. 5C, after removing the subgraph module, the pathway clusters (yellow) are situated between the drug clusters (red) and disease clusters (blue). Compared to Fig. 5B, D removing the subgraph module, drug clusters (red) are relatively close to pathway clusters (yellow) and relatively far from disease clusters (blue). These results indicate that the subgraph capture module can uncover potential associations between drugs and diseases, bringing them closer in the embedding space, which benefits the improvement of downstream DDA prediction performance.

Fig. 5

HNF-DDA embedding visualization experiment. A Visualization on KEGG. B Visualization on HetioNet. C Visualization on KEGG without subgraph module. D Visualization on HetioNet without subgraph module

Case study

To further validate the reliability of DDA predictions by HNF-DDA in drug repositioning, we conduct literature verification on candidate drugs for breast cancer and prostate cancer from the KEGG dataset. Firstly, we average the prediction scores of all DDAs obtained from tenfold cross-validation repeated 10 times, ensuring different data splits for each tenfold cross-validation. Then, we sort the predicted scores of all unknown DDAs. Finally, we select the top 10 candidate drugs with the highest predicted scores for breast cancer and prostate cancer for literature validation analysis. Table 2 lists candidate drugs and the corresponding literature reports.

Table 2 Top 10 candidates of HNF-DDA for breast cancer and prostate cancer

As shown in Table 2, among the top 10 candidate drugs for breast cancer predicted by HNF-DDA, 9 have supporting literature evidence. For prostate cancer, 8 out of the top 10 candidate drugs have supporting literature evidence. Among the candidate drugs for breast cancer and prostate cancer, seven drugs overlap (including drugs for which there is no literature evidence). These drugs are either chemotherapy drugs that can treat various types of cancer (such as Etoposide and Vincristine sulfate) or can play a supportive role in managing cancer, or the symptoms and side effects related to its treatment. For example, Ephedrine is mainly used as a bronchodilator and decongestant and can sometimes be used in supportive care to manage low blood pressure during surgery or chemotherapy; Desmopressin is primarily used to treat diabetes insipidus and certain bleeding disorders but can also be used to manage bleeding complications in cancer patients; Prednisone, a corticosteroid, is used to treat various diseases, including inflammation, autoimmune diseases, and as part of certain chemotherapy regimens. The other three non-overlapping drugs also have specific related literature reports. These results further demonstrating the reliability of HNF-DDA in practical disease applications.

Discussion

HNF-DDA shows multiple improvements over existing SOTA models. It outperforms other models in prediction accuracy in different scenarios of both datasets, including predictions in new drug or disease categories. This study highlights the importance of using large language models and capturing both global and local structures of heterogeneous networks for DDA prediction. Although contrastive learning methods has shown creativity, the quality of negative samples limits the prediction performance.

Although we use a subgraph capture strategy to preserve the local structure of nodes and learn the high-level semantic information of nodes, the subgraph negative samples obtained through the random replacement strategy have relatively low interference and discriminability. Additionally, the classifier used in HNF-DDA is also trained on drug-disease negative pairs generated by random sampling, which may result in false negative pairs. Therefore, in our future work, we plan to investigate sampling strategies for negative samples to obtain more realistic and reliable negative samples for more accurate DDA prediction.

Conclusions

We propose HNF-DDA, a subgraph contrast-driven transFormer-based Heterogeneous Network embedding model for predicting drug–disease associations (DDAs). HNF-DDA utilizes an all-pair message passing strategy to preserve the global information of heterogeneous network nodes, enabling the integration of multi-omics data. It also proposes a subgraph capture module to retain the local structure of drug-disease subgraphs, learning the multiple high-level semantic information. Experimental results on two benchmark datasets demonstrate that HNF-DDA outperforms 10 state-of-the-art methods. Dataset split experiments reveal HNF-DDA’s potential in predicting DDAs for novel drug or disease categories. Ablation and visualization experiments indicate that the proposed all-pairs message passing and subgraph capturing strategies effectively reveal latent associations between drugs and diseases, enhancing DDA prediction performance. Finally, a literature validation analysis of the top 10 candidate drugs for breast and prostate cancer confirms the reliability of HNF-DDA in identifying candidate drugs. In summary, our model, HNF-DDA, offers a powerful tool for drug-disease prediction.

Methods

The main objective of this paper is to predict associations between drugs and diseases. We propose a subgraph contrastive-driven transFormer-style Heterogeneous Network embedding model, HNF-DDA, as illustrated in Fig. 1. First, we construct a biomedical heterogeneous network and utilize a biological language model to learn the initial embeddings of the heterogeneous network nodes. Next, we learn the embeddings of drugs and diseases using all-pairs message passing and subgraph capture strategies. Finally, we employ an XGBoost classifier to predict the association probabilities between drugs and diseases. Table 3 is a summary of all notations used in the “Methods” section.

Table 3 Summary of all notations

Biomedical heterogeneous network

In this study, we construct a biomedical heterogeneous network using the interactions between biological entities. The nodes represent drugs, diseases, proteins, and other biological entities, while the edges represent the relationships between these entities. A schematic diagram of this network is shown in Fig. 1.

A biomedical heterogeneous network is defined as an undirected network $G=(V, E, \mathcal{A},\mathcal{R})$ and $N=|V|$ represent number of nodes. $V=\{{v}_{1}, {v}_{2}, \dots , {v}_{N}\}$ is the set of nodes in the network, where ${v}_{i}\in V$ represents the $i$ node in the network. $E=\{\left({v}_{i}, {v}_{j}\right)|{v}_{i}, {v}_{j}\in V\}\subseteq V\times V$ is the set of edges in the network, and each edge represents the interaction or association that exists between two nodes, where $\left({v}_{i}, {v}_{j}\right)$ represents the connection between nodes $i$ and $j$. $\mathcal{A}=\{{a}_{1}, {a}_{2}, \dots , {a}_{N}\}$ is the set of attributes of a node, including the SMILES structure of drugs, protein sequences, and biological text descriptions of diseases and other biological entities, where ${a}_{i}\in \mathcal{A}$ represents the attribute feature associated with node ${v}_{i}$. $\mathcal{R}$ is the set of type of a node, we describe the type of each node by a mapping function $\tau :V\to R$, namely: $\tau \left({v}_{i}\right)\in \{Drug, Protein, Disease, Others\}$, where $\tau \left({v}_{i}\right)$ represents the type of node ${v}_{i}$. Additionally, we removed all DDAs from the biomedical heterogeneous network, allowing the integration of network structure and biological entity semantic information during the heterogeneous network embedding process without relying on drug-disease treatment information. These DDAs will serve as supervisory information for the DDA prediction task using the XGBoost classifier.

Computing initial embeddings

The biomedical heterogeneous network contains various types of biological entities. We employ specific model for each type of entity based on their attributes to extract semantic information, which serves as the initial embeddings for the nodes in the heterogeneous network. This approach integrates external knowledge with the structure of the heterogeneous network. These models utilized to compute the initial embeddings are outlined below:

• We utilize the SMILES as the attribute information for drugs. SMILES encodes the structure of a molecule into a string of characters, with each character representing information about atoms, bonds, and rings [92, 93]. This encoding provides a comprehensive description of the molecular structure, including the connections between atoms, ring structures, and stereochemistry. We employ a language model for drug molecules, MolFormer [94], to obtain embeddings from the drug’s SMILES. MolFormer employs masked language modeling and combines linear attention Transformers with rotary embeddings.

• We utilize amino acid sequence data as the attribute information for proteins. This data comprises a sequence of characters that represent the specific amino acids constituting a protein and their sequential arrangement. Each amino acid is represented by a letter, and the sequence can reflect the protein’s structure, function, and activity. We employ a pre-trained protein model, ProtBert [95], to obtain initial embeddings from the protein sequence data. ProtBert is based on the BERT [96] architecture and encodes amino acid sequences into token-level or sentence-level representations, which can be used for downstream protein tasks, such as contact prediction.

• We utilize biological text descriptions as attribute information for diseases and other biological entities. We employ a biomedical text language model, BiomedBERT [97], to obtain initial embeddings. BiomedBERT is based on the BERT architecture and is pre-trained from scratch using text abstracts from PubMed and full-text articles from PubMed Central as its corpus.

Finally, we obtain the initial embeddings ${H}^{init}=\{{H}_{t}^{init}\in {\mathbb{R}}^{\left|{V}_{t}\right|\times {d}_{t}}\}$, where $t\in \{Drug, Protein, Disease, Others\}$. Details of these models are in the Additional file 1.

Heterogeneous network embedding

This section introduces the heterogeneous network embedding (HNFormer) module of HNF-DDA. In this module, we employ a Transformer-style graph embedding method and design a subgraph capture strategy to learn the embeddings of heterogeneous network nodes.

All-pair message passing encoder

The mechanisms of drug action and disease pathology involve various types of biomolecules and signaling pathways. Additionally, the known edges in the biological heterogeneous network are incomplete, and many potential associations exist between nodes. Therefore, signal transmission in a heterogeneous network should not be limited to entities of the same type or local entity relationships. Inspired by NodeFormer [40], we employ an all-pairs message passing encoder to enable signal transmission between any pair of entities in the heterogeneous network, ensuring the full integration of multi-source heterogeneous information.

First, we utilize multiple MLPs to map the initial embeddings ${H}^{init}$ of different type node into the same space:

$${H}_{t}^{0}=\sigma \left({W}_{t}^{init}{H}_{t}^{init}+{B}_{t}^{init}\right),$$

(5)

where ${H}_{t}^{0}\in {\mathbb{R}}^{\left|{V}_{t}\right|\times d}$, $t\in \{Drug, Protein, Disease, Others\}$, and $\sigma (\bullet )$, ${W}_{t}^{init}$, ${B}_{t}^{init}$ represent Exponential Linear Units activation function, weight, bias parameter, respectively. We concatenate the embeddings of different types to form the complete node embeddings ${H}^{0}\in {\mathbb{R}}^{N\times d}$:

$${H}^{0}=\left[\begin{array}{c}\begin{array}{c}{H}_{Drug}^{0}\\ {H}_{Protein}^{0}\end{array}\\ \begin{array}{c}{H}_{Disease}^{0}\\ {H}_{Others}^{0}\end{array}\end{array}\right],$$

(6)

where ${z}_{u}^{0}\in {H}^{0}$ represents the $u$ th node representation vector in the 0 layer, and 0 represents initial representation vector of heterogeneous network.

Next, for any node $u$, we use ${z}_{u}^{(l)}$ to represent its corresponding representation vector at layer $l$. Thus, the update for the next layer ${z}_{u}^{(l+1)}$ is:

$${z}_{u}^{(l+1)}=\sum_{s=1}^{N}\frac{\text{exp}({({q}_{u})}^{T}{k}_{s})}{{\sum }_{w=1}^{N}\text{exp}({({q}_{u})}^{T}{k}_{w})}\bullet {v}_{s},$$

(7)

where ${k}_{u}={W}_{K}^{(l)}{z}_{u}^{(l)}$, ${q}_{u}={W}_{Q}^{(l)}{z}_{u}^{(l)}$, ${v}_{u}={W}_{V}^{(l)}{z}_{u}^{(l)}$ are obtained from the feature transformation of the $l$ th layer, and ${W}_{K}^{(l)}$, ${W}_{Q}^{(l)}$, and ${W}_{V}^{(l)}$ are learnable parameters in $l$ th layer. Equation (7) can be viewed as a graph attention mechanism defined on a fully connected graph where all nodes are pairwise connected.

Because for any node, it is necessary to calculate the attention of the other $N$ nodes separately. Therefore, using a kernel method approximate the exponential-then-dot operation, which is $\text{exp}\left({a}^{T}b\right)=\kappa (a,b)\approx \phi {\left(a\right)}^{T}\phi (b)$, where $\phi : {\mathbb{R}}^{d}\to {\mathbb{R}}^{m}$ is a low-dimensional feature map (RF). For example, the commonly used Positive Random Feature (PRF) [98] can be defined as:

$$\phi \left(x\right)=\frac{\text{exp}(\frac{-{\Vert x\Vert }_{2}^{2}}{2})}{\sqrt{m}}\left[\text{exp}\left({w}_{1}^{T}x\right),\dots ,\text{exp}\left({w}_{m}^{T}x\right)\right],$$

(8)

This enables us to rewrite Eq. (7):

$${z}_{u}^{(l+1)}=\sum_{s=1}^{N}\frac{\phi {\left({q}_{u}\right)}^{T}\phi ({k}_{s})}{{\sum }_{w=1}^{N}\phi {\left({q}_{u}\right)}^{T}\phi ({k}_{w})}\bullet {v}_{s}=\frac{\phi {\left({q}_{u}\right)}^{T}{\sum }_{s=1}^{N}\phi ({k}_{s}){v}_{s}^{T}}{{\left({q}_{u}\right)}^{T}{\sum }_{w=1}^{N}\phi ({k}_{w})}$$

(9)

In this way, only one computation is needed, the total complexity is kept within $O(N)$.

The above process assumes that each edge has a continuous attention weight. To further consider the “discretization” of edges, for any node $u$, need to find an “optimal” set of neighbors in each layer for information passing. Therefore, treating the attention weights generated by $N$ nodes as a categorical distribution and then sample the neighbor set from it. Specifically, replacing the Softmax in Eq. (7) with Gumbel-Softmax:

$${z}_{u}^{(l+1)}=\sum_{s=1}^{N}\frac{\text{exp}(({q}_{u}^{T}{k}_{u}+{g}_{s})/\mathcal{T})}{{\sum }_{w=1}^{N}\text{exp}(({q}_{u}^{T}{k}_{w}+{g}_{w})/\mathcal{T})}\bullet {v}_{s}, {g}_{u}\sim Gumbel\left(\text{0,1}\right),$$

(10)

Then, following the before approximate using RF with linear complexity:

$${z}_{u}^{(l+1)}\approx \sum_{s=1}^{N}\frac{\phi {\left({q}_{u}/\sqrt{\mathcal{T}}\right)}^{T}\phi {\left({k}_{s}/\sqrt{\mathcal{T}}\right)}^{{{e}{g}_{s/}}\mathcal{T}}}{{\sum }_{w=1}^{N}\phi {\left({q}_{u}/\sqrt{\mathcal{T}}\right)}^{T}\phi {\left({k}_{w}/\sqrt{\mathcal{T}}\right)}^{{{e}{g}_{w/}}\mathcal{T}}}\bullet {v}_{s}=\frac{\phi {\left({q}_{u}/\sqrt{\mathcal{T}}\right)}^{T}{\sum }_{s=1}^{N}{e}^{{g}_{s}/}\mathcal{T}\phi ({k}_{s}/\sqrt{\mathcal{T}}){v}_{s}^{T}}{{\left({q}_{u}/\sqrt{\mathcal{T}}\right)}^{T}{\sum }_{w=1}^{N}{e}^{{g}_{w}/}\mathcal{T}\phi ({k}_{w}/\sqrt{\mathcal{T}})}$$

(11)

In addition to considering the message passing between all node pairs in the network, the topology of the heterogeneous network itself contains a lot of useful information. During each layer of message passing, to strengthen the weights of the observed edges. Therefore, assigning a shared learnable weight to each edge, referred to as relational bias, and update the formula for each layer as follows:

$${z}_{u}^{(l+1)}\leftarrow {z}_{u}^{(l+1)}+\sum_{s,{a}_{us=1}}\sigma ({b}^{(l)})\bullet {v}_{s},$$

(12)

where ${b}^{(l)}$ is the learnable weight parameter corresponding to layer $l$, ${a}_{us=1}$ indicates that there is an association between nodes $u$ and $s$. We can obtain the last layer of node $\text{u}$ embeddings ${\text{z}}_{\text{u}}\in \text{H}$ based on all-pair message passing.

Finally, we employ an MLP to predict the labels of the nodes:

$$\widehat{Y}=MLP(H)$$

(13)

where $\widehat{Y}\in {\mathbb{R}}^{N\times \left|\mathcal{R}\right|}$ represent the predicted probability of the label.

Subgraph structure capture

Drugs act on multiple target proteins and participate in various functional pathways, working together to treat diseases. Therefore, in a heterogeneous network, the relationships between drugs, diseases, and other biological entities collectively form higher-order subgraph structures. In addition to considering signal transmission between nodes in the heterogeneous network, it is crucial to preserve the contextual semantic information contained in the high-order structures of the heterogeneous network. While existing methods often rely on meta-path approaches to explore high-order structures, they may fall short in capturing rich semantics and extracting high-order patterns [99]: (1) Meta paths often focus on single relationships, ignoring the multiple associations between different entities; (2) Starting from a source node, the number of nodes that a meta path can reach is too large, resulting in the extracted structure lacking restrictions and containing insufficient semantic information.

Inspired by CPT-HG [99], we recognize that the mechanisms of drug action and disease pathology involve various types of biomolecules and signaling pathways. Consequently, drugs, diseases, and other biological entities collectively form subgraph with intricate high-order structures. To address this, we construct positive and negative subgraphs and leverage contrastive learning to capture the intricate subgraph structures and subtle contextual semantic information within the heterogeneous network.

Specifically, given a drug (disease) node as the starting node $u$ and a disease (drug) node as the ending node $s$, we take the common first-order neighbors between the drug and the disease as the intermediate node set $C$. We construct subgraphs using only the first-order common neighbors of drugs and diseases as intermediate nodes, capturing the strong associations between drugs and diseases, and enhancing the structural constraints of the subgraph to include rich high-level semantic information. Therefore, the positive subgraph corresponding to node $u$ is:

$${M}_{u}^{+}=\{s\}\cup C$$

(14)

Then, we randomly replace half of the elements in the intermediate node set $C$ to obtain a new set ${C}^{-}$, thus we can obtain the negative subgraph for node $u$:

$${M}_{u}^{-}=\{s\}\cup {C}^{-}$$

(15)

Finally, we apply the concept of contrastive learning to ensure that node $u$ is closer to its positive subgraph embedding and farther from its negative subgraph embedding. The subgraph-level loss objective is defined as:

$${\mathcal{L}}_{sub}=-\frac{1}{\left|{V}^{dd}\right|}\sum_{u\in {V}^{dd}}\frac{\text{exp}\left({H}_{u}f\left({M}_{u}^{+}\right)\right)}{\text{exp}\left({H}_{u}f\left({M}_{u}^{+}\right)\right)+\text{exp}\left({H}_{u}f\left({M}_{u}^{-}\right)\right)},$$

(16)

where ${V}^{dd}$ is the set of drug and disease nodes, and $f(\bullet )$ denotes the pooling function (e.g., average pooling) that gets the subgraph embeddings.

Learning objectives

Given the node labels $Y$ and the predicted labels $\widehat{Y}$, the node-level supervised loss is defined as:

$${\mathcal{L}}_{n}=-\frac{1}{N}\sum_{v\in V}\sum_{r=1}^{\left|\mathcal{R}\right|}{\mathbb{I}}\left[{y}_{v}=r\right]log\widehat{{y}_{v,r}},$$

(17)

where ${\mathbb{I}}[\bullet ]$ is an indicator function. $\widehat{{y}_{v,r}}$ represent the probability that the $v$ th node belongs to the class $r$.

Treating the attention estimates of each layer in the model as a categorical distribution, with the observed edges as samples. Thus, we define an edge-level loss objective using maximum likelihood estimation:

$$\begin{array}{c}{\mathcal{L}}_{e}=-\frac{1}{NL}\sum_{l=1}^{L}\sum_{\left(u,s\right)\in E}\frac{1}{{d}_{u}}log{\pi }_{us}^{\left(l\right)}\\ {\pi }_{us}^{\left(l\right)}=\frac{\phi {\left({q}_{u}\right)}^{T}\phi ({k}_{s})}{\phi {\left({q}_{u}\right)}^{T}{\sum }_{w=1}^{N}\phi ({k}_{w})}\end{array},$$

(18)

where ${d}_{u}$ represents the in-degree of node $u$, and ${\pi }_{us}^{\left(l\right)}$ represents the predicted probability for edge $(u, s)$ at the $l$ th layer of the model.

The final objective is the sum of the node-level, edge-level, and subgraph-level loss:

$$\mathcal{L}={\mathcal{L}}_{n}+\alpha {\mathcal{L}}_{e}+\beta {\mathcal{L}}_{sub},$$

(19)

where $\alpha$ and $\beta$ are weight parameters.

Drug–disease association prediction

After obtaining the embeddings of the heterogeneous network nodes, we utilize the known DDAs as supervision information and predict DDA scores based on drug and disease embeddings using the XGBoost model. To enhance the stability of the prediction results, we conduct multiple independent training sessions with XGBoost and average the resulting prediction scores.

XGBoost (eXtreme Gradient Boosting) is a powerful and widely used machine learning algorithm, primarily designed for supervised learning tasks such as classification and regression [100]. XGBoost is an implementation of gradient boosting machines (GBM), which are a type of ensemble learning method. Ensemble learning methods combine multiple base learners (in this case, decision trees) to improve overall performance. Gradient boosting, specifically, builds models sequentially, where each new model attempts to correct the errors made by the previous models. XGBoost has gained popularity due to its high performance, speed, and scalability. Details of the training procedure and the parameters of XGBoost are in the Additional file 1.