Endonuclease that preferentially catalyzes the cleavage of double-stranded 5-hydroxymethylcytosine (5hmC)-modified DNA (PubMed:25355512). The 5hmC-modified nucleotide does not increase the binding affinity, but instead increases the efficiency of cutting and specifies the site of cleavage for the modified DNAs (By similarity). Shows significantly higher affinity for four-stranded Holliday junction over duplex and single-stranded DNAs (By similarity). Promotes conservative recombination when the DNA is 5hmC-modified (PubMed:25355512). Promotes autophagy through the suppression of mTOR by its phosphorylation-mediated interaction with YWHAG and its endonuclease activity-mediated DNA damage response (PubMed:33473107). GSK3-beta mediated phosphorylation of ENDOG enhances its interaction with YWHAG, leading to the release of TSC2 and PIK3C3 from YWHAG resulting in mTOR pathway suppression and autophagy initiation (PubMed:33473107). Promotes cleavage of mtDNA in response to oxidative and nitrosative stress, in turn inducing compensatory mtDNA replication (PubMed:29719607). {ECO:0000250|UniProtKB:O08600, ECO:0000269|PubMed:25355512, ECO:0000269|PubMed:29719607, ECO:0000269|PubMed:33473107}. This is the function of ENDOG (endonuclease G, ENSG00000167136).