PEP-CMV + Nivolumab for Newly Diagnosed Diffuse Midline Glioma/High-grade Glioma and Recurrent Diffuse Midline Glioma/High-grade Glioma, Medulloblastoma, and Ependymoma

Inclusion Criteria for All Patients: * Patients must be ≥4 and ≤25 years of age (inclusive) at the time of study enrollment * Metastatic Disease: Patients with M+ disease are eligible. * Adequate bone marrow function defined as: * ANC (Absolute neutrophil count) ≥ 1000/µl. * Platelets ≥ 75,000/µl. * Hemoglobin \> 8 g/dL. (may be supported) * Adequate renal function defined as: * Creatinine clearance or radioisotope GFR ≥ 70ml/min/1.73 m\^2 OR A serum creatinine based on age/gender as listed in the protocol. Note: The threshold creatinine values in this table were derived from the Schwartz formula for estimating GFR utilizing child length and stature data published by the CDC. * Adequate liver function defined as: * Total bilirubin ≤1.5 times institutional ULN * AST(SGOT) ≤3 × institutional upper limit of normal * ALT(SGPT) ≤3 × institutional upper limit of normal * The effects of PEP-CMV and nivolumab on the developing human fetus are unknown. For this reason, female participants of childbearing potential and male participants who are sexually active must agree to use adequate contraception prior to study entry, for the duration of study participation, and for at least 5 months after completion of study participation. Should a female participant become pregnant or suspect she is pregnant while participating in this study, or should a male participant suspect he has fathered a child, s/he must inform the treating physician immediately. * Ability to understand and willingness to sign an IRB approved written informed consent document. Legally authorized representatives may sign and give informed consent on behalf of study participants. All patients and/or their parents or legal guardians must sign an IRB approved written informed consent document. Inclusion Criteria for Patients with Newly-Diagnosed High-Grade Gliomas (HGG) and Newly-Diagnosed (DMG) (Stratum I): * Stratum I patients must have histologically confirmed, newly-diagnosed HGG (such as anaplastic astrocytoma, glioblastoma) or newly-diagnosed DMG (such H3K27M mutant diffuse midline glioma). * Patients with a newly-diagnosed HGG/DMG must enroll within 6 weeks of their final dose of standard radiation therapy with or without chemotherapy. * Patients with primary spinal cord tumors are eligible * Patients with a radiographically typical diffuse intrinsic pontine glioma (DIPG), a subset of DMG, defined as a tumor with a pontine epicenter and diffuse involvement of more than 2/3 of the pons, are eligible without histologic confirmation. * Patients with brainstem lesions that do not meet these radiographic criteria will be eligible if there is histologic confirmation of an infiltrating astrocytoma WHO grades II-IV (excluding grade II BRAF altered gliomas). * Karnofsky \>50 for patients \> 16 years of age and Lansky \>50 for patients \<16 years of age. Patients who are unable to walk because of paralysis, but who are up in a wheelchair, will be considered ambulatory for the purpose of assessing the performance score. * Patients must have adequate neurologic function defined as: * Patients with neurological deficits should have deficits that are stable for a minimum of 1 week prior to enrollment. * Patients must have received no prior therapy other than surgery, radiation, chemotherapy during radiotherapy and/or steroids (dexamethasone with goal to wean dexamethasone throughout protocol therapy), with the following qualifications: * Patients must enroll within 42 days of their final dose of standard radiation therapy. * Patients with a newly diagnosed high-grade glioma or DMG must enroll within 42 days of their final dose of standard of care radiation therapy with or without chemotherapy. * Patients with HGG or DMG are permitted, but not required, to have received chemotherapy during radiation. Bevacizumab is permitted prior to enrollment in patients with DMG or HGG. Patients must have received their last dose of bevacizumab at least 14 days prior to enrollment. * For HGG patients, Patients must have received radiotherapy at a standard dose of \~54 Gy in \~1.8 Gy fractions for approximately 6 weeks with an acceptable variance of 10%. Radiation therapy must have begun no later than 30 days after the date definitive surgery. * For patients with DIPG, Patients must have received radiotherapy at a standard dose of radiotherapy of \~54 Gy in \~1.8 Gy daily fractions for approximately 6 weeks with an acceptable variance rate of 10%. Radiation therapy must have begun no later than 30 days after the date of radiographic diagnosis or biopsy. * For patients with spinal cord HGG/DMG: Patients must have received radiotherapy at a standard dose of \~54 Gy in \~1.8 Gy fractions for approximately 6-7 weeks with an acceptable variance of 10%. * For patients with metastatic disease: Patients may have received standard dose craniospinal therapy. Inclusion Criteria for patients with recurrent/progressive HGG/DMG (stratum II) or recurrent /progressive MB or EPN (stratum III): * Recurrent MB, EPN, DMG or HGG: Patients must have a diagnosis of medulloblastoma that is recurrent, progressive or refractory. All patients must have histological verification of a MB, EPN, DMG or HGG at either original diagnosis or relapse. * Patients must have adequate pretrial tumor material available (except DMG). * Patients must have measurable disease defined as a lesion that can be measured in two perpendicular diameters on MRI. * Karnofsky ≥ 50% for patients \>16 years of age or Lansky ≥ 50 for patients ≤ 16 years of age. Patients who are unable to walk because of paralysis, but who are up in a wheelchair, will be considered ambulatory for the purpose of assessing the performance score. * Adequate neurologic function defined as: * Patients with neurological deficits should have deficits that are stable for a minimum of 2 weeks prior to enrollment. * Patients with current seizure disorders may be enrolled if seizures are well-controlled on antiepileptic therapies. * Previous enrollment and treatment on an interventional clinical trial(s) is allowed. * Patients must have received prior disease-directed therapy including radiotherapy for their initial diagnosis of MB, EPN, HGG, or DMG, unless the patient had a supratentorial EPN with GTR and radiation was not deemed necessary by the treating team. * Patients must have had their last fraction of: * Craniospinal irradiation (if completed), total body irradiation or radiation to ≥ 50% of pelvis \> 3 months prior to enrollment. * Focal irradiation \> 4 weeks prior to enrollment. * Patients must have received their last dose of myelosuppressive anticancer therapy at least 21 days prior to enrollment. * Patients must have received their last dose of any immunotherapy agents at least 30 days prior to enrollment. * Patients must have received their last dose of non-myelosuppressive anticancer agents at least 7 days prior to study enrollment. * Patients must have received their last dose of any antibodies at least 21 days prior to enrollment. * Patients must have received their last dose of hematopoietic growth factors at least 14 days prior to enrollment for a long-acting growth factor (e.g. pegfilgrastim) or 7 days prior to enrollment for short-acting growth factor. * At least 90 days must have elapsed after an autologous stem cell infusion. Exclusion Criteria: * Pregnant or breast-feeding women will not be entered on this study due to known or unknown risks of fetal and teratogenic adverse events as seen in animal/human studies. Pregnancy tests must be obtained in girls and women who are post-menarchal at least 7 days prior to study enrollment. Males or females of reproductive potential may not participate unless they have agreed to use an effective contraceptive method, which includes abstinence. * Active infection requiring treatment. * Patients with malignancy related to HIV or solid organ transplant: known history of HIV, HBV surface antigen positivity or positive HCV antibody are not eligible. Viral testing is not required unless clinically indicated in patients without a known history * Known immunosuppressive disease. * Patients with active unrelated systemic illness including but not limited to renal, hepatic cardiac (congestive cardiac failure, myocardial infarction, myocarditis), or moderate to severe pulmonary problems generally defined by need for medical intervention (e.g., oxygen, medications) and/or limiting activities of daily living (generally CTCAE Grade 2 or higher) or shortness of breath with limited exertion are not eligible. Pulmonary conditions include (but are not limited to) COPD, asthma, and hemi-pneumectomy * Patients receiving concomitant immunosuppressive agents for medical conditions; inhaled corticosteroids for asthma are allowed. * Patients receiving concomitant tumor-directed therapy. * Patients receiving any other investigational drug therapy. * Previous enrollment and treatment on an interventional clinical trial (Stratum 1 only). * Patients on dexamethasone \> 0.1 mg/Kg/day up to maximum dose of 4 mg/day or equivalent. * Patients with inability to return for follow-up visits or obtain follow-up studies required to assess toxicity to therapy. * Patients at high risk for imminent neurologic decline due to extensive bulk disease, midline shift, or herniation on MRI. These patients should be discussed with the study chairs.