Immune Checkpoint Inhibitor Nivolumab in People With Recurrent Select Rare CNS Cancers

* INCLUSION CRITERIA: * Histopathologically proven diagnosis of Ependymoma, Medulloblastoma, Parenchymal Pineal Region Tumors (Pineoblastoma, Pineocytoma, Pineal Tumor of Intermediate Differentiation, Papillary Tumor of the Pineal Region), Choroid Plexus Tumors (Carcinoma, Papilloma, Atypical Papilloma), Histone Mutated Gliomas, Gliomatosis Cerebri, Atypical Teratoid/Rhabdoid Tumor (ATRT), Malignant/Atypical Meningioma\*, Gliosarcoma or Primary CNS Sarcoma, Pleomorphic Xanthoastrocytoma (PXA) and Anaplastic Pleomorphic Xanthoastrocytoma (APXA), and tumors formerly known as Primitive Neuro-Ectodermal Tumors (Embryonal Tumor with Multilayered Rosettes, Medulloepithelioma, Central Nervous System (CNS) Neuroblastoma, CNS Ganglioneuroblastoma, CNS Embryonal Tumor NOS; and tumor entities emerging from methylation profiling of CNS-PNETs: CNS neuroblastoma with Forkhead box protein R2 (FOXR2) activation, CNS Ewing sarcoma family tumor with CIC alteration, CNS high-grade neuroepithelial tumor with meningioma 1 (MN1) alterations, and CNS high-grade neuroepithelial tumor with BCOR alteration) prior to registration. \*Individuals with extra CNS metastases from meningioma will be eligible even if pathology review fails to demonstrate high grade features on available tumor samples. * The tumor tissue (e.g., block or 20 unstained slides) must be available to be sent for immunophenotyping by National Cancer Institute (NCI) Laboratory of Pathology. * Individuals must have progressive tumor growth after having received established standard of care and/or other experimental treatments for their newly diagnosed or recurrent disease. Individuals will be enrolled into 2 different cohorts (cohort 1 or heavily pretreated; cohort 2 or not heavily pretreated). * Age \>= 18 * Karnofsky performance status \>= 70 within 14 days prior to Step 2 registration; Individuals with severe paraparesis/paraplegia who need minimal assistance for selfcare due to their motor deficit but are otherwise functionally independent will be considered eligible. * Adequate hematologic function based on complete blood count (CBC)/differential within 14 days prior to Step 2 registration defined as follows: * Absolute neutrophil count \>= 1,500 cells/mm\^3; * Platelet count \>= 100,000 cells/mm\^3 * Hemoglobin \> 9.0 g/dl (may be transfused to achieve this level) * Adequate renal function within 14 days prior to Step 2 registration defined as follows: * Blood urea nitrogen (BUN) \<= 30 mg/dl and * Serum creatinine \<= 1.7 mg/dl Note: If the serum creatinine is greater than 1.7 mg/dl, a 24-hour urine creatinine clearance will be obtained and if the result of this study is within normal limits\*, the patient would be eligible to enroll onto study. (\*Normal Creatinine Clearance Range: Male: 90 - 130 ml/min; Female: 80 - 125 ml/min) * Adequate hepatic function within 14 days prior to Step 2 registration defined as follows: * Total bilirubin (except patients with Gilbert's Syndrome, who are eligible for the study but exempt from the total bilirubin eligibility criterion) \<= 2.0 mg/dl and * Alanine aminotransferase (ALT) and aspartate aminotransferase (AST) \<= 2.5x ULN * No active or chronic hepatitis infection. Hepatitis C virus (HCV) antibody (for Hepatitis C) and Hepatitis B Surface antigen and Hepatitis B core antibody must be negative. This has been routinely incorporated into immunotherapy trials with checkpoint inhibitors because of concerns that the risk of treatment-induced hepatic injury is increased in the setting of active viral hepatitis. * The individual must not be on a corticosteroid dose greater than physiologic replacement dosing defined as 30 mg of cortisone per day or its equivalent. * The individual must provide study-specific informed consent prior to study entry. No Durable Power of Attorney or Next of Kin can provide initial consent. * The effects of nivolumab on the developing human fetus are unknown. For this reason, women of childbearing potential (WOCBP) must use appropriate method(s) of contraception. WOCBP should use an adequate method to avoid pregnancy for 5 months (30 days plus the time required for nivolumab to undergo five half-lives) after the last dose of investigational drug. NOTE: Based on the evidence cited in Nivolumab IB ver. 20, given that nivolumab is not a genotoxic agent, and that relevant systemic concentrations sufficient to produce a risk of fetal toxicity are not expected in individuals of childbearing potential (IOCBP) partners from exposure to an individual's seminal fluid, men that can father children will not be required to use contraceptive measures and/or a latex or other synthetic condom during sexual activity with an WOCBP partner. EXCLUSION CRITERIA: * Individuals who are receiving any other investigational agents. * Prior use of an immunotherapy such as (but not limited to) a vaccine therapy, dendritic cell vaccine, other checkpoint inhibitors, or intracavitary or convectional enhanced delivery of chemotherapy. * Prior or concurrent malignancy unless its natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen. * Severe, active co-morbidity defined as follows: * Unstable angina within the last 6 months prior to Step 2 registration. * Transmural myocardial infarction within the last 6 months prior to Step 2 registration. * Evidence of recent myocardial infarction or ischemia by the findings of S-T elevations of \>= 2 mm using the analysis of an electrocardiogram (EKG) performed within 14 days prior to Step 2 registration. * New York Heart Association grade II or greater congestive heart failure requiring hospitalization within 12 months prior to Step 2 registration. * History of stroke, cerebral vascular accident (CVA) or transient ischemic attack within 6 months prior to Step 2 registration, with the exception of pericavitary ischemia due to tumor resection. * Serious and inadequately controlled cardiac arrhythmia. * Significant vascular disease (e.g., aortic aneurysm, history of aortic dissection) or clinically significant peripheral vascular disease. * Evidence of bleeding diathesis or coagulopathy. * Serious or non-healing wound, ulcer, or bone fracture or history of abdominal fistula, gastrointestinal perforation, intra-abdominal abscess, major surgical procedure, open biopsy, or significant traumatic injury within 28 days prior to Step 2 registration, with the exception of the craniotomy for tumor resection. * Acute bacterial or fungal infection requiring intravenous antibiotics at the time of registration. * Chronic obstructive pulmonary disease exacerbation or other respiratory illness requiring hospitalization or precluding study therapy at the time of registration. * Hepatic insufficiency resulting in clinical jaundice and/or coagulation defects. * Known acquired immune deficiency syndrome (AIDS) based upon current Centers for Disease Control (CDC) definition; note, however, that human immunodeficiency virus (HIV) testing is not required for entry into this protocol. The need to exclude participants with acquired immunodeficiency syndrome (AIDS) is based on the lack of information regarding the safety of nivolumab in patients with active HIV infection. * Active connective tissue disorders, such as lupus or scleroderma, which in the opinion of the treating physician may put the patient at high risk for immunologic toxicity. * Individuals with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including systemic corticosteroids, should be excluded. These include but are not limited to individuals with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome or chronic inflammatory demyelinating polyneuropathyI (CIDP), myasthenia gravis; systemic autoimmune disease such as systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, hepatitis; and individuals with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or phospholipid syndrome should be excluded because of the risk of recurrence or exacerbation of disease. Of note, individuals with vitiligo, endocrine deficiencies including thyroiditis managed with replacement hormones including physiologic corticosteroids are eligible. Participants with rheumatoid arthritis and other arthropathies, Sjogren's syndrome and psoriasis controlled with topical medication and patients with positive serology, such as antinuclear antibodies (ANA), anti-thyroid antibodies should be evaluated for the presence of target organ involvement and potential need for systemic treatment but should otherwise be eligible. However, individuals with vitiligo, diabetes mellitus, and Hashimoto thyroiditis on appropriate replacement therapy may be enrolled. * Any other major medical illnesses or psychiatric impairments that in the investigator's opinion will prevent administration or completion of protocol therapy. * Allergies and Adverse Drug Reaction: History of allergy to study drug components. * Pregnancy or lactating women due to possible adverse effects on the developing fetus or infant due to study drug. Women of childbearing potential must have a negative serum pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin (HCG) within 24 hours prior to Step 2 registration. * History of severe hypersensitivity reaction to any monoclonal antibody. * Individuals unable to have magnetic resonance imaging (MRIs).