Pembrolizumab in Treating Younger Patients With Recurrent, Progressive, or Refractory High-Grade Gliomas, Diffuse Intrinsic Pontine Gliomas, Hypermutated Brain Tumors, Ependymoma or Medulloblastoma

Inclusion Criteria: * INCLUSION CRITERIA FOR STRATA A, B, D AND E * Tumor: patient must have one of the following diagnoses to be eligible: * Stratum A, currently closed to enrollment: Patients must have a recurrent, progressive or refractory DIPG following radiation therapy with or without chemotherapy * Histologic diagnosis is not required for patients with typical imaging findings of DIPG (defined as patients with a diffuse expansile mass centered in and involving at least 2/3 of the pons); patients with brainstem tumors who have undergone biopsy with a diagnosis of high-grade glioma or diffuse infiltrating glioma are also eligible * Stratum B: Patients must have a histologically confirmed diagnosis of a non-brainstem high-grade glioma (NB-HGG) that is recurrent, progressive or refractory following therapy which included radiotherapy; spinal primary disease is eligible * Stratum D: Patients must have a histologically confirmed diagnosis of ependymoma that is recurrent, progressive or refractory following therapy which included radiotherapy * Stratum E: Patients must have a histologically confirmed diagnosis of medulloblastoma that is recurrent, progressive or refractory following therapy which included radiotherapy * Patients must have adequate pre-trial formalin-fixed paraffin-embedded (FFPE) tumor material available for use in the biology studies mutational analysis and genome wide sequencing for each stratum * Patients with DIPG who have tissue available are requested to submit similar tissue as patients in other strata; however, this is not required for eligibility * All subjects must have measurable disease in 2-dimensions on MRI scan of the brain; disease should be consistently measured with the two largest perpendicular dimensions * Patient must be \>= 1 but =\< 18 years of age at the time of enrollment during the safety portion. Patients \< 22 may be enrolled during the efficacy portion of the study. * Patients must have received prior radiation therapy and/or chemotherapy and recovered from the acute treatment related toxicities (defined as =\< grade 1 if not defined in eligibility criteria) of all prior chemotherapy, immunotherapy or radiotherapy prior to entering this study; there is no upper limit to the number of prior therapies that is allowed * Patients must have received their last dose of known myelosuppressive anticancer therapy at least three (3) weeks prior to study enrollment or at least six (6) weeks if prior nitrosourea * Biologic or investigational agent (anti-neoplastic): Patient must have received their last dose of the investigational or biologic agent \>= 7 days prior to study enrollment * For agents that have known adverse events occurring beyond 7 days after administration, this period must be extended beyond the time during which adverse events are known to occur; the duration must be discussed with and approved by the study chair * Monoclonal antibody treatment and/or agents with prolonged half-lives: Patient must have recovered from any acute toxicity potentially related to the agent and received their last dose of the agent \>= 28 days prior to study enrollment * Patient must have completed immunotherapy (e.g. tumor vaccines, oncolytic viruses, etc.) at least 42 days prior to enrollment * Patients must have had their last fraction of: * Craniospinal irradiation \>= 3 months prior to enrollment * Other substantial bone marrow irradiation \>= 6 weeks prior to enrollment * Local palliative radiation therapy (XRT) (small port) \>= 2 weeks * Patient must be \>= 12 weeks since autologous bone marrow/stem cell transplant prior to enrollment * Patients must be fully recovered from all acute effects of prior surgical intervention * Both males and females of all races and ethnic groups are eligible for this study * Patients with neurological deficits should have deficits that are completely stable for a minimum of 1 week (7 days) prior to enrollment * Karnofsky performance scale (KPS for \> 16 years of age) or Lansky performance score (LPS for =\< 16 years of age) assessed within two weeks of enrollment must be \>= 70; patients who are unable to walk because of neurologic deficits, but who are up in a wheelchair, will be considered ambulatory for the purpose of assessing the performance score * Absolute neutrophil count \>= 1000 cells/uL * Platelets \>= 75,000 cells /uL (unsupported, defined as no platelet transfusion within 7 days) * Hemoglobin \>= 8 g/dl (may receive transfusions) * Total bilirubin =\< 1.5 times institutional upper limit of normal (ULN) * Alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional upper limit of normal * Albumin \>= 2 g/dl * Serum creatinine based on age/gender as noted below; patients that do not meet the criteria below but have a 24 hour creatinine clearance or glomerular filtration rate (GFR) (radioisotope or iothalamate) \>= 70 ml/min/1.73 m\^2 are eligible * Age: 1 to \< 2 years; maximum serum creatinine (mg/dL): 0.6 (male); 0.6 (female) * Age: 2 to \< 6 years; maximum serum creatinine (mg/dL): 0.8 (male); 0.8 (female) * Age: 6 to \< 10 years; maximum serum creatinine (mg/dL): 1 (male); 1 (female) * Age: 10 to \< 13 years; maximum serum creatinine (mg/dL): 1.2 (male); 1.2 (female) * Age: 13 to \< 16 years; maximum serum creatinine (mg/dL): 1.5 (male); 1.4 (female) * Age: \>= 16 years; maximum serum creatinine (mg/dL): 1.7 (male); 1.4 (female) * Pulse oximetry \> 93% on room air and no evidence of dyspnea at rest * Human immunodeficiency virus (HIV)- infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial * Patients must be off all colony-forming growth factor(s) for at least 1 week prior to registration (e.g. filgrastim, sargramostim, erythropoietin); 2 weeks must have elapsed for long-acting formulations * Patients must be willing to use brief courses (at least 72 hours) of steroids as directed for potential inflammatory side effects of the therapy if recommended by their treating physician * Female subjects of childbearing potential must not be pregnant or breast-feeding; female patients of childbearing potential must have a negative serum or urine pregnancy test within 72 hours prior to receiving the first dose of study medication; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required; pregnant women are excluded from this study because pembrolizumab (MK-3475) is an agent with the potential for teratogenic effects; because there is unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with pembrolizumab (MK-3475), breastfeeding should be discontinued if the mother is to be treated with pembrolizumab (MK-3475) * Patients of childbearing or child fathering potential must be willing to use 2 methods of birth control or be surgically sterile or abstain from heterosexual activity while being treated on this study and for 4 months after the last dose of study medication * The patient or parent/guardian is able to understand the consent and is willing to sign a written informed consent document, inclusive of assent where appropriate, according to institutional guidelines * STRATUM C: Diagnosis of hypermutated brain tumors Patients with brain tumors and increased tumor mutation burden as determined by * Confirmed diagnosis of CMMRD syndrome by Clinical Laboratory Improvement Act (CLIA)-certified germline gene sequencing OR * Confirmation of high mutation burden by whole genome/exome sequencing performed in a CLIA-certified laboratory and/or the use of Foundation One next generation sequence panel or another CLIA approved targeted sequencing lab with publicly available correlations between number of mutations found in the panel and mutations per megabase and/or genome; for protocol purposes a high mutation burden will be defined as at least 180 non-synonymous coding-region mutations by whole exome/genome sequencing (well above two standard deviations of the number of median similar mutations described in pediatric CNS cancers) AND/OR a high tumor mutation burden (TMB) or intermediate TMB based on the reporting parameters of the panel; TMB parameters provided for the Foundation One panel are as follows: high TMB is \>= 20 mutations per megabase or intermediate TMB is between 6 to 19 mutations per megabase OR * Confirmed diagnosis of Lynch syndrome by CLIA-certified germline gene sequencing; patients with Lynch syndrome will not be accounted for in primary objective unless their tumors are determined to have the minimum number of mutations described above but they will still be eligible for this study * Low-grade tumors in patients with CMMRD or Lynch syndrome do not have to reach the threshold of 100 mutations for study inclusion * STRATUM C: Patients must have a histologically confirmed primary brain tumor that is recurrent, progressive or refractory; inclusion criteria encompasses all types of brain tumors (e.g. gliomas, embryonal tumors or any other type of brain tumor as long as other eligibility criteria are met; * Patients with high-grade gliomas are eligible for this clinical trial at least 2 weeks after completion of radiotherapy independent of tumor progression/recurrence as long as they are not enrolled on any other therapeutic clinical trial and there is macroscopic residual disease * Patients with other concomitant tumors associated with CMMRD and Lynch syndrome including gastrointestinal polyps/adenomas and carcinomas, lymphomas and leukemias will be eligible as long as they are not requiring anticancer therapy directed against these other cancers and meet all other eligibility criteria * STRATUM C: Patients must have adequate pre-trial FFPE tumor material available and be willing to provide a blood sample for use in the genome wide sequencing studies; while tissue is required for genome-wide sequencing of tumor and germline samples, patients will be deemed eligible for the study with a minimum of approximately 10 unstained slides for the planned analysis * STRATUM C: Subjects must have measurable disease in 2-dimensions on MRI scan of the brain and/or spine with the exception allowed for non-progressed HGGs; disease should be consistently measured with the two largest perpendicular dimensions * STRATUM C: Patient should be \< 30 years at the time of enrollment * STRATUM C: Patients must have received prior radiotherapy and/or chemotherapy with the following exceptions: * Patients with secondary CNS cancers after a previous medical problem/malignancy who cannot receive full dose of radiotherapy (\> 50 Gy) as long as they meet all other eligibility criteria * Patients with progressive low-grade gliomas and CMMRD or Lynch syndrome Patients must have recovered from the acute treatment related toxicities (defined as =\< grade 1 if not defined in eligibility criteria) of all prior chemotherapy, immunotherapy or radiotherapy prior to entering this study; there is no upper limit to the number of prior therapies that is allowed * STRATUM C: Patients must have received their last dose of known myelosuppressive anticancer therapy at least three (3) weeks prior to study enrollment or at least six (6) weeks if prior nitrosourea * STRATUM C: Patient must have received their last dose of the investigational or biologic agent \>= 7 days prior to study enrollment * For agents that have known adverse events occurring beyond 7 days after administration, this period must be extended beyond the time during which adverse events are known to occur; the duration must be discussed with and approved by the study chair * STRATUM C: Monoclonal antibody treatment and/or agents with prolonged half-lives: Patient must have recovered from any acute toxicity potentially related to the agent and received their last dose of the agent \>= 28 days prior to study enrollment * STRATUM C: Patient must have completed immunotherapy (e.g. tumor vaccines, oncolytic viruses, etc.) at least 42 days prior to enrollment * STRATUM C: Patients must have had their last fraction of: * Craniospinal irradiation \>= 3 months prior to enrollment * Other substantial bone marrow irradiation \>= 6 weeks prior to enrollment * Local palliative XRT (small port) \>= 2 weeks * STRATUM C: Patient must be: * \>= 12 weeks since autologous bone marrow/stem cell transplant prior to enrollment * \>= 5 years since allogeneic stem cell transplant prior to enrollment with no evidence of active graft versus (vs.) host disease * STRATUM C: Patients must be fully recovered from all acute effects of prior surgical intervention * STRATUM C: Both males and females of all races and ethnic groups are eligible for this study * STRATUM C: Patients with neurological deficits should have deficits that are completely stable for a minimum of 1 week (7 days) prior to enrollment * STRATUM C: Karnofsky performance scale (KPS for \> 16 years of age) or Lansky performance score (LPS for =\< 16 years of age) assessed within two weeks of enrollment must be \>= 60; patients who are unable to walk because of neurologic deficits, but who are up in a wheelchair, will be considered ambulatory for the purpose of assessing the performance score * STRATUM C: Absolute neutrophil count \>= 1000 cells/uL * STRATUM C: Platelets \>= 75,000 cells/uL (unsupported, defined as no platelet transfusion within 7 days) * STRATUM C: Hemoglobin \>= 8 g/dl (may receive transfusions) * STRATUM C: Total bilirubin =\< 1.5 times institutional upper limit of normal (ULN) * STRATUM C: ALT (SGPT) =\< 3 x institutional upper limit of normal * STRATUM C: Albumin \>= 2 g/dl * STRATUM C: Serum creatinine based on age/gender as noted below; patients that do not meet the criteria below but have a 24 hour creatinine clearance or GFR (radioisotope or iothalamate) \>= 70 ml/min/1.73 m\^2 are eligible * Age: 1 to \< 2 years; maximum serum creatinine (mg/dL): 0.6 (male); 0.6 (female) * Age: 2 to \< 6 years; maximum serum creatinine (mg/dL): 0.8 (male); 0.8 (female) * Age: 6 to \< 10 years; maximum serum creatinine (mg/dL): 1 (male); 1 (female) * Age: 10 to \< 13 years; maximum serum creatinine (mg/dL): 1.2 (male); 1.2 (female) * Age: 13 to \< 16 years; maximum serum creatinine (mg/dL): 1.5 (male); 1.4 (female) * Age: \>= 16 years; maximum serum creatinine (mg/dL): 1.7 (male); 1.4 (female) * STRATUM C: Pulse oximetry \> 93% on room air and no evidence of dyspnea at rest * STRATUM C: HIV- infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial * STRATUM C: Patients must be off all colony-forming growth factor(s) for at least 1 week prior to registration (i.e. filgrastim; sargramostim; erythropoietin); 2 weeks must have elapsed for long-acting formulations * STRATUM C: Patients must be willing to use brief courses (at least 72 hours) of steroids as directed for potential inflammatory side effects of the therapy if recommended by their treating physician * STRATUM C: Female subjects of childbearing potential must not be pregnant or breast-feeding; female patients of childbearing potential must have a negative serum or urine pregnancy test within 72 hours prior to receiving the first dose of study medication; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required; pregnant women are excluded from this study because pembrolizumab (MK-3475) is an agent with the potential for teratogenic effects; because there is unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with pembrolizumab (MK-3475), breastfeeding should be discontinued if the mother is to be treated with pembrolizumab (MK-3475) * STRATUM C: Patients of childbearing or child fathering potential must be willing to use 2 methods of birth control or be surgically sterile or abstain from heterosexual activity while being treated on this study and for 4 months after the last dose of study medication * STRATUM C: The patient or parent/guardian is able to understand the consent and is willing to sign a written informed consent document, inclusive of assent where appropriate, according to institutional guidelines Exclusion Criteria: * EXCLUSION CRITERIA FOR STRATA A, B, D AND E * Concurrent Illness * Patients with active autoimmune disease or documented history of autoimmune disease/syndrome that requires ongoing systemic steroids or systemic immunosuppressive agents, except * Patients with vitiligo or resolved asthma/atopy * Patients with hypothyroidism stable on hormone replacement or Sjogren's syndrome * History of or ongoing pneumonitis or significant interstitial lung disease Note: This would include non-infectious pneumonitis that required steroid use * Patients with any clinically significant unrelated systemic illness (serious infections or significant cardiac, pulmonary, hepatic or other organ dysfunction), that in the opinion of the investigator would compromise the patient's ability to tolerate protocol therapy, put them at additional risk for toxicity or would interfere with the study procedures or results * Patients with other current malignancies * Patients with known hypermutated brain tumors including those with CMMRD and Lynch syndrome are ineligible for enrollment in Strata A, B, D and E * Patients who have received a solid organ transplant * Patients with bulky tumor on imaging are ineligible; treating physicians are encouraged to contact the study chair to request a rapid central imaging review to confirm fulfilment of these eligibility criteria, if they have concerns Bulk tumor is defined as: * Tumor with evidence of clinically significant uncal herniation or midline shift * Tumor with diameter of \> 5 cm in one dimension on T2/fluid attenuated inversion recovery (FLAIR) * Tumor that in the opinion of the site investigator, shows significant mass effect in either the brain or spine * Multi-focal/ metastatic disease: Note: Multiple foci of enhancement in a single FLAIR abnormality is permissible and will not exclude the subject * Patients with multi-focal parenchymal disease are ineligible * Patients with leptomeningeal metastatic disease are eligible; this includes disease that is discrete from the primary lesion but that has a radiographic appearance consistent with leptomeningeal spread, rather than likely trans-parenchymal spread * Strata B, D and E - patients whose tumor has a significant component involving the brainstem or with significant fourth ventricular compression are ineligible * Patients who are receiving any other anti-cancer or investigational drug therapy are ineligible * Patients who have a known active hepatitis B or hepatitis C infection are ineligible; patient must have documented evidence of negative tests for the presence of hepatitis B surface antigen and hepatitis C (anti-hepatitis C virus \[HCV\] antibody OR hepatitis \[Hep\] C RNA-qualitative) * Patients who have received the last vaccination of a live vaccine =\< 30 days prior to enrollment are ineligible; examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella, yellow fever, rabies, bacillus Calmette-Guerin (BCG), and typhoid (oral) vaccine; seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g. Flu-Mist \[registered trademark\]) are live attenuated vaccines, and must meet timeline for live vaccine * Patients with a history severe (\>= grade 3) hypersensitivity reaction to a monoclonal antibody are ineligible * Patients who have received previous therapy with an anti-CTLA4, anti-CD137