Retifanlimab With or Without Difluoromethylornithine for the Treatment of Progressive High Grade Gliomas

Inclusion Criteria: * Age ≥ 18 years * Diagnosis of high-grade glioma, including any of the following: * Glioblastoma, IDH-wild type (WT) * Grade 3 or 4 IDH1/2 mutant astrocytoma or * Grade 3 oligodendroglioma * Any prior grade 2 astrocytoma or oligodendroglioma that is suspected to have recurred at a higher grade * Other high-grade glioma * Plan for surgical resection as part of routine clinical care * Radiographic disease progression, with or without tissue confirmation * Measurable disease * Eastern Cooperative Oncology Group (ECOG) performance status (PS) of 0, 1 or 2 and Karnofsky Performance Status (KPS) ≥ 60 * NOTE: PS must be assessed (again) within 7 days prior to first dose of study drug * Hemoglobin ≥ 9.0 g/dL (obtained ≤ 15 days prior to registration) * Absolute neutrophil count (ANC) ≥ 1500/mm\^3 (obtained ≤ 15 days prior to registration) * Platelet count ≥ 100,000/mm\^3 (obtained ≤ 15 days prior to registration) * Total bilirubin ≤ 1.5 x upper limit of normal (ULN) (obtained ≤ 15 days prior to registration) * Alanine aminotransferase (ALT) and aspartate transaminase (AST) ≤ 3 x ULN (≤ 5 x ULN for patients with liver involvement) (obtained ≤ 15 days prior to registration) * Calculated creatinine clearance ≥ 45 ml/min using the Cockcroft-Gault formula (obtained ≤ 15 days prior to registration) * Negative pregnancy test done ≤ 7 days prior to registration, for persons of childbearing potential only * Provide written informed consent for the current study * Willing to provide consent for the Neuro-oncology biorepository (IRB 12-003458) for archiving of tissue, cerebrospinal fluid (CSF), and/or blood samples * Ability to complete forms by themselves or with assistance * Willing to return to enrolling institution for follow-up (during the Active Monitoring Phase of the study) Exclusion Criteria: * Any of the following because this study involves an investigational agent, the genotoxic, mutagenic, and teratogenic effects of which on the developing fetus and newborn are unknown: * Pregnant persons * Nursing persons * Persons of childbearing potential or able to father a child who are unwilling to employ adequate contraception * Uncontrolled intercurrent illness that by the judgement of the investigator would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the regimens including, but not limited to: * ongoing or active infection (e.g., pneumonia, sepsis, etc.) requiring systemic therapy * current diagnosis or previous history of immune-related (non-infectious) pneumonitis or interstitial lung disease that requires or required steroids * active autoimmune disease that required systemic treatment other than replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroids) ≤ 2 years prior to registration * symptomatic congestive heart failure * unstable angina pectoris * psychiatric illness/social situations that would limit compliance with study requirements (e.g., drug addiction) * concurrent active Hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] positive and/or detectable hepatitis B virus \[HBV\] deoxyribonucleic acid \[DNA\]) and Hepatitis C virus (defined as anti-hepatitis C virus \[HCV\] antibody \[Ab\] positive and detectable HCV ribonucleic acid \[RNA\]) infection EXCEPTIONS: * Patients with evidence of hepatitis B virus (HBV) infection (HBsAg positive) must have completed at least 4 weeks of HBV antiviral therapy, and the HBV viral load must be undetectable at the time of registration * Patients with a history of hepatitis C virus (HCV) are eligible if they have an undetectable HCV viral load. Patients must have completed curative anti-viral treatment ≥ 4 weeks prior to registration. * NOTE: Patients without symptoms or prior history do not require testing prior to registration * Co-morbid systemic illnesses or other severe concurrent disease that would make the patient inappropriate for entry into the study or interfere with proper assessment of safety and toxicity * History of myocardial infarction ≤ 6 months prior to registration or congestive heart failure requiring use of ongoing maintenance therapy for life-threatening ventricular arrhythmias * Active autoimmune disease that has required systemic treatment (other than replacement therapy) ≤ 1 year prior to registration * History of allogeneic stem cell transplant * Receiving any other investigational agent with therapeutic intent * Participants who are unable to swallow the DFMO solution or who are at risk for impaired absorption of oral medication. * NOTE: This restriction includes, but is not limited to, refractory vomiting, gastric resection/bypass, and duodenal/jejunal resection * Patients with known hypersensitivity or allergy to DFMO or retifanlimab * Contraindication to MRI or administration of gadolinium