Tremelimumab and Durvalumab in Combination or Alone in Treating Patients With Recurrent Malignant Glioma

Inclusion Criteria: * Patients must have a grade III or IV glioma that has progressed after standard radiotherapy (RT) and temozolomide (TMZ) (Note: Pathology will need to be reviewed locally but registration can occur based on pathology report) * Patients must have had radiographic evidence of tumor progression by brain MRI or computed tomography (CT) scan with contrast * Patients must be \> 12 weeks from completion of radiation therapy unless there is tissue confirmation of tumor recurrence or there is progression outside the radiation treatment field * Prior therapy with gamma knife or other focal high-dose radiotherapy is allowed, but the patient must have subsequent histologic documentation of recurrence, unless the recurrence occurs remote from the treated site * Patients must be surgical candidates * Patients must have had no more than 3 prior lines of chemotherapy; this includes the initial treatment and two relapses; concurrent and adjuvant TMZ-based chemotherapy, including the combination of TMZ with another agent, is considered one line of chemotherapy; for clarification, please contact the principle investigator (PI), Dr. Jeffrey Raizer, at (312) 695-0990 * Patients must be \>= * 4 weeks from TMZ * 6 weeks from a nitrosoureas * 3 weeks from a biologic or targeted agent (i.e. small molecule) * 4 weeks for a vascular endothelial growth factor (VEGF) inhibitor (i.e. bevacizumab) * Patients must exhibit a Karnofsky performance status (KPS) \>= 70 * Life expectancy of \>= 12 weeks (per treating investigator's discretion) * Patients must be on a stable or decreasing dose of corticosteroids within 5 days prior to CT scan or MRI (which is done to determine eligibility); the goal should be dexamethasone 4 mg or less at the time of starting treatment; if patient requires \> 4mg of steroid, please check with the principle investigator (PI); requirement for greater than 10mg of steroid will make the patient ineligible * Leukocytes \>= 3,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcl * Hemoglobin (Hb) \> 10.0 g/dL (can be transfused to this level) * International Normalized Ratio (INR), prothrombin time (PT), or activated partial thromboplastin time (aPTT) as follows: * In the absence of therapeutic intent to anticoagulate the patient: INR \< 1.5 or PT \< 1.5 x upper normal limit (ULN) or aPTT \< 1.5 x ULN * In the presence of therapeutic intent to anticoagulate the patient: INR or PT and aPTT within therapeutic limits (according to the medical standard in the institution) and the patient has been on a stable dose of anticoagulants for at least 2 weeks before registration * Total bilirubin =\< 1.5 x ULN (except in patients with Gilbert's disease) * Aspartate aminotransferase (AST) serum glutamic oxaloacetic transaminase (SGOT)/ alanine aminotransferase (ALT) serum glutamic pyruvic transaminase (SPGT) =\< 2.5 X institutional upper limit of normal (ULN) * Serum creatinine \< 1.5 x ULN * Serum creatinine clearance (CL) \> 40 mL/min by the Cockcroft-Gault formula (Cockcroft and Gault 1976) or by 24-hour urine collection for determination of creatinine clearance * Females of child-bearing potential (FOCBP) and males must agree to use adequate contraception (e.g. hormonal or barrier method of birth control prior to registration, for the duration of study participation, and for 180 days after the last dose of MEDI4736 + tremelimumab combination therapy or 90 days after the last dose of MEDI4736 or tremelimumab monotherapy; should a female patient become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately NOTE: A FOCBP is any woman (regardless of sexual orientation, having undergone a tubal ligation, or remaining celibate by choice) who meets both of the following criteria: * Has not undergone a hysterectomy or bilateral oophorectomy * Has had menses at any time in the preceding 12 consecutive months (and therefore has not been naturally postmenopausal for \> 12 months) * FOCBP must have a negative pregnancy test (serum or urine) within 7 days prior to registration on study * Patients must have the ability to understand and the willingness to sign a written informed consent prior to registration on study * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up * Malignancy treated with curative intent and with no known active disease \>=3 years before the first dose of study drug and of low potential risk for recurrence; NOTE: the exceptions to this requirement include adequately treated non-melanoma skin cancer or lentigo maligna or carcinoma in situ without evidence of disease * Patients can only be on non-enzyme inducing anti-convulsants; if they are on an enzyme inducing anti-convulsant, they may be converted to a non-enzyme inducing anti-convulsants but they will need a 2 week wash out period from time of drug discontinuation until day 1 of study treatment * Patients must have given written, signed and dated informed consent prior to registration on the study; NOTE: no study-specific screening procedures may be performed until consent has been given Exclusion Criteria: * Involvement in the planning and/or conduct of the study (applies to both AstraZeneca staff and/or staff at the study site); previous enrolment or randomization in the present study * Has received prior therapy with an anti-programmed cell death protein 1 (PD-1), anti-programmed cell death ligand 2 (PD-L2), anti-cluster of differentiation (CD)137, or anti-Cytotoxic T-lymphocyte-associated antigen-4 (CTLA-4) antibody (including ipilimumab or any other antibody or drug specifically targeting T-cell co-stimulation or checkpoint pathways) * Has known active hepatitis B (e.g., hepatitis B virus antigen \[HBsAg\] reactive) or hepatitis C (e.g., hepatitis c virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] is detected) * Known active human immunodeficiency virus (HIV1/2 antibodies) * Patient has history of primary immunodeficiency OR has received any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment, excluding intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses; attempts should be made to have patient on lowest possible dose of steroids (acceptable range 4-10mg, please contact PI if dose is \>4 mg) and weaned to off as is feasible * Patients receiving any other investigational chemotherapeutic agents within 30 days prior to the first dose of trial treatment * Mean QT interval corrected for heart rate (QTc) \>= 470 ms calculated from an electrocardiograms (ECGs) using Bazett's Correction; if first ECG is abnormal, then the mean will be calculated from 3 consecutive ECGs (taken 2-5 minutes apart); please contact the PI for further clarification * Active or prior documented history of immunologic disorder including autoimmune disease within the past 2 years NOTE: Subjects with vitiligo, Grave's disease, or psoriasis not requiring systemic treatment (within the past 2 years) are not excluded * Active or prior documented inflammatory bowel disease (e.g., Crohn's disease, ulcerative colitis) * History of allogeneic organ transplant * Uncontrolled intercurrent illness including, but not limited to, * Ongoing or active infection, * Symptomatic congestive heart failure, * Uncontrolled hypertension (defined as \> 150/90) * Unstable angina pectoris, * Cardiac arrhythmia, * Active peptic ulcer disease or gastritis, * Active bleeding diatheses * Psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent * Known history of previous clinical diagnosis of tuberculosis * History of leptomeningeal carcinomatosis * Receipt of live attenuated vaccination within 30 days prior to study entry (or due to receive one within 30 days of receiving either MEDI4736 or tremelimumab) * Female subjects who are pregnant, breast-feeding or male or female patients of reproductive potential who are not employing an effective method of birth control * Any condition that, in the opinion of the investigator, would interfere with evaluation of study treatment or interpretation of patient safety or study results * Subjects with uncontrolled seizures * Patients who have a history of allergic reactions attributed to compounds of similar chemical or biologic composition to MEDI3475 are not eligible