Nivolumab and Ipilimumab in People With Aggressive Pituitary Tumors

Inclusion Criteria: * Age greater than or equal to 18 * A pituitary adenoma/carcinoma of any histology ° Patients with unresectable tumors that are radiographically (and/or biochemically) consistent with a pituitary adenoma may be considered for enrollment without pathologic confirmation with approval from the principal investigator. * Progression on imaging following radiotherapy ° Patients with pituitary carcinomas in whom there is not felt to be a palliative benefit to treatment with radiotherapy are eligible for enrollment without prior radiotherapy. * Measurable disease by RANO criteria * At least 4 weeks have elapsed since the patient last received temozolomide and the patient must have recovered hematologically from other chemotherapeutics * Karnofsky Performance Status (KPS) greater than or equal to 70 * Screening laboratory values must meet the following criteria: * WBC \>/= 2000/uL * Neutrophils \>/= 1500/uL * Platelets \>/= 100 x 10\^3/uL * Hemoglobin \> 9.0 g/dL * AST/ALT \</=3 x ULN * Total Bilirubin \</= 1.5 x ULN (except subjects with Gilbert Syndrome, who can have total bilirubin \<3.0 mg/dL) * Serum creatinine \</= 1.5 x ULN or creatinine clearance (CrCl) \>/= 40 mL/min using the Cockcroft-Gault formula * Women of childbearing potential (WOCBP) must use appropriate method(s) or contraception. WOCBP should use an adequate method to avoid pregnancy for 23 weeks (30 days plus the time required for nivolumab to undergo five half-lives) after the last dose of investigational drug * WOCBP is defined as any female who has experienced menarche and who has not undergone surgical sterilization (hysterectomy or bilateral oophorectomy) or who is not postmenopausal. Menopause is defined clinically as 12 months of amenorrhea in a woman over 45 in the absence of other biological or physiological causes. Women who are not of childbearing potential are not required to use contraception * Women of childbearing potential must have a negative serum or urine pregnancy test upon study entry * Men who are sexually active with women of childbearing potential must use adequate contraception upon study entry until 31 weeks after the last dose of study treatment. Men who are surgically sterile or azoospermic do not require contraception. Exclusion Criteria: * A corticosteroid requirement of greater than 4mg per day of dexamethasone (or an equivalent dose). NOTE: Patients requiring a physiologic replacement dose of corticosteroids, who may require stress dose corticosteroids, due to adrenal insufficiency are permitted onto this trial * Active, known, or suspected autoimmune disease within the past 2 years. NOTE: Subjects are permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger * Patients should be excluded if they have had prior systemic treatment with a CTLA-4 antibody. Prior treatment with PD1 or PD-L1 antibodies are permitted as long as the patient did not experience serious toxicities requiring treatment discontinuation related to prior PD-1 or PD-L1 therapy * Patients should be excluded if they have a known history of testing positive for hepatitis B virus surface antigen (HBV sAg) or hepatitis C virus antibody (HCV antibody) indicating acute or chronic infection * Patients should be excluded if they have a known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS) * History of allergy to study drug components * History of severe hypersensitivity reaction to any monoclonal antibody * Women who are pregnant or breast-feeding * Inability to undergo radiographic surveillance