Testing the Addition of an Immunotherapy Drug, Tremelimumab, to the PARP Inhibition Drug, Olaparib, for Recurrent Ovarian, Fallopian Tube or Peritoneal Cancer

Inclusion Criteria: * Patients must have platinum-sensitive, recurrent high-grade serous or high-grade endometrioid (grade 3) ovarian, primary peritoneal, or fallopian tube cancer. Patients with other histologies are also eligible, provided that the patient has a known deleterious germline or somatic BRCA1 or BRCA2 mutation identified through testing at a clinical laboratory. Submission of BRCA testing results (germline and/or somatic) is required for all patients. * Platinum-sensitive disease defined as no clinical or radiographic evidence of disease recurrence for \> 6 months (or 182 days) after last receipt of platinum-based therapy. The date should be calculated from the last administered dose of platinum therapy. * Patients must have had response (complete or partial) to their prior line of platinum therapy and cannot have had progression through prior platinum-based therapy. * Patients must have RECIST 1.1 measurable disease. Patients with biochemical recurrence based on CA125 levels alone are not eligible. * Prior therapy: * Prior chemotherapy must have included a first-line platinum-based regimen with or without consolidation chemotherapy. * Prior bevacizumab therapy as a component of frontline or recurrent treatment is permitted. * Patients may have received an unlimited number of platinum-based therapies in the recurrent setting. * Patients may have received up to one non-platinum-based line of therapy in the recurrent setting. Prior hormonal therapy will not be counted as this non-platinum-based line. * Prior treatment with a PARP inhibitor: * Patients may not have had a prior PARP inhibitor in the recurrent setting. * Prior use of a PARP inhibitor in the upfront maintenance setting is allowed for women with a confirmed BRCA1 or BRCA2 germline or somatic mutation. * Women who received a PARP inhibitor for maintenance therapy in the frontline setting must have received at least one other chemotherapy regimen for recurrence prior to enrolling on this trial. * Patients who demonstrated disease progression while on a PARP inhibitor are excluded. * Prior hormonal-based therapy for ovarian, primary peritoneal, or fallopian tube cancer is acceptable. * Age \>= 18. * Body weight \> 30 kg. * Eastern Cooperative Oncology Group (ECOG) performance status of 0, 1 or 2 * Absolute neutrophil count (ANC) \>= 1,500/mcl (within 14 days prior to enrollment) * Platelets \>= 100,000/mcl (within 14 days prior to enrollment) * Hemoglobin \>= 10 g/dL (within 14 days prior to enrollment) * Note: blood transfusions are not permitted within 28 days prior to enrollment * Creatinine =\< 1.5 x institutional/laboratory upper limit of normal (ULN) (within 14 days prior to enrollment) * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (within 14 days prior to enrollment) * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 3 times institutional ULN (within 14 days prior to enrollment) * Adequately controlled thyroid function, with no symptoms of thyroid dysfunction and thyroid stimulating hormone (TSH) within normal limits. Thyroid replacement therapy is permitted to achieve a TSH within normal limits. * Patients must be able to swallow and retain oral medications and not have gastrointestinal illnesses that would preclude absorption of olaparib as judged by the treating physician. * Evidence of post-menopausal status or negative urinary or serum pregnancy test for pre-menopausal patients. Women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause. The following age-specific requirements apply: * Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy). * Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \> 1 year ago, had chemotherapy-induced menopause with last menses \> 1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy). * Administration of study drugs (olaparib, tremelimumab) may have an adverse effect on pregnancy and poses a risk to the human fetus, including embryo-lethality. Women of childbearing potential (WOCBP) must agree to use two (2) highly effective forms of contraception from up to 14 days prior to enrollment (for oral contraceptives), during treatment, and for 6 months after the last dose of study medication. * Life expectancy \>= 12 weeks. * Patients with brain metastases are eligible if follow-up brain imaging after central nervous system (CNS)-directed therapy shows no evidence of progression. Imaging studies must have been completed no later than 14 days prior to enrollment. In addition, patients must have been successfully weaned off steroid support. Patients should not have received steroids for the treatment of brain metastases within 14 days prior to enrollment. * Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial * The patient or a legally authorized representative must provide study-specific informed consent prior to study entry and, for patients treated in the United States (U.S.), authorization permitting release of personal health information. Exclusion Criteria: * Active infection requiring antibiotic therapy (except for uncomplicated urinary tract infections), including tuberculosis. * Known clinically significant liver disease, including active viral, alcoholic, or other hepatitis; and cirrhosis. For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated. Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load. * Hormonal therapy directed at treatment for the cancer must be discontinued at least 28 days prior to enrollment. Hormone replacement therapy for symptom management is permitted. * Any other therapy directed at treating the cancer including chemotherapy, biologic/targeted agents, and immunologic agents, unless discontinued at least 28 days prior to enrollment. * Any radiation therapy unless discontinued at least 28 days prior to enrollment. * Major surgical procedure within 28 days prior to enrollment. * Current or prior use of immunosuppressive medication within 14 days before enrollment. The following are exceptions to this criterion: * Intranasal, inhaled, topical steroids, or local steroid injections (i.e. intra-articular injection) * Systemic corticosteroids at physiologic doses not to exceed 10mg/day of prednisone or its equivalent * Steroids as premedication for hypersensitivity reactions (i.e. computed tomography \[CT\] scan contrast allergy premedication). * Patients with active autoimmune disease that has required systemic treatment in the past 2 years (i.e., with use of disease modifying agents, corticosteroids, or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. * Patients with autoimmune disease (e.g., psoriasis, extensive atopic dermatitis, severe asthma, inflammatory bowel disease \[IBD\], multiple sclerosis \[M.S.\], uveitis, vasculitis) requiring concurrent use of any systemic immunosuppressants or steroids are excluded from the study. Patients with vitiligo, mild, intermittent asthma requiring only occasional beta-agonist inhaler use, or mild localized eczema are eligible. * Any patient with an allogeneic (allo)-transplant of any kind is excluded, including xenograft heart valve. * Chronic use of immune-suppressive drugs (i.e. systemic corticosteroids) for the management of cancer or non-cancer related illnesses (i.e. chronic obstructive pulmonary disease \[COPD\]). * Note: ongoing steroid use for the management of brain metastases is not permitted. * History of allergic reactions attributed to compounds of similar chemical or biologic composition to olaparib or tremelimumab. * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, interstitial lung disease, serious chronic gastrointestinal conditions associated with diarrhea, or psychiatric illness/social situations that would limit compliance with study requirements, substantially increase risk of incurring adverse events (AEs) or compromise the ability of the patient to give written informed consent. * Subjects must not have evidence of bowel obstruction on imaging or symptoms consistent with a bowel obstruction. Additional workup to rule this out is not required. * Known potent CYP3A4 inhibitors or inducers must be discontinued prior to starting treatment. * Symptoms associated with toxicities (\> Common Terminology Criteria for Adverse Event \[CTCAE version (v) 5.\] grade 2) caused by prior cancer therapy, excluding alopecia, vitiligo, and the laboratory values defined in the inclusion criteria. * Patients with grade \>= 2 neuropathy will be evaluated on a case-by-case basis after consultation with the Study Chair. * Patients who are receiving any other investigational agent. * Resting electrocardiogram (ECG) with corrected QT interval (QTc) \> 470 msec on two or more time points within a 24-hour period, or a family history of long QT syndrome. If an initial ECG is within normal limits, a repeat ECG is not required. * Patients who have previously received anti-CTLA-4 antibody therapy. * Blood transfusions are not permitted within 28 days prior to study enrollment. * Patients must not have signs or symptoms suggestive of myelodysplastic syndrome or acute myeloid leukemia. * Pregnant or lactating patients * Receipt of live attenuated vaccines within 30 days of enrollment. Note: patients, if enrolled, should not receive live vaccines while receiving study treatment and up to 30 days after the last treatment dose. Inactivated vaccines are permitted.