Olaparib, Durvalumab, and Tremelimumab in Treating Patients With Recurrent or Refractory Ovarian, Fallopian Tube or Primary Peritoneal Cancer With BRCA1 or BRCA2 Mutation

Inclusion Criteria: * Patients must have platinum-sensitive or platinum-resistant recurrent or persistent or refractory ovarian, fallopian tube, or primary peritoneal carcinoma AND have one or more of the following characteristics documented on a validated platform (documented genetic test report is required). Historic report is permitted. * A germline BRCA1 or BRCA2 deleterious alteration. * A somatic mutation in BRCA1 or BRCA2 detected in a tumor sample or on circulating tumor DNA * Carry a known or likely loss of function alteration in one or more of the homologous recombination or mismatch repair pathways genes * Demonstrate a genomic phenotype of HR deficiency as measured by a LOH-high score. * Recurrent ovarian cancer is defined as recurrence of disease in a patient who achieved initial complete response to primary therapy * Persistent ovarian cancer is defined as having residual disease in the form of elevated tumor markers or microscopic or clinically evident disease in a patient who has completed and apparently responded to initial chemotherapy * Refractory ovarian cancer is defined as patients who have failed to achieve at least a partial response to therapy including patients with either stable disease or disease progression during primary therapy * Platinum-sensitive is defined as achievement of documented response to initial platinum-based treatment and has been off treatment for an extended period of time (more than 6 months) * Platinum-resistant is defined as relapse within 6 months of last platinum-based chemotherapy or progression while on platinum-based therapy * All patients must have measurable disease as defined by immune-related Response Evaluation Criteria in Solid Tumors (irRECIST); measurable disease is defined as 10 mm in the longest diameter by computed tomography (CT) or magnetic resonance imaging (MRI) scan (or no less than double the slice thickness) for non- nodal lesions and \>= 15 mm in short axis for nodal lesions, 20 mm by chest X-ray, a lymph node must be \>= 15 mm in short axis when assessed by CT scan (CT scan slice thickness recommended to be no greater than 5 mm) * Must have archival tissue available for PD-L1 assessment * Patients should be free of active infection requiring antibiotics (with the exception of uncomplicated urinary tract infection \[UTI\]) * Patients who have the following risk factors are considered to be at increased risk for cardiac toxicities and may be enrolled only with increased monitoring: i) prior treatment with anthracyclines; ii) prior treatment with trastuzumab; iii) a New York Heart Association classification of II controlled with treatment; iv) prior central thoracic radiation therapy (RT), including RT to the heart * Any hormonal therapy being taken as a treatment for cancer must be discontinued at least one week prior to registration; continuation of hormone replacement therapy e.g. thyroid hormone replacement therapy is permitted * Able to tolerate oral medications and no GI illnesses that would preclude absorption of olaparib * Have an Eastern Cooperative Oncology Group (ECOG) performance status of 0-1 * Life expectancy of \> 6 months * Hemoglobin \>= 10 g/dL (no blood transfusion in the 28 days prior to entry \[olaparib guidelines\]) * White blood cell count (WBC) \> 3 x 10\^9/L * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (\>=1500 per mm\^3) * Platelet count \>= 100 x 10\^9/L (\>= 100,000 per mm\^3) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN); this will not apply to subjects with confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only in consultation with their physician * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal unless liver metastases are present, in which case it must be =\< 5 x ULN * Creatinine =\< 1.5 x ULN, serum creatinine clearance (CL) \> 51 ml/min (by the Cockcroft- Gault equation) * Female subjects must either be of non-reproductive potential (i.e., post-menopausal by history: \>= 60 years old and no menses for \>=1 year without an alternative medical cause; OR history of hysterectomy, OR history of bilateral tubal ligation, OR history of bilateral oophorectomy) or must have a negative serum pregnancy test within 28 days of study treatment, confirmed prior to treatment on day 1 * Participants of child-bearing potential must agree to use two highly effective and acceptable forms of contraception from screening, throughout their participation in the study and for 180 days after the last dose of durvalumab + tremelimumab combination therapy or 90 days after last dose of durvalumab or olaparib, whichever is the longer time period (e.g., hormonal or barrier method of birth control); should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately * Participant or legal representative must understand the investigational nature of this study and sign an Independent Ethics Committee/Institutional Review Board approved written informed consent form prior to receiving any study related procedure * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up Exclusion Criteria: * Involvement in the planning and/or conduct of the study (applies to both AstraZeneca staff and/or staff at the study site); previous enrollment in the present study * Participation in another clinical study with an investigational product during the last 4 weeks (prior use of bevacizumab in the upfront setting is allowed) * History of discontinuation of any previous treatment with PARP inhibitors, including olaparib, or a PD-1 or PD-L1 inhibitor, including durvalumab or anti-CTLA4 antibody, including tremelimumab due to toxicity. * Patients with myelodysplastic syndrome/acute myeloid leukemia * History and/or confirmed interstitial lung disease (ILD)/pneumonitis, extensive bilateral lung disease on high-resolution computed tomography (HRCT) scan * Concomitant use of a strong CYP3A inhibitors (e.g., itraconazole, telithromycin, clarithromycin, ketoconazole, voriconazole, nefazodone, posaconazole, ritonavir, lopinavir/ritonavir, indinavir, saquinavir, nelfinavir, boceprevir, telaprevir) and moderate CYP3A inhibitors (e.g., amprenavir, aprepitant, atazanavir, ciprofloxacin, crizotinib, darunavir/ritonavir, diltiazem, erythromycin, fluconazole. Fosamprenavir, imatinib, verapamil) * Concomitant use of strong CYP3A inducers (e.g., phenytoin, rifampicin, carbamazepine, St. John's wort) and moderate CYP3A inducers (e.g., bosentan, efavirenz, etravirine, modafinil, nafcillin) * History of another primary malignancy except for: * Malignancy treated with curative intent and with no known active disease \>= 5 years before the first dose of study drug and of low potential risk for recurrence * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease (e.g. basal cell or squamous cell carcinoma of the skin) * Adequately treated carcinoma in situ without evidence of disease (e.g., breast and cervical cancer in situ) * Receipt of the last dose of anti-cancer therapy (chemotherapy, immunotherapy, endocrine therapy, targeted therapy, biologic therapy, tumor embolization, monoclonal antibodies, radiotherapy or other investigational agent) =\< 21 days prior to the first dose of study drug and within 6 weeks for nitrosourea or mitomycin C) * Mean QT interval corrected for heart rate (QTcF) \>= 470 ms calculated from 3 electrocardiograms (ECGs) using Fridericia's correction * Patients with history of myocardial infarction within 6 months * Current or prior use of immunosuppressive medication within 28 days before the first dose of durvalumab, with the exceptions of intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid * Any unresolved toxicity (Common Terminology Criteria for Adverse Events \[CTCAE\] grade \>= 2) from previous anti-cancer therapy; subjects with irreversible toxicity that is not reasonably expected to be exacerbated by the investigational product may be included (e.g., hearing loss, peripherally neuropathy) * Any prior grade \>= 3 immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE \> grade 1 * Active or prior documented autoimmune disease within the past 2 years; NOTE: Subjects with vitiligo, Grave's disease, or psoriasis not requiring systemic treatment (within the past 2 years) are not excluded * Active or prior documented inflammatory bowel disease (e.g., Crohn?s disease, ulcerative colitis) * Patients with thyroid dysfunction if not adequately controlled * History of primary immunodeficiency * History of allogeneic organ transplant * History of hypersensitivity to durvalumab, tremelimumab, olaparib or, any excipient * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, active peptic ulcer disease or gastritis, active bleeding diatheses including any subject known to have evidence of, or test positive for, acute or chronic hepatitis B, hepatitis C or human immunodeficiency virus (HIV), or psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent * Known history of previous clinical diagnosis of tuberculosis * History of leptomeningeal carcinomatosis * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab (e.g. live attenuated influenza vaccine \[LAIV\], measles/mumps/rubella vaccine \[MMR\], variola virus vaccine \[VAR\], zoster, yellow fever, etc.) * Female subjects who are pregnant, breast-feeding, or of reproductive potential who are not employing an effective method of birth control from screening to 180 days after the last dose of durvalumab + tremelimumab + olaparib combination therapy or 90 days after the last dose of durvalumab and olaparib therapy, whichever is the longer time period * Any condition that, in the opinion of the investigator, would interfere with evaluation of study treatment or interpretation of patient safety or study results or, is an unsuitable candidate to receive study drug (e.g. inability to tolerate oral medications which would preclude absorption of olaparib) * Symptomatic or uncontrolled brain metastases requiring concurrent treatment, inclusive of but not limited to surgery, radiation and/or corticosteroids * Subjects with uncontrolled seizures * Dependency on IV hydration or total parenteral nutrition (TPN)