Olaparib and Durvalumab in Treating Participants With Metastatic Triple Negative Breast Cancer

Inclusion Criteria: * Ability to understand and the willingness to sign a written informed consent document. * Metastatic triple negative breast cancer (TNBC), as defined by: * Estrogen receptor (ER) and progesterone receptor (PR) negative as defined as ER \< 10% and PR \< 10% by immunohistochemistry according to American Society of Clinical Oncology (ASCO)/College of American Pathologists (CAP) guidelines for hormone receptor testing * Human epidermal growth factor receptor 2 (HER2) non-amplified per ASCO/CAP guidelines, defined as: * immunohistochemistry (IHC) score 0/1+ * IHC 2+ and in situ hybridization (ISH) non-amplified with a ratio of HER2 to CEP17 \< 2.0, and if reported, average HER2 gene copy number \< 4 signals/cells; or * ISH non-amplified with a ratio of HER2 to chromosome enumeration probe 17 (CEP17) \< 2.0, and if reported, average HER2 gene copy number \< 4 signals/cells * Participants must have at least one measurable site of disease as defined by Response Evaluation Criteria in Solid Tumors (RECIST) version (v)1.1 that is amendable to biopsy. * Prior therapies for metastatic breast cancer * Frontline patients who have not received prior systemic therapy for metastatic breast cancer are eligible, * Patients who have received =\< 2 prior chemotherapy regimens for metastatic breast cancer are eligible * Participants must have fully recovered from the acute toxic effects of all prior treatment to grade 1 or less, except alopecia and =\< grade 2 neuropathy which are allowed * Participants' life expectancy must be \> 6 months * Participant must have Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 * Participant must consent to undergo a pre-treatment screening biopsy for enrollment and subsequent biomarker analyses. * Participants must consent to undergo one mandatory on-study tumor biopsy following a 4 week, single cycle induction treatment of olaparib. A second on-study biopsy at time of disease progression is optional, but not mandatory. * Participants must not have had prior immunotherapy with anti-PD-L1, anti-PD-1, anti-CTLA4 or similar drugs. * Participants must not have received previous treatment with PARP inhibitors * Participants must not be planning to receive concomitantly other biologic therapy, hormonal therapy, other chemotherapy, surgery or other anti-cancer therapy except radiation therapy while receiving treatment on this protocol. * Participant must be able to swallow tablets or capsules. A participant with any gastrointestinal disease that would impair ability to swallow, retain, or absorb drug is not eligible. * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9 /L (at time of registration and within 4 weeks prior to initiating on-protocol treatment) * May be waived on a case-by-case basis for patient populations recognized to have normal baseline values below this level. * Platelets \>= 100 x 10\^9 /L (at time of registration and within 4 weeks prior to initiating on-protocol treatment) * Hemoglobin \>= 9 g/dL or \>= 5.6 mmol/L (at time of registration and within 4 weeks prior to initiating on-protocol treatment) * Creatinine =\< 1.5 x upper limit of normal (ULN), OR measured or calculated creatinine clearance (glomerular filtration rate (GFR) can also be used in place of creatinine or CrCl) \>= 60 mL/min/1.73m\^2 for participants with creatinine levels \> 1 x institutional ULN (at time of registration and within 4 weeks prior to initiating on-protocol treatment) * Creatinine clearance should be calculated per institutional standard. For participants with a baseline calculated creatinine clearance below normal institutional laboratory values, a measured baseline creatinine clearance should be determined. Individuals with higher values felt to be consistent with inborn errors of metabolism will be considered on a case-by-case basis. * Total bilirubin =\< 1.5 x ULN, OR direct bilirubin =\< ULN for participants with total bilirubin levels \> 1.5 x ULN (at time of registration and within 4 weeks prior to initiating on-protocol treatment) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x ULN (at time of registration and within 4 weeks prior to initiating on-protocol treatment) * Female participants of childbearing potential must have a negative urine or serum pregnancy test within 72 hours prior to receiving the first dose of study medication. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required. * Female participants of childbearing potential must agree to use adequate methods of contraception starting with the first dose of study therapy through 60 days after the last dose of study therapy. * Participants of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year without an alternative medical cause. * Note: Abstinence is acceptable if this is the preferred contraception for the participant. * Male participants must agree to use an adequate method of contraception starting with the first dose of study therapy through 60 days after the last dose of study therapy. * Note: Abstinence is acceptable if this is the preferred contraception for the participant. * Participants must not have received live vaccines within 30 days prior to trial registration. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, chicken pox, shingles, yellow fever, rabies, bacillus Calmette-Guerin (BCG), and typhoid (oral) vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g., Flu-Mist) are live attenuated vaccines, and are not allowed. * Participants must not have known active hepatitis B virus (HBV) or hepatitis C virus (HCV) infection prior to trial registration. Patients who have completed curative therapy for HCV are eligible. Patients with known human immunodeficiency virus (HIV) infection are eligible if they meet each of the following 3 criteria: * CD4 counts \>= 350 mm\^3 * Serum HIV viral load of \< 25,000 IU/ml and * Treated on a stable antiretroviral regimen. * No other prior invasive malignancy is allowed except for the following: adequately treated basal (or squamous cell) skin cancer, in situ breast or cervical cancer. Stage I or II invasive cancer treated with a curative intent without evidence of disease recurrence for at least five years. Exclusion Criteria: * EXCLUSION - PARTICIPANT: Currently participating and receiving study therapy or has participated in a study of an investigational agent and received study therapy or used an investigation device within 4 weeks of first dose of treatment. * Individuals in the follow-up phase of a prior investigational study may participate as long as it has been 4 weeks since last dose of the previous investigational agent of device. * EXCLUSION - PARTICIPANT: Participant received prior chemotherapy or any other targeted therapies within the past 28, or palliative radiation within the past 14 days, prior to going on-study. * EXCLUSION - PARTICIPANT: Participants with known active central nervous system (CNS) metastases and/or carcinomatous meningitis. * EXCLUSION - PARTICIPANT: Participants with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging \[confirmed by computed tomography (CT) scan if CT used at prior imaging, or confirmed by magnetic resonance imaging (MRI) if MRI was used at prior imaging) for at least four weeks prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 7 days prior to trial treatment. This exception does not include carcinomatous meningitis which is excluded regardless of clinical stability. * EXCLUSION - PARTICIPANT: Active infection requiring systemic antibiotic therapy. Participants requiring systemic antibiotics for infection must have completed therapy before treatment is initiated. * EXCLUSION - PARTICIPANT: Clinically significant cardiac disease or impaired cardiac function, including any of the following: * Clinically significant and/or uncontrolled heart disease such as congestive heart failure (New York Heart Association grade \>= 2) uncontrolled hypertension, or clinically significant arrhythmia currently requiring medical treatment * Fridericia's correction formula (QTcF) \> 470 msec for females, or \> 450 msec for males, on screening electrocardiography (ECG) or congenital long QT syndrome * Acute myocardial infarction or unstable angina pectoris \< 6 months prior to screening * EXCLUSION - PARTICIPANT: Psychiatric illness/social situations that would limit compliance with study requirements * EXCLUSION - PARTICIPANT: Participants with a history of hypersensitivity reactions to study agent or their excipients. * EXCLUSION - PARTICIPANT: Participant is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the screening visit through 120 days after the last dose of trial treatment. * EXCLUSION - DURVALUMAB DRUG-SPECIFIC: Participant has evidence of interstitial lung disease or active non-infectious pneumonitis. * EXCLUSION - DURVALUMAB DRUG-SPECIFIC: Major surgical procedure (as defined by the investigator) within 28 days prior to the first dose of durvalumab. * Note: Local surgery of isolated lesions for palliative intent is acceptable per investigator discretion. * EXCLUSION - DURVALUMAB DRUG-SPECIFIC: Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab, with the exceptions of intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid. The following are exceptions to this criterion: * Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection) * Systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or its equivalent * Steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) * EXCLUSION - DURVALUMAB DRUG-SPECIFIC: Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, Sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc.\]). The following are exceptions to this criterion: * Participants with vitiligo or alopecia * Participants with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement * Any chronic skin condition that does not require systemic therapy * Participants without active disease in the last 5 years may be included but only after consultation with the study physician * Participants with celiac disease controlled by diet alone * EXCLUSION - DURVALUMAB DRUG-SPECIFIC: History of allogenic organ transplantation.