Testing Olaparib Either Alone or in Combination With Atezolizumab in BRCA Mutant Non-HER2-positive Breast Cancer

Inclusion Criteria: * Patients must have histologically documented unresectable locally advanced or metastatic non-HER2-positive breast cancer and a known BRCA 1/2 mutation present; both germline and somatic mutations are acceptable, however somatic mutations must be identified by either tumor sequencing of tumor tissue or ctDNA in plasma; patients with BRCA mutations of unknown significance are not allowed * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as \>= 20 mm (\>= 2 cm) by chest x-ray or as \>= 10 mm (\>= 1 cm) with CT scan, MRI, or calipers by clinical exam * Prior chemotherapy is allowed, including platinum therapy; patients must not have received chemotherapy for 4 weeks prior to the initiation of study treatment and must have recovery =\< grade 1 from any adverse events from any prior chemotherapy (other than alopecia); patients must not have had nitrosoureas or mitomycin C for 6 weeks prior to the initiation of study treatment * Prior radiation therapy is allowed; patients must not have received minimal radiation therapy (=\< 5% of their total marrow volume) within 3 weeks prior to the initiation of study treatment; otherwise, patients must not have received radiation therapy (\> 5% of their total marrow volume) within 4 weeks prior to the initiation of study treatment; patients who have received prior radiation to 50% or more of their total marrow volume will be excluded * Patients who have received prior treatment with anti-CTLA-4 may be enrolled, provided the following requirements are met: minimum of 12 weeks from the first dose of anti-CTLA-4 and \> 6 weeks from the last dose, and no history of severe immune-related adverse effects from anti-CTLA-4 (National Cancer Institute \[NCI\] Common Terminology Criteria for Adverse Events \[CTCAE\] grade 3 and 4) * Treatment with systemic immunosuppressive medications (including, but not limited to, prednisone, cyclophosphamide, azathioprine, methotrexate, thalidomide, and anti-tumor necrosis factor \[anti-TNF\] agents) is allowed, provided the following is met: minimum of 2 weeks prior to cycle 1, day 1; patients who have received acute, low dose, systemic immunosuppressant medications (e.g., a one-time dose of dexamethasone for nausea) may be enrolled; the use of corticosteroids and mineralocorticoids (e.g., fludrocortisone) for patients with orthostatic hypotension or adrenocortical insufficiency is allowed * Prior treatment with systemic immunostimulatory agents (including, but not limited to, interferon \[IFN\]-alpha or interleukin \[IL\]-2) is allowed, provided the following is met: minimum of 4 weeks or 5 half-lives of the drug (whichever is longer) prior to cycle 1, day 1 * Patients taking bisphosphonate therapy for symptomatic hypercalcemia are NOT allowed; use of bisphosphonate therapy for other reasons (e.g., bone metastasis or osteoporosis) is allowed * Prior hormone therapy is allowed; patients must not have received hormone therapy for breast cancer for 2 weeks prior to the initiation of study treatment and must have recovery =\< grade 1 from any adverse events related to these therapies (other than alopecia) * Prior experimental (non-Food and Drug Administration \[FDA\] approved) therapies and immunotherapies are allowed; patients must not have received these therapies for 4 weeks prior to the initiation of study treatment and must have recovery =\< grade 1 from any adverse events of these therapies (other than alopecia); prior treatment with any PARP inhibitor or any anti-PD-1/anti-PD-L1 antibody is NOT allowed * Other therapies (e.g. targeted therapy such as cyclin-dependent kinase \[CDK\] inhibitors): patients should have recovered to =\< grade 1 drug related toxicity; they must have completed therapy for either a total of duration equivalent to 5 half-lives of the drug or 28 days, whichever is shorter * Age \>= 18 years. No dosing or adverse event data are currently available on the use of olaparib in combination with atezolizumab in patients \< 18 years of age; hence, children are excluded from this study, but will be eligible for future pediatric trials * Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%) * Life expectancy of greater than 6 months * Absolute neutrophil count \>= 1,500/mcL * Leukocytes \>= 3,000/mcL * Platelets \>= 100,000/mcL * Hemoglobin \>= 8 g/dL * Total bilirubin =\< 1.5 x upper limit of normal (ULN) (however, patients with known Gilbert disease who have serum bilirubin level =\< 3 x ULN may be enrolled) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) =\< 2.5 x upper limit of normal (ULN) if no liver metastasis; =\< 5 x upper ULN if liver metastasis present * Alkaline phosphatase =\< 2.5 x ULN (=\< 5 x ULN for patients with documented liver involvement or bone metastases) * Creatinine clearance \>= 51 mL/min/1.73 m\^2 by Cockcroft-Gault * International normalized ratio (INR) and activated partial thromboplastin time (aPTT) =\< 1.5 x ULN (this applies only to patients who do not receive therapeutic anticoagulation; patients receiving therapeutic anticoagulation, such as low-molecular-weight heparin or warfarin, should be on a stable dose) * No features suggestive of myelodysplastic syndrome (MDS)/acute myeloid leukemia (AML) on peripheral blood smear when performed as clinically indicated * Patients must have tumors determined to be easily accessible for biopsy and must be willing to have serial biopsies (with a third biopsy upon evidence of disease progression) * Administration of atezolizumab and/or olaparib may have an adverse effect on pregnancy and poses a risk to the human fetus, including embryo-lethality; women of child-bearing potential and men must agree to use highly effective contraception prior to study entry, for the duration of study participation, and for at least 5 months (150 days) after the last dose of study agent; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately; men treated or enrolled on this protocol must also agree to use two highly effective forms of contraception in combination prior to the study, for the duration of study participation, and for at least 5 months (150 days) after completion of atezolizumab and/or olaparib administration; women of child-bearing potential: negative serum pregnancy test within 14 days of study treatment and confirmed prior to treatment on day 1; postmenopausal or evidence of non-childbearing status for women of childbearing potential; postmenopausal is defined as: * Amenorrheic for 1 year or more following cessation of exogenous hormonal treatments * Luteinizing hormone (LH) and follicle stimulating hormone (FSH) levels in the post-menopausal range for women under 50 * Radiation-induced oophorectomy with last menses \> 1 year ago * Chemotherapy-induced menopause with \> 1 year interval since last menses * Surgical sterilization (bilateral oophorectomy or hysterectomy) * Ability to understand and the willingness to sign a written informed consent document * Subject is able to swallow and retain oral medication and does not have uncontrolled emesis or gastrointestinal disorders likely to interfere with absorption of the study medication * Patients crossing over from monotherapy to combination therapy do not have to be fully rescreened, as they would have just had tumor assessments and would already have had baseline electrocardiographies (ECGs), etc; however, they do need to meet performance status, organ function, and blood parameters and not meet any of the exclusion criteria * Patients positive for human immunodeficiency virus (HIV) are NOT excluded from this study, but HIV-positive patients must have: * A stable regimen of highly active anti-retroviral therapy (HAART) * No requirement for concurrent antibiotics or antifungal agents for the prevention of opportunistic infections * A CD4 count above 250 cells/mcL and an undetectable HIV viral load on standard polymerase chain reaction (PCR)-based test Exclusion Criteria: * Patients with prior allogeneic bone marrow transplantation, double umbilical cord blood transplantation (dUCBT) or prior solid organ transplantation * Patients with known brain metastases should be excluded from this clinical trial except as those described below, because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events * Patients with known primary central nervous system (CNS) malignancy or symptomatic CNS metastases are excluded, with the following exceptions: * Patients with asymptomatic untreated CNS disease may be enrolled, provided all of the following criteria are met: * Evaluable or measurable disease outside the CNS * No metastases to brain stem, midbrain, pons, medulla, cerebellum, or within 10 mm of the optic apparatus (optic nerves and chiasm) * No history of intracranial hemorrhage unless it is confined within a lesion previously noted and secondary to gamma knife or another equivalent radiologic therapeutic * No history of spinal cord hemorrhage * No ongoing requirement for dexamethasone for CNS disease; patients on a stable dose of anticonvulsants are permitted * No neurosurgical resection or brain biopsy within 28 days prior to cycle 1, day 1 * Patients with asymptomatic treated CNS metastases may be enrolled, provided all the criteria listed above are met as well as the following: * Radiographic demonstration of improvement upon the completion of CNS directed therapy and no evidence of interim progression between the completion of CNS directed therapy and the screening radiographic study * No stereotactic radiation or whole-brain radiation within 28 days prior to cycle 1, day 1 * Screening CNS radiographic study \>= 4 weeks from completion of radiotherapy and \>= 2 weeks from discontinuation of corticosteroids * Known hypersensitivity to Chinese hamster ovary cell products or other recombinant human antibodies * History of severe allergic, anaphylactic, or other hypersensitivity reactions to chimeric or humanized antibodies or fusion proteins * History of allergic reactions attributed to compounds of similar chemical or biologic composition to olaparib and atezolizumab; patients with a known hypersensitivity to olaparib or any of the excipients of the product * Prior treatment with any PARP inhibitor or any anti-PD-1/anti-PD-L1 antibody * Known clinically significant liver disease, including active viral, alcoholic, or other hepatitis; cirrhosis; fatty liver; and inherited liver disease * Patients with past or resolved hepatitis B infection (defined as having a negative hepatitis B surface antigen \[HBsAg\] test and a positive anti-HBc \[antibody to hepatitis B core antigen\] antibody test) are eligible * Patients positive for hepatitis C virus (HCV) antibody are eligible only if polymerase chain reaction (PCR) is negative for HCV RNA * History or risk of autoimmune disease, including, but not limited to, systemic lupus erythematosus, rheumatoid arthritis, inflammatory bowel disease, vascular thrombosis associated with antiphospholipid syndrome, Wegener's granulomatosis, Sjogren's syndrome, Bell's palsy, Guillain-Barre syndrome, multiple sclerosis, autoimmune thyroid disease, vasculitis, lichen sclerosis, or glomerulonephritis * Patients with a history of autoimmune hypothyroidism on a stable dose of thyroid replacement hormone may be eligible * Patients with controlled type 1 diabetes mellitus on a stable insulin regimen may be eligible * Patients with eczema, psoriasis, lichen simplex chronicus of vitiligo with dermatologic manifestations only (e.g., patients with psoriatic arthritis would be excluded) are permitted provided that they meet the following conditions: * Patients with psoriasis must have a baseline ophthalmologic exam to rule out ocular manifestations * Rash must cover less than 10% of body surface area (BSA) * Disease is well controlled at baseline and only requiring low potency topical steroids (e.g., hydrocortisone 2.5%, hydrocortisone butyrate 0.1%, fluocinolone 0.01%, desonide 0.05%, alclometasone dipropionate 0.05%) * No acute exacerbations of underlying condition within the last 12 months (not requiring psoralen plus ultraviolet A radiation \[PUVA\], methotrexate, retinoids, biologic agents, oral calcineurin inhibitors; high potency or oral steroids) * History of idiopathic pulmonary fibrosis, pneumonitis (including drug induced), organizing pneumonia (i.e., bronchiolitis obliterans, cryptogenic organizing pneumonia, etc.), or evidence of active pneumonitis on screening chest computed tomography (CT) scan; history of radiation pneumonitis in the radiation field (fibrosis) is permitted if recovered * Major surgical procedure within 28 days prior to cycle 1, day 1 and patients must have recovered from any effects of any major surgery; anticipation of need for a major surgical procedure during the course of the study * Administration of a live, attenuated vaccine within 4 weeks before cycle 1, day 1 or anticipation that such a live, attenuated vaccine will be required during the study and up to 5 months after the last dose of atezolizumab * Influenza vaccination should be given during influenza season only (approximately October to March); patients must not receive live, attenuated influenza vaccine within 4 weeks prior to cycle 1, day 1 or at any time during the study * Patients considered a poor medical risk due to a serious, uncontrolled medical disorder, non-malignant systemic disease or active, uncontrolled infection; examples include, but are not limited to, uncontrolled ventricular arrhythmia, recent (within 3 months) myocardial infarction, unstable spinal cord compression, superior vena cava syndrome, symptomatic congestive heart failure, unstable angina pectoris, extensive interstitial bilateral lung disease on high resolution computed tomography (HRCT) scan or any psychiatric disorder that prohibits obtaining informed consent and would limit compliance with study requirements * Pregnant women are excluded from this study because olaparib and atezolizumab have the potential for teratogenic or abortifacient effects; because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with olaparib and atezolizumab, breastfeeding should be discontinued if the mother is treated with olaparib and atezolizumab * Any prior grade \>= 3 immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE \> grade 1 * Patients with active seizures or a history of uncontrolled seizure disorder, including focal or generalized seizure within the past year * Patients requiring treatment with a RANKL inhibitor (e.g. denosumab) who cannot discontinue it before treatment with atezolizumab * Resting ECG with corrected QT (QTc) \> 470 msec or family history of long QT syndrome * Concomitant use of known strong CYP3A inhibitors (e.g. itraconazole, telithromycin, clarithromycin, protease inhibitors boosted with ritonavir or cobicistat, indinavir, saquinavir, nelfinavir, boceprevir, telaprevir) or moderate CYP3A inhibitors (e.g. ciprofloxacin, erythromycin, diltiazem, fluconazole, verapamil); the required washout period prior to starting olaparib is 2 weeks; because the lists of these agents are constantly changing, it is important to regularly consult a frequently updated drug information reference; medical reference texts such as the Physicians' Desk Reference may also provide this information; as part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the-counter medicine or herbal product * Concomitant use of known strong (e.g. phenobarbital, enzalutamide, phenytoin, rifampicin, rifabutin, rifapentine, carbamazepine, nevirapine and St John's wort) or moderate CYP3A inducers (e.g. bosentan, efavirenz, modafinil); the required washout period prior to starting olaparib is 5 weeks for enzalutamide or phenobarbital and 3 weeks for other agents; because the lists of these agents are constantly changing, it is important to regularly consult a frequently updated drug information reference; medical reference texts such as the Physicians' Desk Reference may also provide this information; as part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the-counter medicine or herbal product * Severe infection within 4 weeks prior to initiation of study treatment, including, but not limited to, hospitalization for complications of infection, bacteremia, or severe pneumonia * Treatment with therapeutic oral or IV antibiotics within 2 weeks prior to initiation of study treatment. Patients receiving prophylactic antibiotics (e.g., to prevent a urinary tract infection or chronic obstructive pulmonary disease exacerbation) are eligible for the study