Testing Olaparib in Patients With Advanced or Metastatic (Cancer That Has Spread) Bladder Cancer and Other Genitourinary Tumors With DNA-Repair Genetic Changes

Inclusion Criteria: * Patients must have a histologically confirmed diagnosis non-prostate GU cancer * Patients with the presence of cancer-associated genetic mutations in one or more pathogenic or likely pathogenic gene alterations tested in the FoundationOne FoundationOne®CDx (F1CDx) panel will be enrolled in cohorts 1 or 2 as follows: * Cohort 1: BRCA1, BRCA2, ATM, BAP1, PALB2, and BRIP1, or tumor mutational burden (TMB) where 10 or greater mutations/megabase * ABL1, FANCE, POLD1, ATR, FANCG, POLE, ATRX, FANCL, RAD51, BARD1, IKBKE, SMARCB1, BRD4, MEN1, STK11, CCND1, MLH1, TP53, CHEK1, MSH2, CHEK2, MSH6, DOT1L, MUTYH, FANCA, NPM1, FANCC, PMS2 * Patients with benign or variants of unknown significance as determined by FoundationOne FoundationOne®CDx (F1CDx) panel and Genetics Review Panel review will be enrolled in Cohort 3 to be followed for survival * Foundation One mutation analysis results performed prior to enrollment on this study may be accepted for eligibility review and in the event that a patient cannot undergo a biopsy and tumor is not available, Foundation Medicine liquid biopsy may be performed * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as \>= 20 mm (\>= 2 cm) by chest x-ray or as \>= 10 mm (\>= 1 cm) with CT scan, MRI, or calipers by clinical exam * Evidence of disease progression as defined by Response Evaluation Criteria in Solid Tumors (RECIST) (version 1.1) during treatment or after the most recent dose of therapy with at least one platinum-based regimen of chemotherapy and/or an immune-checkpoint inhibitor (atezolizumab, pembrolizumab, nivolumab, avelumab or durvalumab) (2-week washout from chemotherapy and 4-weeks washout from monoclonal antibodies is required) * Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of olaparib in patients \< 18 years of age, children are excluded from this study, but will be eligible for future pediatric trials * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (or Karnofsky \>= 70%) * Leukocytes \>= 3,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 x upper limit of normal (ULN) (for subjects with documented Gilbert's disease total bilirubin =\< 3.0 mg/dL) * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal (for subjects with liver metastasis AST/ALT =\< 5 x ULN) * Creatinine clearance \>= 50 mL/min/1.73 m\^2 * Hemoglobin \>= 9 g/dL; transfusions are allowed * Prothrombin time (PT)/international normalized ratio (INR) and activated partial thromboplastin time (aPTT) within 1.25 x ULN institutional limits, except where a lupus anti-coagulant has been confirmed or the patient is on direct oral anticoagulant (DOA) * Patients must be able to tolerate oral medications and not have gastrointestinal illnesses that would preclude absorption of olaparib * Pre-clinical data indicate that olaparib adversely affects embryofetal survival and development. Therefore, women of child-bearing potential and their partners should agree to use two (2) highly effective forms of contraception throughout study participation and for at least six (6) months after the last dose of olaparib. Male study participants should avoid fathering a child or donating sperm during the study and for three (3) months after the last dose of olaparib. * Note: Olaparib is a PARP inhibitor with the potential for teratogenic or abortifacient effects. Because there is a potential risk for adverse events in nursing infants secondary to treatment of the mother with olaparib, breastfeeding should be discontinued if the mother is treated with olaparib * Ability to understand and the willingness to sign a written informed consent document or patients with impaired decision making capacity (IDMC) if they are represented by a legally authorized representative (LAR) * Patients must provide archival tumor sample for mutation analysis or be willing to undergo mandatory screening biopsy. In the event that the patient cannot undergo biopsy and tumor is not available, Foundation Medicine liquid biopsy will be performed * Postmenopausal or evidence of non-childbearing status for women of childbearing potential: negative urine or serum pregnancy test within 28 days of study treatment at screening. Note: This will also need to be confirmed within 3 days prior to treatment on day 1. Postmenopausal is defined as at least one (1) of the following: * Amenorrheic for 1 year or more following cessation of exogenous hormonal treatments * Luteinizing hormone (LH) and follicle stimulating hormone (FSH) levels in the post menopausal range for women under 50 * Radiation-induced oophorectomy with last menses \> 1 year ago * Chemotherapy-induced menopause with \> 1 year interval since last menses * Surgical sterilization (bilateral oophorectomy or hysterectomy) * Patients is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations Exclusion Criteria: * Patients who have had prior treatment with olaparib or any other PARP inhibitor (PARPi) * Patients with myelodysplastic syndrome/acute myeloid leukemia; or baseline features suggestive of myelodysplastic syndrome or acute myelogenous leukemia on peripheral blood smear or bone marrow biopsy, if clinically indicated * Persistent toxicities (\>= Common Terminology Criteria for Adverse Events \[CTCAE\] grade 2) with the exception of alopecia or peripheral neuropathy, caused by previous cancer therapy * Patients who are receiving any other investigational agents. Patients may be on other clinical trials or treatment during screening to determine eligibility * Patients with known brain metastases should be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events. A scan to confirm the absence of brain metastases is not required. Patients with spinal cord compression unless considered to have received definitive treatment for this and evidence of clinically stable disease for 28 days * History of allergic reactions attributed to compounds of similar chemical or biologic composition of olaparib * Patients receiving strong or moderate CYP3A inhibitors or inducers are ineligible. A washout period prior to the first dose of olaparib for patients on CYP3A inhibitors/inducers is 5 half-lives or 3 weeks , whichever is shorter. Medications with limited systemic absorption (e.g., ophthalmic, otic) do not require a washout and are permitted. * Because the lists of these agents are constantly changing, it is important to regularly consult a frequently-updated medical reference texts such as the Physicians' Desk Reference. As part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the-counter medicine or herbal product * Pregnant women are excluded from this study because olaparib is a PARP inhibitor agent with the potential for teratogenic or abortifacient effects * Any chronic or concurrent acute liver disease * History of stroke, transient ischemic attack (TIA), or myocardial infarction, within 6 months prior to enrollment * Uncontrolled concurrent disease or illness including but not limited to: * Ongoing or active infection * Symptomatic congestive heart failure, unstable angina pectoris, clinically significant cardiac arrhythmia * Unstable or untreated cardiac conditions or ejection fraction of \< 50% as determined by echocardiogram (ECHO) or multiple gated acquisition scan (MUGA) * Uncontrolled diabetes mellitus * Psychiatric illness/social situations that would limit compliance with study requirements * Other severe, acute, or chronic medical or psychiatric condition or laboratory abnormality that may increase the risk associated with study participation or study drug administration or that may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for the study * Immunocompromised patients, e.g., those with known HIV not on highly active antiretroviral therapy (HAART) with detectable levels of virus * Patients with known active hepatitis (i.e., hepatitis B or C) * Other malignancy within the last two (2) years except: adequately treated non-melanoma skin cancer, curatively treated in situ cancer of the cervix, ductal carcinoma in situ (DCIS), stage 1, grade 1 endometrial carcinoma, or other solid tumors including lymphomas (without bone marrow involvement) curatively treated with no evidence of disease for \>= 5 years. Patients with a history of localized triple negative breast cancer or localized resected prostate cancer may be eligible, provided they completed their adjuvant chemotherapy more than two (2) years prior to registration, and that the patient remains free of recurrent or metastatic disease * Patients receiving any systemic chemotherapy or radiotherapy (except for palliative reasons) within 2 weeks prior to study treatment * Major surgery within 2 weeks of starting study treatment and patients must have recovered from any effects of any major surgery * Previous allogenic bone marrow transplant or double umbilical cord blood transplantation (dUCBT)