Radiation Therapy and Durvalumab With or Without Tremelimumab in Treating Participants With Unresectable, Locally Advanced, or Metastatic Bladder Cancer

Inclusion Criteria: * Written informed consent obtained from the patient/legal representative prior to performing any protocol-related procedures, including screening evaluations. * cT2 - T4 or metastatic (N+ or M+) urothelial bladder cancer. Mixed histologies with predominant urothelial pattern are allowed. Patients with localized disease must either refuse cystectomy or be deemed not ideal cystectomy candidates. * If metastatic disease present patients must have an intact, symptomatic bladder tumor, appropriate for palliative RT to the bladder. * Measurable metastatic disease according to Response Evaluation Criteria in Solid Tumors (RECIST) version (v)1.1 criteria. Thus, patients with metastatic disease must have at least 1 lesion, not previously irradiated, that can be accurately measured at baseline as \>= 10 mm in the longest diameter (except lymph nodes which must have a short axis \>= 15 mm) with computed tomography (CT) or magnetic resonance imaging (MRI) and that is suitable for accurate repeated measurement as per RECIST v1.1 guidelines. * Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 1/ * Life expectancy of \>= 12 weeks. * Patients must be ineligible for or refuse cisplatin-based chemotherapy. Cisplatin ineligibility is defined as meeting 1 of the following criteria: * Creatinine clearance (calculated or measured) \< 60 mL/min * Common Terminology Criteria for Adverse Events (CTCAE) grade \>= 2 audiometric hearing loss * CTCAE grade 2 or higher peripheral neuropathy * New York Heart Association class III heart failure * Any other criteria deemed by the investigator to make the patient unsuitable for cisplatin-based chemotherapy * Note: The reason for cisplatin ineligibility must be documented. Patients may refuse cisplatin-based chemotherapy after an informed discussion of the risks and benefits, and the reason for refusal must be documented. * Subjects must consent to provide an archived tumor specimen from within 12 months prior to study entry (ie, from subject signing consent to participate in the study) for immunologic characterization. If not available, subjects should have at least 1 lesion amenable to biopsy and consent to provide a pre-treatment fresh biopsy. Tumor lesions used for biopsy should not be lesions used as target lesions. Additional archival tissue from beyond 12 months prior to study entry is also requested, if available, to support exploratory analyses. * Hemoglobin \>= 9.0 g/dL * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (\>= 1500 per mm\^3) * Platelet count \>= 100 x 10\^9/L (\>= 100,000 per mm\^3) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN). This will not apply to subjects with confirmed Gilbert?s syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only in consultation with their physician. * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal unless liver metastases are present, in which case it must be =\< 5 x ULN * Serum creatinine clearance (CL) \> 30 mL/min by the Cockcroft-Gault formula or by 24-hour urine collection for determination of creatinine clearance. * Female subjects must either be of non-reproductive potential (ie, post-menopausal by history: \>= 60 years old and no menses for 1 year without an alternative medical cause; OR history of hysterectomy, OR history of bilateral tubal ligation, OR history of bilateral oophorectomy) or must have a negative serum pregnancy test upon study entry. Evidence of post-menopausal status or negative urinary or serum pregnancy test for female pre-menopausal patients. Women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause. The following age-specific requirements apply: * Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy). * Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \> 1 year ago, had chemotherapy-induced menopause with last menses \> 1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy) * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visit and examinations and follow-up. Exclusion Criteria: * ECOG PS 2 or higher. * Prior cystectomy or definitive RT to the bladder. * History of autoimmune disease, including, but not limited to myasthenia gravis, myositis, autoimmune hepatitis, systemic lupus erythematosus, rheumatoid arthritis, inflammatory bowel disease, vascular thrombosis associated with antiphospholipid syndrome, Wegner?s granulomatosis, Sjogren?s syndrome, Guillain-Barre syndrome, multiple sclerosis, vasculitis, or glomerulonephritis. * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[eg, colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], celiac disease, systemic lupus erythematosus, Sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc\]). The following are exceptions to this criterion: * Subjects with childhood atopy or asthma, vitiligo, alopecia, Hashimoto syndrome, Grave?s disease, or psoriasis not requiring systemic treatment (within the past 2 years) are not excluded. * Patients with a history of autoimmune-related hypothyroidism on a stable dose of thyroid replacement hormone are not excluded. * Patients with controlled type 1 diabetes mellitus on a stable dose of insulin regimen are not excluded. * Any chronic skin condition that does not require systemic therapy are not excluded. * Patients without active autoimmune disease in the last 5 years may be included after consultation with the study physician. * Patients with human immunodeficiency virus (HIV), active hepatitis B (HBV) or active hepatitis C (HCV) * Patients with past HBV infection or resolved HBV infection, defined as the presence of hepatitis B core antibody (HBc Ab) and absence of hepatitis B surface antigen (HBsAg) are eligible. HBV deoxyribonucleic acid (DNA) must be obtained in these patients prior to day 1 of therapy, but detection of HBV DNA in these patients will not exclude study participation. * Patients positive for HCV antibody are eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA). * History of idiopathic pulmonary fibrosis, organizing pneumonia (e.g., bronchiolitis obliterans), drug-induced pneumonitis, idiopathic pneumonitis, or evidence of active pneumonitis on screening chest CT scan. * Involvement in the planning and/or conduct of the study (applies to both AstraZeneca staff and/or staff at the study site). * Previous investigational product (IP) assignment in the present study. * Current enrollment/participation in another clinical study, unless it is an observational (noninterventional) clinical study or during the follow-up period of an interventional study. * Prior radiation therapy to the abdomen or pelvis, including radiation therapy to the bladder, prostate, or rectum. * Any concurrent systemic chemotherapy, biologic, or hormonal therapy for cancer treatment. Concurrent use of hormonal therapy for non-cancer related conditions (i.e. hormonal replacement therapy) is acceptable. Prior systemic therapies are allowed with a washout period of \>= 45 days. * Major surgical procedure (as defined by the investigator) within 28 days prior to the first dose of IP. Note biopsy and transurethral resection of the bladder tumor (TURBT) of the primary tumor is acceptable. * History of another primary malignancy except for: * Malignancy treated with curative intent and with no known active disease \>= 3 years before the first dose of study drug and of low potential risk for recurrence. * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease. * Adequately treated carcinoma in situ without evidence of disease eg, cervical cancer in situ \>\> (Optional criteria that are dependent on the patient population under investigation.) * Note: Patients with incidental finding of early stage prostate cancer will not be eligible. * QT interval corrected for heart rate using Fridericia?s formula (QTcF) \>= 470 ms calculated. Any clinically significant abnormalities detected require triplicate electrocardiography (ECG) results and a mean QT interval corrected for heart rate using Fridericia?s formula (QTcF) \>= 470 ms calculated from 3 ECGs). * Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab or tremelimumab, with the exceptions of intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid. * History of primary immunodeficiency. * History of allogeneic organ transplant. * History of hypersensitivity to durvalumab, tremelimumab, or any excipient. * History of hypersensitivity to the combination therapy of durvalumab and tremelimumab. * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, active peptic ulcer disease or gastritis, active bleeding diatheses including any subject known to have evidence of acute or chronic hepatitis B, hepatitis C or human immunodeficiency virus (HIV), or psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent. * Active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination, and radiographic findings, and tuberculosis \[TB\] testing in line with local practice). * History of leptomeningeal carcinomatosis. * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab or tremelimumab. * Any condition that, in the opinion of the investigator, would interfere with evaluation of study treatment or interpretation of patient safety or study results. * Brain metastases or spinal cord compression unless the patient?s condition is stable (asymptomatic, no evidence of new or emerging brain metastases) and off steroids for at least 14 days prior to the start of study treatment. Following radiotherapy and/or surgery, patients with suspected brain metastases at screening should have a magnetic resonance imaging (MRI)(preferred)/CT, preferably with IV contrast. * Subjects with uncontrolled seizures. * Female patients who are pregnant or breastfeeding or male or female patients of reproductive potential who are not willing to employ effective birth control from screening to 180 days after the last dose of durvalumab + tremelimumab combination therapy or 90 days after the last dose of durvalumab monotherapy, whichever is the longer time period.