Durvalumab and Tremelimumab in Treating Patients With Muscle-Invasive, High-Risk Urothelial Cancer That Cannot Be Treated With Cisplatin-Based Therapy Before Surgery

Inclusion Criteria: * Patients must have tissue resected by transurethral resection of bladder tissue (TURBT) at the MD Anderson Cancer Center * Patients must have histologic proof of urothelial cancer; this includes bladder cancer, in addition to other tumors of the urothelial lining including renal pelvis, ureteral, and urethral cancer; upper tract urothelial carcinoma will also be included; this group may include any patient requiring cystectomy (or applicable surgery to resect tumors), including muscle invasive disease (cT2-3aN0M0), whose tumor could not be completely removed at transurethral resection * Patients with the following high-risk features who are not candidates for traditional neoadjuvant chemotherapy will be included for this trial: micropapillary, sarcomatoid and plasmacytoid features; 3-dimensional (3-D) mass on exam under anesthesia (EUA); lymphovascular invasion; hydronephrosis (unless in the opinion of the treating physician, this is not due to tumor); high grade (grade 3) tumors of the ureter, renal pelvis, or tumors in these areas with radiographic abnormality large enough to recognize as an abnormal mass by computed tomography (CT) or magnetic resonance imaging (MRI) imaging; direct invasion of the prostatic stroma or the vaginal wall (i.e. cT4a disease); patients who are candidates for but refusing conventional chemotherapy may be considered eligible; for patients in whom eligibility is unclear, final arbitration will be determined by the principal investigator * Subjects must have no prior exposure to immunotherapy, such as, but not limited to other anti-CTLA-4, anti-PD-1, or anti-PD-L1 antibodies excluding vaccines * Eastern Cooperative Oncology Group (ECOG) performance status of 0-1 * Able and willing to give valid written consent for available archival tumor samples or fresh tumor biopsies/resections * Adequate organ and marrow function (subjects must not have received transfusions or growth factor support within 28 days prior to first dose) * Hemoglobin \>= 9 g/dL * Absolute neutrophil count \>= 1,000/mm\^3 * Platelet count \>= 100,000/mm\^3 * Total bilirubin =\< 1.5 x upper limit of normal (ULN) except subjects with documented Gilbert's syndrome (\> 3 x ULN), who must have a baseline total bilirubin =\< 3.0 mg/dL * Subjects must be considered cisplatin ineligible as per treating physician due to renal dysfunction, or hearing impairment, or co-morbidities; cisplatin ineligibility defined as: glomerular filtration rate (GFR) less than 60 or; congestive heart failure (CHF) New York Heart Association (NYHA) class III or higher or; peripheral neuropathy grade 2 or higher or; Eastern Cooperative Oncology Group (ECOG) performance status (PS) 2 or higher or; impaired hearing; patient's refusal of traditional chemotherapy * Females of childbearing potential who are sexually active with a nonsterilized male partner must use a highly effective method of contraception from the time of screening, and must agree to continue using such precautions for 180 days after the final dose of investigational product; cessation of contraception after this point should be discussed with a responsible physician; periodic abstinence, the rhythm method, and the withdrawal method are not acceptable methods of contraception; they must also refrain from egg cell donation for 180 days after the final dose of investigational product * Nonsterilized males who are sexually active with a female partner of childbearing potential must agree to use a highly effective method of contraception from day 1 through 90 days after receipt of the final dose of investigational product; in addition, they must refrain from sperm donation for 90 days after the final dose of investigational product * Mild autoimmune conditions (such as localized psoriasis) requiring minimal treatment or systemic autoimmune conditions well controlled by target agents such as an anti-IL-17 that do not affect overall immune system; patients with a history of Hashimoto's thyroiditis only requiring hormone replacement, type I diabetes, or conditions not expected to recur in the absence of an external trigger are allowed to participate Exclusion Criteria: * Concurrent enrollment in another clinical trial, unless in a follow-up period or it is an observational study * Any concurrent chemotherapy, immunotherapy, biologic, or hormonal therapy for cancer treatment; concurrent use of hormones for non-cancer-related conditions (e.g. insulin for diabetes and hormone replacement therapy) is acceptable * Any investigational anticancer therapy received within 28 days prior to the first dose of durvalumab and tremelimumab * Major surgical procedure (as defined by the principal investigator \[PI\] or co-PIs within 28 days prior to the first dose of durvalumab and tremelimumab or still recovering from prior surgery * Unresolved toxicities from prior anticancer therapy, defined as having not resolved to National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) version (v) 4.03 grade 0 or 1 with the exception of alopecia and laboratory values listed per the inclusion criteria; subjects with irreversible toxicity that is not reasonably expected to be exacerbated by durvalumab and tremelimumab may be included (e.g. hearing loss) after consultation with the principal investigator * Known or suspected autoimmune disease; patients with a history of inflammatory bowel disease (including Crohn's disease and ulcerative colitis) and autoimmune disorders such as rheumatoid arthritis, systemic progressive sclerosis (scleroderma), systemic lupus erythematosus or autoimmune vasculitis (e.g., Wegener's granulomatosis) are excluded from this study; patients with a history of Hashimoto's thyroiditis only requiring hormone replacement, type I diabetes, or psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are allowed to participate; any condition requiring systemic treatment with corticosteroids (\> 10mg daily prednisone equivalents) or other immunosuppressive medications within 14 days prior to first dose of study drug; inhaled steroids and adrenal replacement steroids doses \> 10mg daily prednisone equivalents are permitted in the absence of active autoimmune disease * History of primary immunodeficiency * Patients who have organ allografts * True active infections of hepatitis B, or C during screening * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS) * Receipt of live, attenuated vaccine within 28 days prior to the first dose of durvalumab and tremelimumab (NOTE: subjects, if enrolled, should not receive live vaccine during the study and for 180 days after the last dose of both drugs) * Females who are pregnant, lactating, or intend to become pregnant during their participation in the study * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, current pneumonitis, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, interstitial lung disease, or psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent * Other invasive malignancies within 2 years except for noninvasive malignancies such as cervical carcinoma in situ, non-melanomatous carcinoma of the skin, or ductal carcinoma in situ of the breast, etc. that has/have been surgically cured * Any evidence of metastatic urothelial carcinoma * Known allergy or hypersensitivity to study drug formulations * Patient currently on dialysis