Nivolumab and RT in Treating Patients With Localized/Locally Advanced Urothelial Bladder Cancer Ineligible for Chemo

Inclusion Criteria: \- Localized urothelial cancer of bladder with presence of transitional cell carcinoma (TCC) component; mixed histologies are allowed Clinical or pathologic stage T2 -T4 disease including T4a and 4b if feasible to treat with radiation therapy Locoregional lymph node metastases are permitted but patients with distant metastases are ineligible; imaging to evaluate for distant metastases should consist of a minimum of computed tomography (CT)/magnetic resonance imaging (MRI) of abdomen/pelvis or CT urogram and a chest x-ray (CXR) or CT chest; patients for which there is clinical suspicion or symptoms of bone metastasis should have a bone scan completed to rule out metastatic disease prior to enrollment on study Agreeable to consider radiation therapy (RT) for the urothelial cancer: patients have to be evaluated by a radiation oncologist and deemed to be candidates for RT The patients must not be candidates for chemotherapy due to at least one of the following reasons: * Performance status of 2 * Creatinine clearance =\< 60 ml/min as calculated by the Cockcroft-Gault formula * Cardiac disease such as New York Heart Association (NYHA) class III or IV heart failure or cardiac ischemia within the last 12 months, grade 2 or greater neuropathy, or other comorbidities based on which patient is not considered a candidate for chemotherapy Alkaline phosphatase =\< 3 x upper limit of normal Aspartate aminotransferase (AST) =\< 3 x upper limit of normal Alanine aminotransferase (ALT) =\< 3 x upper limit of normal Bilirubin \< 1.5 x upper limit of normal (ULN) Absolute neutrophil count \>= 1500/mm\^3 Hemoglobin \>= 9 g/dL Platelets \>= 100 K/mm\^3 Performance score (PS) of 0-2 by Zubrod score Life expectancy of 12 months Willingness to sign informed consent Patients cannot have active autoimmune disease or immunosuppressive conditions Serum creatinine =\< 1.5 X institutional ULN or creatinine clearance \> 40 ml/min as calculated by the Cockcroft-Gault formula In females with childbearing potential, or men with partners of child bearing potential, willingness to use adequate contraception for a minimum duration of 155 days in females and 215 days in males, after last dose of nivolumab Maximal tumor resection has been performed as feasible Exclusion Criteria: \- The subject has received cytotoxic chemotherapy (including investigational cytotoxic chemotherapy) or biologic agents (e.g., cytokines or antibodies) for urothelial cancer within 4 weeks, or intravesical Bacillus Calmette-Guerin (BCG) within 6 weeks of the first dose of study treatment Prior treatment with any PD-1 or PDL-1 inhibitor The subject has received therapeutic radiation: * To the bladder/prostate/rectum pelvis * To any other site(s) within 28 days of the first dose of study treatment Obstructive renal failure that is not relieved with stents or nephrostomy tube/s The subject has received any other type of investigational agent within 28 days before the first dose of study treatment Steroid doses greater than an equivalent of prednisone 10 mg daily The subject has prothrombin time (PT)/international normalized ratio (INR) or partial thromboplastin time (PTT) test results at screening \>= 2 x the laboratory ULN Uncontrolled hematuria The subject has uncontrolled, significant intercurrent or recent illness including, but not limited to, the following conditions: * Cardiovascular disorders such as uncontrolled arrhythmias or uncontrolled congestive heart failure * Gastrointestinal disorders particularly those associated with a high risk of perforation or fistula formation including: * Any of the following at the time of screening * Active peptic ulcer disease, * Active inflammatory bowel disease (including ulcerative colitis and Crohn's disease), diverticulitis, cholecystitis, symptomatic cholangitis or appendicitis * Any of the following within 6 months before the first dose of study treatment: * History of abdominal fistula * Bowel perforation The subject has a previously identified allergy or hypersensitivity to components of the study treatment formulation The subject is unable or unwilling to abide by the study protocol or cooperate fully with the investigator or designee Presence of another invasive malignancy, which required systemic therapy within 12 months of protocol enrollment, except for resected skin cancers or prostate cancer that is in remission Pregnant or nursing women Patient is a candidate for radical cystectomy as a potentially curative option. The patient may not be a candidate for radical cystectomy due to any of the following reasons: comorbidities, patient preference, or physician discretion. Patients with inherited syndromes associated with hypersensitivity to ionizing radiation (e.g., ataxia-telangiectasia, Nijmegen breakage syndrome)