A Study to Test the Safety of Immunotherapy With Nivolumab Alone or With Ipilimumab Before Surgery for Bladder Cancer Patients Who Are Not Suitable for Chemotherapy

Inclusion Criteria: * Histologically confirmed diagnosis of urothelial carcinoma of the bladder. Variant histology is acceptable if there is a predominant urothelial component. * For MUSCLE-INVASIVE UROTHELIAL CANCER OF THE BLADDER (Cohorts 1 - 3): ° Cystoscopically and radiographically confirmed cT2-4a cN0 cM0 disease. Patients with cT4a disease invading into the prostatic stroma with no cystoscopic confirmation of muscle invasion are eligible. * For UROTHELIAL CARCINOMA OF THE UPPER URINARY TRACT (URETER OR RENAL PELVIS) (Cohort U): °Histologically confirmed high grade urothelial carcinoma of the upper tract and/or radiographically visible tumor stage T2-T4a N0/x M0 disease with positive selective urinary cytology. Hydronephrosis associated with tumor on imaging or biopsy will be considered invasive by definition. (Variant histology is acceptable if there is a predominant urothelial component) * Patients ineligible for cisplatin based on any of the following criteria: * Estimated or calculated creatinine clearance ≥ 30ml/min but \< 60 ml/min * Grade 2 or above audiometric hearing loss (per CTCAE v4.0) * Grade 2 or above peripheral neuropathy (per CTCAE v4.0) * Availability of tumor specimen block or 30 unstained slides from diagnosis of muscle-invasive disease. Patients with fewer than 30 slides available may be enrolled after discussion with the Principal Investigator. * Karnofsky performance status ≥ 70%. * Medically appropriate candidate for radical cystectomy, as per MSK Attending Urologic Oncologist * Age ≥ 18 years. * Required initial laboratory values: * Absolute neutrophil count ≥ 1.5 x 10\^9/L * Platelets ≥ 100 x 10\^9/L * Bilirubin ≤1.5 times the upper limit of normal (x ULN) * Aspartate transaminase (AST) and alanine transaminase (ALT) ≤ 3 x ULN * PTT/PT ≤1.5 x ULN or INR \< 1.7 x ULN for patients who are not receiving therapeutic anticoagulation. Patients receiving therapeutic anticoagulation should be on a stable dose. Exclusion Criteria: * Prior treatment with systemic chemotherapy for urothelial cancer, including immune checkpoint inhibitors for non-muscle invasive bladder cancer. (Prior intravesical treatment such as BCG is allowed.) * Prior bladder-directed radiotherapy (exclusion applies only to MIBC Cohorts 1 - 3). * Presence of active autoimmune disease, symptoms, or conditions, with the following exceptions: °Subjects with type I diabetes mellitus, residual hypothyroidism due to autoimmune thyroiditis only requiring hormone replacement, skin disorders (such as vitiligo, psoriasis, or alopecia) not requiring systemic treatment, asymptomatic laboratory evidence of autoimmune disease (e.g.: +ANA, +RF, anti-thyroglobulin antibodies), or conditions not expected to recur in the absence of an external trigger are permitted to enroll. * Subjects with a condition requiring systemic treatment with either corticosteroids (\>10 mg daily prednisone equivalent) or other immunosuppressive medications within 14 days of first dose of study drug. Inhaled or topical steroids, and adrenal replacement steroid doses are permitted in the absence of active autoimmune disease. * Unstable angina. * New York Heart Association (NYHA) Grade II or greater congestive heart failure. * History of myocardial infarction within 6 months. * History of stroke within 6 months. * Evidence of bleeding diathesis or coagulopathy. Therapeutic anticoagulation is permitted, but patients must be on a stable dose. * Major surgical procedure within 28 days prior to the study. (Transurethral resection of bladder tumor is permitted * Serious, non-healing wound, ulcer, or bone fracture. * Other prior malignancy active within the previous 2 years except for local or organ-confined early stage cancer that has been definitively treated with curative intent or does not require treatment, does not require ongoing treatment, has no evidence of active disease, and has a negligible risk of recurrence and is therefore unlikely to interfere with the endpoints of the study. * Subjects who have received prior therapy with any T cell co-stimulation or checkpoint pathways such as anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CTLA-4, anti-CD137; or other medicines specifically targeting T cells are prohibited. Prior IL-2 is permitted. * Prior therapy with intravesical BCG within 6 weeks of treatment. * Positive test for hepatitis B virus (HBV) using HBV surface antigen (HBV sAg) test or positive test for hepatitis C virus (HCV) using HCV ribonucleic acid (RNA) or HCV antibody test indicating acute or chronic infection. * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS). * History of allergy to study drug component or history of severe hypersensitivity reaction to any monoclonal antibody. * Women who are breastfeeding or pregnant as evidenced by a positive pregnancy test within 14 days of first dose. * Male subjects who are unwilling to use contraception during the treatment and for at least 31 weeks after the last dose of study treatment (5 half-lives of study drug plus 90 days duration of sperm turnover). * Women of childbearing potential (WOCBP) not using a medically acceptable means of contraception throughout the study treatment and for at least 23 weeks following the last dose of study treatment (5 half-lives of study drug plus 30 days duration of ovulatory cycle). * WOCBP are defined as those who have experienced menarche and who have not undergone successful surgical sterilization (hysterectomy, bilateral tubal ligation, or bilateral oophorectomy) or are not postmenopausal. Post-menopausal is defined as: * Amenorrhea ≥ 12 consecutive months without another cause, or * For women with irregular menstrual periods and on hormone replacement therapy (HRT), a documented serum follicle stimulating hormone (FSH) level \> 35 mIU/mL * Subjects who are compulsorily detained for treatment of either a psychiatric or physical (e.g., infectious disease) illness. * Inability to comply with study and/or follow-up procedures.