Evaluating Immune Therapy, Durvalumab (MEDI4736) With Tremelimumab for Metastatic, Non-transitional Cell Carcinoma of the Urinary Tract

Inclusion Criteria: * Age ≥ 18 years at time of informed consent * Body weight \> 30 kg * Histologically or cytologically confirmed small cell carcinoma, squamous cell carcinoma or adenocarcinoma (confirmed at MSKCC) of the bladder, ureter, urethra, urachus, or renal pelvis. Patients with squamous cell carcinoma and adenocarcinoma are required to have a predominant squamous or adenocarcinoma component as reviewed by the pathologist at MSKCC. However, if any element of small cell or neuroendocrine differentiation is present, the patients will be classified as small cell/neuroendocrine. * Confirmation of availability of sufficient tissue from a prior surgery for correlative studies is required prior to enrollment. Patients must have representative non-TCC or the urothelial tract FFPE archival tumor specimens (tumor blocks or 30 unstained slides; preference for tumor blocks). These samples may be submitted between the time of consent and the start of treatment. Patients with \< 30 slides may be enrolled after discussion with the principal or co-principal investigators. * Clinical evidence of metastatic (T4b, any N; any T, N2-3; M1) disease. * Life expectancy of 12 weeks of greater based on assessment by the treating investigator. * Evidence of measurable disease by RECIST 1.1. * Patients with small cell carcinoma must have progressed after at least one prior systemic therapy. Patients with squamous cell carcinoma or adenocarcinoma may be previously untreated or have progressed after prior systemic therapy. Chemotherapy administered in conjunction with primary radiation as a radio-sensitizer WILL be counted as a systemic chemotherapy regimen. NOTE: There is no maximum number of prior treatments allowed. * Patients with brain metastases are allowed onto the study as long as patients have completed their treatment for brain metastasis, no longer require corticosteroids, and are asymptomatic. Subjects with neurological symptoms should undergo a head CT scan or brain MRI to exclude brain metastasis, at the discretion of the treating physician. * Eastern Cooperative Oncology Group (ECOG) performance status of 0-1. * Adequate normal organ and marrow function as defined below: * Hemoglobin ≥ 9.0 g/dL * Absolute neutrophil count (ANC) ≥ 1.0 x 109/L (\> 1000 per mm\^3) * Platelet count ≥ 100 x 109/L (\>100,000 per mm\^3) * Serum bilirubin ≤ 1.5 x institutional upper limit of normal (ULN) (≤ 3 x institutional ULN in patients with Gilbert"s syndrome) * AST (SGOT)/ALT (SGPT) ≤ 2.5 x institutional ULN unless liver metastases are present, in which case it must be ≤ 5 x ULN * Calculated creatinine clearance \> 30 mL/min by the Cockcroft-Gault formula (Cockcroft and Gault 1976) or by 24-hour urine collection for determination of creatinine clearance: Males: Creatinine CL (mL/min) = Weight (kg) x (140 - Age) . 72 x serum creatinine (mg/dL) Females: Creatinine CL (mL/min) * Weight (kg) x (140 - Age) x 0.85 72 x serum creatinine (mg/dL) * Evidence of post-menopausal status or negative serum pregnancy test for female pre-menopausal patients. Women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause. The following age-specific requirements apply: * Women \<50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy). * Women ≥50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \>1 year ago, had chemotherapy-induced menopause with last menses \>1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy). * Female patients of reproductive potential and non-sterilized males who are sexually active with a female partner of childbearing potential must be willing to adhere to the following restrictions: * Females of reproductive potential who are sexually active with a non-sterilized male partner must agree to use at least 1 highly effective method of contraception (Table 3) from the time of screening until 180 days after the last dose of durvalumab + tremelimumab combination therapy or 90 days after the last dose of durvalumab monotherapy. Cessation of birth control after this point should be discussed with a responsible physician. Periodic abstinence, the rhythm method, and the withdrawal method are not acceptable methods of birth control. It is strongly recommended that non-sterilized male partners of a female patient must use male condom plus spermicide throughout this period. Not engaging in sexual activity for the total duration of the drug treatment and the drug washout period is an acceptable practice. * Non-sterilized males who are sexually active with a female partner of childbearing potential must use a male condom plus spermicide from screening through 180 days after receipt of the final dose of durvalumab + tremelimumab combination therapy or 90 days after receipt of the final dose of durvalumab monotherapy. Periodic abstinence, the rhythm method, and the withdrawal method are not acceptable methods of birth control. Not engaging in sexual activity is an acceptable practice. Male patients should refrain from sperm donation throughout this period. It is strongly recommended that female partners (of childbearing potential) of male patients to also use a highly effective method of contraception throughout this period. * Highly effective methods of contraception, defined as one that results in a low failure rate (i.e., less than 1% per year) when used consistently and correctly. * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up. Exclusion Criteria: * Previous enrollment in the present study * Participation in another clinical study with an investigational product during the last 14 days * Mean QT interval corrected for heart rate (QTc) ≥470 ms calculated from 3 electrocardiograms (ECGs) using Fredericia"s Correction * Any previous treatment with a PD-1 or PD-L1 inhibitor, including durvalumab or an anti-CTLA-4, including tremelimumab * Receipt of the last dose of anti-cancer therapy (chemotherapy, immunotherapy, endocrine therapy, targeted therapy, biologic therapy, tumor embolization, monoclonal antibodies, other investigational agent) \< 21 days prior to enrollment. * Major surgery within 28 days of starting study treatment. There is no minimum time requirement for minor procedures such as biopsy or vascular access placement. * Radiation within 14 days of starting study treatment * Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab or tremelimumab. The following are exceptions to this criterion: * Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection) * Systemic corticosteroids at physiologic doses not to exceed \<\<10 mg/day\>\> of prednisone or its equivalent * Steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) * Any unresolved toxicity from previous anti-cancer therapy must have resolved to at least ≤ Grade 1 (or baseline) at time of enrollment. Patients with irreversible toxicity that is not reasonably expected to be exacerbated by treatment with durvalumab and tremelimumab may be included after consultation with the Principal Investigator or Co-Principal Investigator (e.g. alopecia, hearing loss, peripheral neuropathy). * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis, celiac disease, systemic lupus erythematosus, Sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc\]). The following are exceptions to this criterion: * Patients with vitiligo or alopecia * Patients with hypothyroidism (eg, following Hashimoto syndrome) stable on hormone replacement * Any chronic skin condition that does not require systemic therapy * Patients without active autoimmune disease in the last 5 years may be included but only after consultation with the study physician * Patients with diverticulosis * Patients with celiac disease controlled by diet alone * History of primary immunodeficiency * History of allogeneic organ transplant * History of hypersensitivity to durvalumab, tremelimumab or any excipient * Uncontrolled intercurrent illness, including but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, interstitial lung disease, serious chronic gastrointestinal conditions associated with diarrhea, or psychiatric illness/social situations that would limit compliance with study requirement, substantially increase risk of incurring AEs or compromise the ability of the patient to give written informed consent. * Active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and TB testing in line with local practice), hepatitis B (known positive HBV surface antigen (HBsAg) result), hepatitis C, or human immunodeficiency virus (positive HIV 1/2 antibodies). Patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible. Patients positive for hepatitis C (HCV) antibody are eligible only if polymerase chain reaction is negative for HCV RNA. * History of leptomeningeal carcinomatosis * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab or tremelimumab. Inactivated vaccines, such as the injectable influenza vaccine, are permitted. * Any condition that, in the opinion of the investigator, would interfere with evaluation of study treatment or interpretation of patient safety or study results * Malignancies other than the disease under study within 5 years prior to Cycle 1, Day 1, with the exception of those with a negligible risk of metastasis or death and with expected curative outcome (such as adequately treated carcinoma in situ of the cervix, basal or squamous cell skin cancer, localized prostate cancer treated surgically with curative intent, or ductal carcinoma in situ treated surgically with curative intent) or undergoing active surveillance per standard-of-care management (e.g. prostate cancer with Gleason score ≤ 6, and prostate-specific antigen \[PSA\] ≤ 10 mg/mL, etc). * Patients should agree to not donate blood while participating in this study or for at least 90 days following the last infusion of durvalumab or tremelimumab * Female patients who are pregnant or breastfeeding or male or female patients of reproductive potential who are not willing to employ effective birth control from screening to 90 days after the last dose of durvalumab monotherapy or180 days after the last dose of durvalumab + tremelimumab combination therapy.