Durvalumab and Tremelimumab in Treating Chemotherapy Naive Patients With Metastatic Castration-Resistant Prostate Cancer

Inclusion Criteria: * Written informed consent. * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1. * Life expectancy of \>= 52 weeks. * Hemoglobin \>= 11.0 g/dL. * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (\> 1500 per mm\^3). * Platelet count \>= 100 x 10\^9/L (\>100,000 per mm\^3). * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN). This will not apply to subjects with confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only in consultation with their physician. * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]/alanine aminotransferase \[ALT\] serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal. * Serum creatinine clearance (CL) \> 40 mL/min by the Cockcroft-Gault formula (Cockcroft and Gault 1976) or by 24-hour urine collection for determination of creatinine clearance. * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up. * Consent to MD Anderson laboratory protocol PA13-0291 and LAB02-152. * Willing to have peripheral blood mononuclear cells and bone marrow biopsies to be collected prior to receiving the first dose of durvalumab and tremelimumab, after 2-doses and 4-doses of durvalumab and tremelimumab, after 2nd treatment administration and 4th treatment administration. * Histologically or cytologically confirmed adenocarcinoma of the prostate. * Evidence of metastatic disease to the bone seen on most recent bone scan, computed tomography (CT) scan and/or magnetic resonance imaging (MRI). * Asymptomatic or minimally symptomatic patients (do not require narcotics for prostate cancer-related pain). * Tumor progression while on hormone therapy with castrate levels serum testosterone (=\< 1.7 nmol/L or 50 ng/dL) defined by prostate-specific antigen (PSA) and/or radiographic criteria according to the Prostate Cancer Working Group 3 (PCWG3). Castrate levels of testosterone must be maintained by surgical or medical means throughout the conduct of the study. Exclusion Criteria: * Involvement in the planning and/or conduct of the study (applies to both AstraZeneca staff and/or staff at the study site), previous enrollment in the present study. * Participation in another clinical study with an investigational product during the last 4 weeks. * Any previous treatment with a PD1 or PD-L1 inhibitor, including durvalumab or an anti-CTLA4, including tremelimumab. * History of another primary malignancy except for: 1) Malignancy treated with curative intent and with no known active disease \>= 5 years before the first dose of study drug and of low potential risk for recurrence. 2) Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease. 3) Adequately treated carcinoma in situ without evidence of disease (e.g., superficial bladder cancer). * Evidence of visceral metastasis to the liver. * Prior use of taxane-based chemotherapy for treatment of castrate resistant prostate cancer. * Receipt of the last dose of anti-cancer therapy (immunotherapy, endocrine therapy \[e.g., abiraterone acetate, enzalutamide\], targeted therapy, biologic therapy, tumor embolization, monoclonal antibodies, other investigational agent) =\< 28 days prior to the first dose of study drug. (with the exception of any previous treatment with a PD1 or PD-L1 inhibitor, including durvalumab or an anti-CTLA4, including tremelimumab.) * Major surgical procedure (as defined by the investigator) within 28 days prior to the first dose of durvalumab or tremelimumab. Note: Local surgery of isolated lesions for palliative intent is acceptable. * QT interval corrected for heart rate using Fridericia's formula (QTcF) \>= 470 ms. Any clinically significant abnormalities detected, require triplicate electrocardiogram (ECG) results and a mean QT interval corrected for heart rate using Fridericia's formula (QTcF) \>= 470 ms calculated from 3 electrocardiograms (ECGs). * Current or prior use of immunosuppressive medication within 28 days before the first dose of durvalumab or tremelimumab, with the exceptions of: intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid or steroids as pre-medication for hypersensitivity reactions (e.g. CT scan premedication). * Any unresolved toxicity (Common Terminology Criteria for Adverse Events \[CTCAE\] grade \>= 2) from previous anti-cancer therapy. Subjects with irreversible toxicity that is not reasonably expected to be exacerbated by the investigational product may be included (e.g., hearing loss, peripherally neuropathy). * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc.\]). The following are exceptions to this criterion: * Patients with vitiligo or alopecia; * Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement; * Any chronic skin condition that does not require systemic therapy; * Patients without active disease in the last 5 years may be included but only after consultation with the study physician; * Patients with celiac disease controlled by diet alone. * Subjects with history of diverticulitis may be included only after consultation and approval of the study physician. * History of primary immunodeficiency. * History of allogeneic organ transplant. * History of hypersensitivity to the combination of durvalumab and tremelimumab. * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, active peptic ulcer disease or gastritis, active bleeding diathesis including any subject known to have evidence of acute or chronic hepatitis B, hepatitis C or human immunodeficiency virus (HIV), or psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent. * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS). Known history of positive test for hepatitis B virus (HBV) using HBV surface antigen (HBV sAg) test or positive test for hepatitis C virus (HCV) using HCV ribonucleic acid (RNA) or HCV antibody test indicating acute or chronic infection. * History of leptomeningeal carcinomatosis. * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab or tremelimumab. * Any condition that, in the opinion of the investigator, would interfere with evaluation of study treatment or interpretation of patient safety or study results. * Brain metastases or spinal cord compression unless asymptomatic or treated and stable off steroids and anti-convulsants for at least 28 days prior to study treatment start. Patients with suspected brain metastases at screening should have a CT/MRI of the brain prior to study entry. * Subjects with uncontrolled seizures. * Male patients of reproductive potential who are not willing to employ effective birth control from screening to 180 days after the last dose of durvalumab + tremelimumab combination therapy or 90 days after the last dose of durvalumab monotherapy, whichever is the longer time period. * A malignancy \[other than the one treated in this study\] which required radiotherapy or systemic treatment within the past 5 years, or has a \>= 30% probability of recurrence within 24 months (except for non-melanoma skin cancer or Ta urothelial carcinomas)