Durvalumab in Treating Patients With Metastatic Hormone-Resistant Prostate Cancer

Inclusion Criteria: * Patient must have evidence of castration resistant prostate cancer as evidenced by a confirmed rising PSA (per Prostate Cancer Working Group 3 \[PCWG3\] criteria) and a castrate serum testosterone level (i.e. =\< 50 mg/dL); if a subject also received an anti-androgen, he must first progress through antiandrogen withdrawal prior to being eligible; the minimum timeframe to document failure of anti-androgen withdrawal will be four weeks * Patients must have received at least one of the approved products known to improve the overall survival of patients with metastatic castration resistant prostate cancer (i.e. abiraterone, enzalutamide, sipuleucel-t, radium-223, docetaxel or cabazitaxel) * Microsatellite instability as determined by MSI-plus assay * Ability to understand and the willingness to sign a written informed consent * Written informed consent and any locally-required authorization (e.g., Health Insurance Portability and Accountability Act \[HIPAA\] authorization) obtained from the subject prior to performing any protocol-related procedures, including screening evaluations * Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 * Life expectancy of \>= 4 months * Hemoglobin \>= 9.0 g/dL Note: patient may receive blood transfusion to achieve a hemoglobin \>= 9.1 g/dL; however, hemoglobin must be stable at or above 9 g/dL two weeks prior to dosing * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9 /L (\>= 1500 per mm\^3) * Platelet count \>= 100 x 10\^9/L (\>= 100,000 per mm\^3) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN); this will not apply to subjects with confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only in consultation with their physician and the study principal investigator (PI) * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal unless liver metastases are present, in which case it must be =\< 5 x ULN * Serum creatinine clearance (CL) \> 40 mL/min by the Cockcroft-Gault formula or by 24-hour urine collection for determination of creatinine clearance * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up * Body weight \> 30 kg Exclusion Criteria: * Involvement in the planning and/or conduct of the study (applies to both AstraZeneca staff and/or staff at the study site) * Previous enrollment in the present study * Participation in another clinical study with an investigational product during the last 14 days * Any previous treatment with a PD1 or PD-L1 inhibitor, including durvalumab * History of another primary malignancy except for: * Malignancy treated with curative intent and with no known active disease \>= 5 years before the first dose of study drug and of low potential risk for recurrence * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease * Adequately treated carcinoma in situ without evidence of disease (e.g., cervical cancer in situ) * Receipt of the last dose of anti-cancer therapy (chemotherapy, immunotherapy, endocrine therapy, targeted therapy, biologic therapy, tumor embolization, monoclonal antibodies, other investigational agent) =\< 28 days prior to the first dose of study drug (if sufficient wash-out time has not occurred due to the schedule or pharmacokinetics \[PK\] properties of an agent, a longer wash-out period may be required) * Major surgical procedure (as defined by the local/lead site PI) within 28 days prior to the first dose of investigational product (IP); Note: local surgery of isolated lesions for palliative intent is acceptable * Ongoing systemic therapy (other than a luteinizing hormone-releasing hormone agonists \[LHRH\] agonist/antagonist) for prostate cancer including, but not limited to: * CYP-17 inhibitors (e.g. ketoconazole, abiraterone) * Antiandrogens (e.g. bicalutamide, nilutamide) * Second generation antiandrogens (e.g. ARN-509) * Immunotherapy (e.g. sipuleucel-T) * Chemotherapy (e.g. docetaxel, cabazitaxel) * Radiopharmaceutical therapy (e.g. radium-223, strontium-89, samarium- 153) * QT interval corrected for heart rate using Fridericia's formula (QTcF) \>= 70 ms; any clinically significant abnormalities detected, require triplicate electrocardiography (ECG) results and a mean QT interval corrected for heart rate using Fridericia's formula (QTcF) \>= 470 ms calculated from 3 ECGs * Current or prior use of immunosuppressive medication within 28 days before the first dose of durvalumab, with the exceptions of intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid (steroids as pre-med for hypersensitivity reactions eg. computed tomography (CT) scan premedication is acceptable) * Any unresolved toxicity (\> CTCAE grade 2) from previous anti-cancer therapy; subjects with irreversible toxicity that is not reasonably expected to be exacerbated by the investigational product may be included (e.g., hearing loss, peripherally neuropathy) * Any prior grade \>= 3 immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE \> grade 1 * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[eg, colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, Sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc\]); the following are exceptions to this criterion: * Patients with vitiligo or alopecia * Patients with hypothyroidism (eg, following Hashimoto syndrome) stable on hormone replacement * Any chronic skin condition that does not require systemic therapy * Patients without active disease in the last 5 years may be included but only after consultation with the study physician * Patients with celiac disease controlled by diet alone * History of primary immunodeficiency * History of allogeneic organ transplant * History of hypersensitivity to durvalumab or any excipient * Uncontrolled intercurrent illness including, but not limited to, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, active peptic ulcer disease or gastritis, active bleeding diatheses, or psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent * Active infection including tuberculosis (TB), hepatitis B, hepatitis C, or human immunodeficiency virus. Note: TB testing will be at the discretion of the treating physician and should be in line with local practice * History of leptomeningeal carcinomatosis * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab * Patients of reproductive potential who are not employing an effective method of birth control; male patients of reproductive potential who are not willing to employ effective birth control from screening to 90 days after the last dose of durvalumab monotherapy, whichever is the longer time period * Any condition that, in the opinion of the local/lead site PI, would interfere with evaluation of study treatment or interpretation of patient safety or study results * Brain metastases or spinal cord compression unless asymptomatic or treated and stable off steroids and anti-convulsants for at least 14 days prior to study treatment start; patients with suspected brain metastases at screening should have a CT/magnetic resonance imaging (MRI) of the brain prior to study entry * Subjects with uncontrolled seizure