Durvalumab and Tremelimumab in Treating Patients With Microsatellite Stable Metastatic Colorectal Cancer to the Liver

Inclusion Criteria: * Documented informed consent of the subject and/or legally authorized representative * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 * Life expectancy of \> 12 weeks * Hemoglobin \>= 9.0 g/dL * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (\>= 1500 per mm\^3) * Platelet count \>= 75 x 10\^9/L (\>= 75,000 per mm\^3) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN) * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 5 x institutional upper limit of normal given that all patients have liver metastases * Serum creatinine clearance (CL) \> 40 mL/min by the Cockcroft-Gault formula or by 24-hour urine collection for determination of creatinine clearance * Female subjects must either be of non-reproductive potential (ie, post-menopausal by history: \>= 60 years old and no menses for \>= 1 year without an alternative medical cause; AND/OR history of hysterectomy, AND/OR history of bilateral tubal ligation, AND/OR history of bilateral oophorectomy) or must have a negative serum pregnancy test upon study entry * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up * Patients must have received at least one prior line of therapy for the treatment of metastatic disease with a fluoropyrimidine in combination with oxaliplatin and/or irinotecan; patients with prior adjuvant therapy who progressed within 6 months of completion of treatment may be eligible * Patients must have liver-only metastases or predominant liver metastatic disease * Patients should have microsatellite stable (MSS) tumor by polymerase chain reaction (PCR) assay or mismatch repair protein proficient (MMRP) tumor by immunohistochemistry as confirmed by the presence of MLH1, MSH2, MSH6, and PMS2; the diagnosis of colorectal cancer should be confirmed by pathology either on the primary tumor or from a prior biopsy of a metastatic disease site * Patients should have been identified by their respective physicians as candidates for radioembolization and scheduled to undergo such a procedure * Patients should agree to serial liver metastases biopsy pre-treatment, post-radioembolization, and post-combination immunotherapy * Patients should have measurable metastatic disease in the liver, defined (for the purpose of this study) as at least 1 measurable lesion more than 2 cm in size and readily accessible to ultrasound (US) or computed tomography (CT)-guided biopsy * Patients should not be deemed candidate for curative hepatic resection Exclusion Criteria: * Involvement in the planning and/or conduct of the study (applies to both AstraZeneca staff and/or staff at the study site) or previous enrollment in the present study * Participation in another clinical study with an investigational product during the last 4 weeks * Any previous treatment with a PD1 or PD-L1 inhibitor, including durvalumab or an anti-CTLA4, including tremelimumab * History of another primary malignancy except for: * Malignancy treated with curative intent and with no known active disease \>= 2 years before the first dose of study drug and of low potential risk for recurrence * Adequately treated non-melanoma skin cancer or lentigo maligns without evidence of disease * Adequately treated carcinoma in situ without evidence of disease eg, cervical cancer in situ * Receipt of the last dose of chemotherapy or tyrosine kinase inhibitors should be at least 3 weeks prior to durvalumab and tremelimumab dosing; monoclonal antibodies such as bevacizumab, ziv-aflibercept, ramucirumab, cetuximab, and panitumumab should be at least 6 weeks prior to durvalumab and tremelimumab therapy * Clinical ascites * Liver involvement by \> 50% with metastatic disease determined by the investigator * Complete portal vein thrombosis on CT scans * Failure to satisfy minimum criteria of lung shunting (\> 20%) or presence of extrahepatic gastrointestinal activity on microaggregated albumin (MAA) scan or angiogram that preclude SIR-Spheres * Prior external beam radiation to the liver * Mean QT interval corrected for heart rate (QTc) \>= 470 ms calculated from 3 electrocardiograms (ECGs) using Fridericia's correction * Current or prior use of immunosuppressive medication within 28 days before the first dose of durvalumab or tremelimumab, with the exceptions of intranasal, topical, and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid * Any unresolved toxicity (\> Common Terminology Criteria for Adverse Events \[CTCAE\] grade 2) from previous anti-cancer therapy; subjects with irreversible toxicity that is not reasonably expected to be exacerbated by the investigational product may be included (e.g., hearing loss, peripherally neuropathy) * Any prior systemic anti-cancer immunotherapy treatment * Active or prior documented autoimmune disease within the past 2 years; NOTE: Subjects with vitiligo, Grave's disease, or psoriasis not requiring systemic treatment (within the past 2 years) are not excluded * Active or prior documented inflammatory bowel disease (e.g., Crohn's disease, ulcerative colitis) * History of primary immunodeficiency * History of allogeneic organ transplant * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, active peptic ulcer disease or gastritis, active bleeding diatheses including any subject known to have evidence of acute or chronic hepatitis B, hepatitis C or human immunodeficiency virus (HIV), or psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent * Known history of previous clinical diagnosis of tuberculosis * History of leptomeningeal carcinomatosis * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab or tremelimumab * Any condition that, in the opinion of the investigator, would interfere with evaluation of study treatment or interpretation of patient safety or study results * Symptomatic or uncontrolled brain metastases requiring concurrent treatment, inclusive of but not limited to surgery, radiation and/or corticosteroids * Subjects with uncontrolled seizures * Patients with symptomatic extrahepatic metastases * Female patients who are pregnant or breastfeeding or male or female patients of reproductive potential who are not willing to employ effective birth control from screening to 180 days after the last dose of durvalumab + tremelimumab combination therapy or 90 days after the last dose of durvalumab monotherapy, whichever is the longer time period * Known allergy or hypersensitivity to investigational product (IP) or any excipient