Tremelimumab and Durvalumab in Treating Patients With Colorectal Cancer With Liver Metastases That Can Be Removed by Surgery

Inclusion Criteria: * Patients must have histologically or cytologically confirmed colorectal cancer with liver metastases deemed resectable by a general or liver surgeon (resectability may involve the use of ablative techniques to some but not all liver metastases); those patients with known disease outside of the liver are not eligible (except for patients with primary lesions in place that are planned for resection or nonspecific lung metastases \< 1 cm) * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) as \>= 10 mm with spiral computed tomography (CT) scan * All lines of prior therapy accepted; subjects with prior hepatic or extra-hepatic resections of metastatic disease will be included * Life expectancy of greater than 6 months * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 70%) * Leukocytes \>= 3,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin \< 1.5 X institutional normal limits (subjects with known Gilbert syndrome are eligible with total bilirubin \< 3.0 mg/dL) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamic-pyruvic transaminase \[SGPT\]) =\< 3 X institutional upper limit of normal * Creatinine within normal institutional limits OR creatinine clearance \>= 60 mL/min/1.73 m\^2 for patients with creatinine levels above institutional normal * Known or ordered molecular testing for MSI, BRAF, and KRAS status * Evidence of post-menopausal status or negative urinary or serum pregnancy test for female pre-menopausal patients; women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause; the following age-specific requirements apply: * Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy) * Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \> 1 year ago, had chemotherapy-induced menopause with last menses \> 1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy) * Ability to understand and the willingness to sign a written informed consent document * Weight \> 30 kg (required for flat dose-based administration of study agents) Exclusion Criteria: * Prior chemotherapy \< 2 weeks prior to study drug treatment and treatment related adverse events that have not recovered to baseline or grade 1 (alopecia excluded); prior radiation therapy \< 4 weeks prior to study drug treatment * Patients may not be receiving any other investigational agents * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc.\]); the following are exceptions to this criterion: a) patients with vitiligo or alopecia; b) patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement; c) any chronic skin condition that does not require systemic therapy; d) patients without active disease in the last 5 years may be included but only after consultation with the study physician; d) patients with celiac disease controlled by diet alone * Subjects with a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration; inhaled or topical steroids and adrenal replacement doses \> 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease * Prior exposure to T cell checkpoint inhibitor therapies, including durvalumab and tremelimumab * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, interstitial lung disease, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Known allergy or hypersensitivity to any of the study drugs or any of the study drug excipients * History of active primary immunodeficiency * Women who are pregnant, which includes women with a positive pregnancy test at enrollment or prior to the administration of study medication, or breastfeeding are not allowed on study * Receipt of a live vaccine within 30 days of study entry * Any unresolved toxicity National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade \>= 2 from previous anticancer therapy with the exception of alopecia, vitiligo, and the laboratory values defined in the inclusion criteria: a) patients with grade \>= 2 neuropathy will be evaluated on a case-by-case basis after consultation with the study physician; b) patients with irreversible toxicity not reasonably expected to be exacerbated by treatment with durvalumab or tremelimumab may be included only after consultation with the study physician * Any concurrent chemotherapy, biologic, or hormonal therapy for cancer treatment, concurrent use of hormonal therapy for non-cancer-related conditions (e.g., hormone replacement therapy) is acceptable * Major surgical procedure within 28 days prior to the first dose of IP; Note: local surgery of isolated lesions for palliative intent is acceptable * History of allogenic organ transplantation * Known active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and tuberculosis \[TB\] testing in line with local practice), hepatitis B (known positive hepatitis B virus \[HBV\] surface antigen (HBsAg) result), hepatitis C, or human immunodeficiency virus (positive HIV 1/2 antibodies); patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible; patients positive for hepatitis C (hepatitis C virus \[HCV\]) antibody are eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA) * Female patients who are pregnant or breastfeeding or male or female patients of reproductive potential who are not willing to employ effective birth control from screening to 90 days after the last dose of durvalumab monotherapy or 180 days after the last dose of durvalumab + tremelimumab combination therapy