Utomilumab, Cetuximab, and Irinotecan Hydrochloride in Treating Patients With Metastatic Colorectal Cancer

Inclusion Criteria: * Patients must have histologically and or cytologically confirmed metastatic colorectal cancer * Patients must have a wild type or mutated RAS tumor status known prior to enrollment * Metastatic colorectal cancer patients have progressed following at least one line of fluorouracil (5-FU)-based chemotherapy * Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 2 * Patients must have measurable disease per irRECIST criteria for part 2 (dose expansion) * Absolute neutrophil count (ANC) \>= 1.0 x 10\^9/L (1,000/uL) * Platelet count \>= 75 x 10\^9/L (75000/L) * Hemoglobin \>= 8.0 g/dL (\>= 5.0 mmol/L) * Patients must be transfusion independent (i.e., no blood product transfusions for a period of at least 14 days prior to screening) * Serum creatinine \< 2 x upper limit of normal (ULN) or estimated creatinine clearance \> 30 ml/min as calculated using the method standard for the institution * Total serum bilirubin \< 1.5 x ULN, unless the patient has documented Gilbert syndrome * Aspartate and Alanine Aminotransferase (AST and ALT) \< 3 x ULN * Left ventricular ejection fraction (LVEF) that is greater than 40%, or the absence of New York Heart Association (NYHA) classification of greater than stage II congestive heart failure * Resolved acute effects of any prior therapy to baseline severity or grade =\< 2 Common Terminology Criteria for Adverse Events (CTCAE) version (v.) 4.03 except for adverse events (AEs) not constituting a safety risk by investigator judgment * Serum or urine pregnancy test (for females of childbearing potential) negative at screening and at the baseline visit (before the patient may receive the investigational product) * Male and female patients of childbearing potential and at risk for pregnancy must agree to use two highly effective methods of contraception throughout the study and for at least 90 days after the last dose of assigned treatment. Female patients who are not of childbearing potential (permanently sterilized or postmenopausal; i.e., meet at least one of the following criteria): * Have undergone a documented hysterectomy and/or bilateral oophorectomy; or * Have medically confirmed ovarian failure; or * Achieved postmenopausal status, defined as follows: cessation of regular menses for at least 12 consecutive months with no alternative pathological or physiological cause; status may be confirmed by having a serum follicle stimulating hormone (FSH) level within the laboratory's reference range for postmenopausal women * High microsatellite instability (MSI-H) colorectal cancer patients must have received an approved PD-1 targeted agent prior to enrolling in this trial * Evidence of a personally signed and dated informed consent document indicating that the patient has been informed of all pertinent aspects of the study * Co-consent for protocol LAB10-0982 * Willingness and ability to comply with the study scheduled visits, treatment plans, laboratory tests and other procedures Exclusion Criteria: * Patients with known symptomatic brain metastases requiring steroids. Patients with previously diagnosed brain metastases are eligible if they are asymptomatic or have completed their treatment and have recovered from the acute effects of radiation therapy or surgery prior to the start of study medication, have discontinued corticosteroid treatment for these metastases for at least 4 weeks and are neurologically stable * Patient has had any treatment specific for tumor control within 3 weeks of dosing, or for investigational drugs and cytotoxic agents, within 5 half-lives or 3 weeks, whichever is shorter * Patients receiving any medications or substances that are strong inhibitors or inducers of CYP3A4 complex. Lists including medications and substances known or with the potential to interact with the CYP3A4 isoenzymes * Prior therapy with a compound of the same mechanism as PF-05082566 (immunomodulation of 4-1BB) * Major surgery within 28 days of starting study treatment * Radiation therapy within 14 days of starting study treatment * Autoimmune disorders (e.g., Crohn's disease, rheumatoid arthritis, scleroderma, systemic lupus erythematosus) and other diseases that compromise or impair the immune system except patients who have grade 1 psoriasis (in remission or controlled with topical steroids) or mild degree of autoimmune thyroiditis that are controlled with medications * Active and clinically significant bacterial, fungal or viral infection including hepatitis B (HBV), hepatitis C (HCV), known human immunodeficiency virus (HIV) or acquired immunodeficiency syndrome (AIDS)-related illness (HIV testing is not required) * Unstable or serious concurrent medical conditions in the previous 6 months, e.g., pancreatitis, severe/unstable angina, prolonged QT interval corrected by Fridericia's formula (QTcF) \> 470 msec (calculated as average of triplicate readings, taken no greater than 2 minutes apart, and no history of torsades de pointes or symptomatic corrected QT \[QTc\] abnormality), symptomatic congestive heart failure, myocardial infarction and/or pulmonary hypertension, ongoing maintenance therapy for life-threatening ventricular arrhythmia, stroke, and uncontrolled major seizure disorder * Concurrent active malignancy other than non-melanoma skin cancer or carcinoma in situ of the cervix * Patients who are pregnant or breastfeeding * Patients with intolerance to or who have had a severe allergic or anaphylactic reaction to antibodies or infused therapeutic proteins, or patients who have had a severe allergic or anaphylactic reaction to any of the substances included in the study drug (including excipients) * Other severe acute or chronic medical or psychiatric condition, including recent (within the past year) or active suicidal ideation or behavior, or laboratory abnormality that may increase the risk associated with study participation or investigational product administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for entry into this study