Cetuximab and Pembrolizumab in Treating Patients With Colorectal Cancer That is Metastatic or Cannot Be Removed by Surgery

Inclusion Criteria: * Have a pathologically confirmed diagnosis of colorectal cancer, which is metastatic or otherwise unresectable * Have received at least 1 prior systemic therapy in the metastatic or unresectable disease setting; patients who have recurred within six months of adjuvant chemotherapy are not required to have received an additional line of chemotherapy * Retrovirus-associated deoxyribonucleic acid (DNA) sequence (RAS) wild-type; v-Ki-ras2 Kirsten rat sarcoma viral oncogene homolog (KRAS) testing must be completed, with full KRAS and neuroblastoma RAS viral (v-ras) oncogene homolog (NRAS) testing strongly advised; the presence of known mutations in KRAS or NRAS is exclusionary; primary tumor or metastatic tumor may be tested; (note: in the case of multiple genomic evaluations with conflicting results - e.g. KRAS mutant in one sample, but wild-type in another - the patient may be included as RAS wild-type, if clinically justified, after review with the principal investigator \[PI\]) * Appropriate for anti-EGFR therapy: Naive to anti-EGFR therapy (cetuximab or panitumumab) or a candidate for rechallenge by virtue of the following: 1. the investigator deems anti-EGFR retreatment with cetuximab to be a reasonable standard of care option AND 2. outcome of prior anti-EGFR therapy was not rapid progression (i.e. \<= 3 months on therapy) AND 3. prior anti-EGFR therapy was administered \> 6 months prior to the start of protocol therapy * Have an Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 * Have measurable disease per Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria present * Be willing to provide tissue from a newly obtained core or excisional biopsy of a tumor lesion; newly-obtained is defined as a specimen obtained up to 30 days prior to initiation of treatment on day 1 * Hemoglobin \>= 8 g/dL (performed within 14 days of treatment initiation) * Absolute neutrophil count \>= 1000/mm3 (performed within 14 days of treatment initiation) * Platelet count \>= 100,000/mm3 (performed within 14 days of treatment initiation) * Serum creatinine =\< 2 upper limit of normal (ULN) or, \>= 15 mL/min for participants with creatinine levels \> 2 ULN (performed within 14 days of treatment initiation) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 ULN or, =\< 5 ULN for participants with liver metastases (performed within 14 days of treatment initiation) * Female participants of childbearing potential are to have a negative serum pregnancy test * Female participants of child-bearing potential must agree to use an acceptable method of birth control, be surgically sterile, or abstain from heterosexual activity for the course of the study through 120 days after the last dose of study medication; participants of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately * Male participants must agree to use an adequate method of contraception starting with the first dose of study therapy through 120 days after the last dose of study therapy * Participant or legal representative must understand the investigational nature of this study and sign an Independent Ethics Committee/Institutional Review Board approved written informed consent form prior to receiving any study related procedure Exclusion Criteria: * Participants who have had chemotherapy, targeted therapies, radiotherapy, or used an investigational device within 2 weeks prior to the first dose of treatment or those who have not recovered from adverse events (i.e., =\< grade 1 or at baseline) due to agents administered more than 2 weeks earlier; note: participants with =\< grade 2 neuropathy are an exception to this criterion and may qualify for the study * Has a diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment * Has a known history of active TB (Bacillus Tuberculosis) * Hypersensitivity to pembrolizumab or any of its excipients * Prior severe infusion reaction to cetuximab * Has a known additional malignancy that requires active treatment; exceptions include basal cell carcinoma of the skin or squamous cell carcinoma of the skin that has undergone potentially curative therapy or in situ cervical cancer * Has known active central nervous system (CNS) metastases and/or carcinomatous meningitis; participants with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least four weeks prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 7 days prior to trial treatment; this exception does not include carcinomatous meningitis which is excluded regardless of clinical stability * Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs); replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency) is not considered a form of systemic treatment * Has known history of, or any evidence of active, non-infectious pneumonitis * Uncontrolled clinically significant intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, unstable cardiac arrhythmia, or psychiatric illness, substance abuse disorders or social situations that would limit compliance with study requirements * Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment * Has a known history of human immunodeficiency virus (HIV or HIV 1/2 antibodies); testing not required * Has known active hepatitis B (e.g., hepatitis B surface antigen \[HBsAg\] reactive) or hepatitis C (e.g., hepatitis C virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] is detected); testing not required * Has received a live vaccine within 30 days of planned start of study therapy (note: seasonal influenza vaccines for injection are generally inactivated flu vaccines and are allowed; however intranasal influenza vaccines (e.g., Flu-Mist®) are live attenuated vaccines, and shingles are not allowed) * Received an investigational agent within 30 days prior to starting study treatment * Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the subject's participation for the full duration of the trial, or is not in the best interest of the subject to participate, in the opinion of the treating investigator * Unwilling or unable to follow protocol requirements