Testing the Addition of Nivolumab to Standard Treatment for Patients With Metastatic or Unresectable Colorectal Cancer That Have a BRAF Mutation

Inclusion Criteria: * Participants must have a histologically or cytologically confirmed diagnosis of adenocarcinoma of the colon or rectum. The date of diagnosis will be determined according to the pathologic date of diagnosis * Participants must have measurable disease according to RECIST1.1 criteria. Computed tomography (CT) scans or magnetic resonance imaging (MRIs) used to assess measurable disease must have been completed within 28 days prior to registration. CT scans or MRIs used to assess non-measurable disease must have been completed within 42 days prior to registration. All disease must be assessed and documented on the Baseline Tumor Assessment Form * Participants must have documented unresectable and/or metastatic disease on CT or MRI imaging. All disease must be assessed and documented on the Baseline Tumor Assessment Form * Participants must have BRAF\^V600E mutated colorectal cancer as tested in a Clinical Laboratory Improvement Act (CLIA)-certified laboratory * Participants must have proficient mismatch repair (pMMR) or microsatellite stable (MSS) status as tested in a CLIA-certified laboratory and documented by the treating clinician. Proficient mismatch repair status can be determined by intact expression by immunohistochemistry of all 4 mismatch repair proteins (MLH1, MSH2, MSH6, and PMS2). Microsatellite instability can be determined by polymerase chain reaction (PCR) * Participants with brain metastases must have completed surgery or radiation therapy \>= 28 days prior to registration. These participants must have a CT or MRI of the brain showing no new or enlarging lesions within 42 days prior to registration. These participants must also be neurologically asymptomatic and without corticosteroid treatment for at least 7 days prior to registration. Metastatic brain parenchymal disease must have been treated and participant must be off steroids for 7 days prior to registration. The presence of leptomeningeal disease (LMD) is not considered stable disease, and participants with LMD are not eligible for this study * Participants with known evidence of chronic hepatitis B virus (HBV) infection must have undetectable HBV viral load on suppressive therapy within 28 days prior to registration * Participants with a history of hepatitis C virus (HCV) infection must have been treated and cured. Participants with HCV infection who are currently on treatment must have an undetectable HCV viral load within 28 days prior to registration * Participants must have had one or two prior regimens of systemic chemotherapy for metastatic or locally advanced, unresectable disease. (A maintenance regimen of 5-fluorouracil or capecitabine, with or without bevacizumab, should not be counted as a separate line of treatment. The re-introduction of an initially successful induction regimen will not be counted as one additional line of treatment). Prior treatment for metastatic disease is not required for patients who experienced disease recurrence during or within 6 months of completion of adjuvant chemotherapy * Participants must be of age \>= 18 years at the time of informed consent * Participants must have a Zubrod performance status of 0 or 1 * Participants must have a complete medical history and physical exam within 28 days prior to registration * Absolute neutrophil count \>= 1.0 x 10\^3/uL (within 28 days prior to registration) * Hemoglobin \>= 9 g/dL (within 28 days prior to registration) * Platelets \>= 75 x 10\^3/uL (within 28 days prior to registration) * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (within 28 days prior to registration) * Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 3 x institutional ULN (within 28 days prior to registration) * If liver metastases are present, then it is acceptable for AST level =\< 5.0 x ULN, and/or an ALT level =\< 5.0 x ULN (within 28 days prior to registration) * Participants must have serum creatinine =\< the IULN OR measured OR calculated creatinine clearance \>= 50 mL/min using the Cockroft-Gault Formula. This specimen must have been drawn and processed within 28 days prior to registration * Participants must be able to swallow and retain pills * Participants must have adequate cardiac function. Participants with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, must have a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, participants must be class 2 or better * Participants with a known significant risk of corrected QT interval (QTc) prolongation must have an electrocardiogram (ECG) performed within 28 days prior to prior to registration * Note: Per package insert for encorafenib, participants already at risk for development of QTc prolongation include those who "already have or who are at significant risk of developing QTc prolongation, including participants with known long QT syndromes, clinically significant bradyarrhythmias, severe or uncontrolled heart failure and those taking other medicinal products associated with QT prolongation." * Participants must be offered the opportunity to participate in specimen banking. With participant consent, specimens must be collected and submitted via the Southwest Oncology Group (SWOG) Specimen Tracking System * Participants must be informed of the investigational nature of this study and must sign and give informed consent in accordance with institutional and federal guidelines. * Note: As a part of the OPEN registration process, the treating institution's identity is provided in order to ensure that the current (within 365 days) date of institutional review board approval for this study has been entered in the system Exclusion Criteria: * Participants must not have a known positive serology for human immunodeficiency virus (HIV). Encorafenib is contraindicated with concomitant use of non-nucleoside analog reverse transcriptase inhibitors like efavirenz and etravirine. In addition, it is recommended in the investigator brochure of encorafenib to avoid using encorafenib with protease inhibitors. Therefore, because all participants on this study would receive encorafenib for either randomized arm of treatment, participants with HIV who receive these components of highly active antiretroviral therapy (HAART) would be at high risk for complications of drug-drug interaction * Participants must not have had prior treatment with a BRAF inhibitor (including, but not limited to, encorafenib, dabrafenib, or vemurafenib), MEK inhibitor (including, but not limited to, trametinib, selumetinib, or binimetinib), or ERK inhibitor (of note, regorafenib is not considered a BRAF inhibitor for the context of eligibility criteria) * Participants must not have had prior treatment with anti-EGFR therapies * Participants must not have had prior treatment with an anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CTLA-4 antibody, or any other antibody or drug specifically targeting T-cell co-stimulation or immune checkpoint pathways * Participants must not have a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of registration. Inhaled or topical steroids and adrenal replacement doses \< 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease. Patients are permitted to use topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption). Physiologic replacement doses of systemic corticosteroids are permitted, even if \< 10 mg/day prednisone equivalents. A brief course of corticosteroids for prophylaxis (e.g., contrast dye allergy) or for treatment of non-autoimmune conditions (e.g., delayed-type hypersensitivity reaction caused by contact allergen) is permitted, as long as there has been a washout period for corticosteroids of \>= 7 days prior to registration * Participants must not have received a live vaccine within 30 days prior to study registration. Seasonal flu and COVID vaccines that do not contain a live virus are permitted * Participants must not be receiving any other investigational agents * Participants must not have impaired gastrointestinal function or disease that may significantly alter the absorption of study drug (e.g., ulcerative diseases, uncontrolled vomiting, malabsorption syndrome, small bowel resection with decreased intestinal absorption) * Participants must not have a history of inflammatory bowel disease, (including ulcerative colitis and Crohn's disease), symptomatic autoimmune disease (e.g., rheumatoid arthritis, systemic progressive sclerosis \[scleroderma\], systemic lupus erythematosus, autoimmune vasculitis \[e.g., Wegener's Granulomatosis\]); central nervous system (CNS) or motor neuropathy considered of autoimmune origin (e.g., Guillain-Barre Syndrome and myasthenia gravis, multiple sclerosis). * Note: Participants with Graves' disease will be allowed * Participants must not have a history of pneumonitis that has required oral or intravenous (IV) steroids within the last 12 months * Participants must not have a history of a grade 3 or 4 allergic reaction attributed to humanized or human monoclonal antibody therapy * Participants must not have a history of a prior allogeneic tissue or solid organ transplant * Participants must not have a history of acute coronary syndromes (including myocardial infarction, unstable angina, coronary artery bypass grafting, coronary angioplasty, or stenting) within 6 months prior to study registration * Participants must not have uncontrolled blood pressure and hypertension within 28 days prior to registration. * Uncontrolled blood pressure and hypertension is defined as systolic blood pressure (SBP) \>= 160 mmHg or diastolic blood pressure (DBP) \> 100 mmHg within 28 days prior to registration. Participants are permitted to be receiving multiple anti-hypertensive medications (unless otherwise indicated in the study). All blood pressure measurements within the 28 days prior to registration must be SBP \< 160 and DBP =\< 100. An exception can be made by a healthcare provider for a participant with a single blood pressure elevation who upon rechecking has a normal blood pressure * Participants must not have a prior or concurrent malignancy whose natural history or treatment (in the opinion of the treating physician) has the potential to interfere with the safety or efficacy assessment of the investigational regimen or requires concurrent therapy * Participants must not be pregnant or nursing. Individuals who are of reproductive potential must have agreed to use an effective contraceptive method with details provided as a part of the consent process. A person who has had menses at any time in the preceding 12 consecutive months or who has semen likely to contain sperm is considered to be of "reproductive potential." In addition to routine contraceptive methods, "effective contraception" also includes refraining from sexual activity that might result in pregnancy and surgery intended to prevent pregnancy (or with a side-effect of pregnancy prevention) including hysterectomy, bilateral oophorectomy, bilateral tubal ligation/occlusion and vasectomy with testing showing no sperm in the semen * Participants must not be planning treatment with other systemic anti-cancer agents (e.g., chemotherapy, hormonal therapy, immunotherapy) or other treatments not part of protocol-specified anti-cancer therapy including concurrent investigational agents of any type