Encorafenib, Binimetinib, and Nivolumab in Treating Microsatellite Stable BRAF V600E Metastatic Colorectal Cancer

Inclusion Criteria: * Participants must have histologically or cytologically confirmed diagnosis of adenocarcinoma of the colon or rectum, with clinical confirmation of unresectable and/or metastatic disease that is measurable according to Response Evaluation Criteria in Solid Tumors (RECIST 1.1) criteria * Confirmation of BRAFV600E tumor as detected from testing performed in a Clinical Laboratory Improvement Act (CLIA)-certified laboratory * Confirmation of MSS status from testing performed in a CLIA-certified laboratory * Prior treatment with at least one systemic chemotherapy regimen for mCRC, or recurrence/progression with development of unresectable or metastatic disease within 6 months of adjuvant chemotherapy for resected CRC * Eastern Cooperative Oncology Group performance status (ECOG PS) =\< 1 (Karnofsky \>= 70%) * Absolute neutrophil count (ANC) \>= 1.0 x 10\^9/L * Hemoglobin (Hgb) \>= 9 g/dL with or without transfusions * Platelets (PLT) \>= 100 x 10\^9/L without transfusions * Total bilirubin =\< 1.5 x upper limit of normal (ULN) and \< 2 mg/dL * Note: Patients who have a total bilirubin level \> 1.5 x ULN and/or have Gilbert's disease will be allowed if their direct bilirubin level is =\< 0.5 mg/dl or ULN, whichever is higher. * Aspartate aminotransferase (AST) and/or alanine aminotransferase (ALT) =\< 3 x ULN * Serum creatinine =\< 1.5 x ULN, or calculated creatinine clearance (determined as per Cockcroft-Gault) \>= 50 mL/min at screening * Corrected QT (QTc) interval =\< 480 ms (preferably the mean from triplicate electrocardiograms \[ECGs\]) * Ability to understand a written informed consent document, and the willingness to sign it * The effects of the study drugs on the developing human fetus are unknown. Female patients must either be postmenopausal for at least 1 year, surgically sterile for at least 6 weeks, or must agree to take appropriate precautions to avoid pregnancy from screening through 5 months after discontinuation of study treatment if of childbearing potential * Female participants of childbearing potential (WOCBP) must agree to use adequate contraception: \*see list below this paragraph\* for the duration of study participation and for 5 months (i.e., 30 days \[duration of ovulatory cycle\] plus the time required for the investigational drug to undergo approximately five half-lives) after last administration of study treatment. Only for all females of childbearing potential, the pregnancy test result must be negative within 24 hours of starting treatment. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately * Males who are sexually active with WOCBP must agree to follow instructions for method(s) of contraception \*see list below this paragraph\* for the duration of study treatment with nivolumab and 7 months after the last dose of study treatment (i.e., 90 days \[duration of sperm turnover\] plus the time required for the investigational drug to undergo approximately five half-lives.) * Accepted means of contraception: * Complete abstinence from sexual intercourse when this is in line with the preferred and usual lifestyle of the patient * Double barrier methods * Condom with spermicide in conjunction with use of an intrauterine device * Condom with spermicide in conjunction with use of a diaphragm * Birth control patch or vaginal ring * Oral, injectable, or implanted contraceptives * Due to the potential of encorafenib to induce CYP3A4, hormonal agents (including but not limited to birth control patch, vaginal ring, oral, injectable, or implanted contraceptives) are permissible only when combined with other highly effective or acceptable methods * Surgical sterilization (bilateral oophorectomy with or without hysterectomy, tubal ligation or vasectomy) at least 6 weeks prior to taking study treatment. In the case of oophorectomy alone, only when the reproductive status of the woman has been confirmed by follow-up levels of luteinizing hormone, follicle-stimulating hormone, and/or estradiol * Men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 100 days after last administration of study treatment * Able to take oral medications Exclusion Criteria: * Concurrent corticosteroid therapy or concurrent use of any other immunosuppressive medication (corticosteroid use on study as a pre-medication for IV contrast allergies/reactions is allowed). Subjects who are receiving daily steroid replacement therapy (the equivalent of prednisone =\< 10 mg daily) serve as an exception to this rule * Prior immune checkpoint therapy including, but not limited to, an anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CD137, or anti-cytotoxic T-lymphocyte-associated antigen-4 (CTLA-4) antibody, or any other prior immunotherapy agent administered with antineoplastic intent * Prior B-raf (BRAF)- or mitogen-activated extracellular kinase (MEK)-targeted therapy * Known hypersensitivity or contraindication to any component of binimetinib or encorafenib or their excipients * Prior allogeneic tissue/solid organ transplant * Interstitial lung disease (ILD) or history of pneumonitis that has required oral or IV steroids * Receipt of a live vaccine within 30 days prior to the first administration of study medication. Seasonal flu vaccines that do not contain a live virus are permitted * History of a grade 3 or 4 allergic reaction attributed to humanized or human monoclonal antibody therapy * Active infection requiring concurrent antibiotic use * Any symptomatic brain metastasis * Note: Patients previously treated or untreated for this condition who are asymptomatic in the absence of corticosteroid and anti-epileptic therapy are allowed. Brain metastases must be stable for \>= 4 weeks, with imaging (e.g., magnetic resonance imaging \[MRI\] or computerized tomography \[CT\]) demonstrating no current evidence of progressive brain metastases at screening * Leptomeningeal disease * Previous or concurrent malignancy within 3 years of study entry, with the following exceptions: adequately treated basal or squamous cell skin cancer, superficial bladder cancer, prostate intraepithelial neoplasm, carcinoma in-situ of the cervix, or other noninvasive or indolent malignancy; other solid tumors treated curatively without evidence of recurrence for at least 3 years prior to study entry * Impaired cardiovascular function or clinically significant cardiovascular diseases, including any of the following: * History of acute coronary syndromes (including myocardial infarction, unstable angina, coronary artery bypass grafting, coronary angioplasty, or stenting) \< 6 months prior to screening * Symptomatic chronic heart failure (i.e. grade 2 or higher), history or current evidence of clinically significant cardiac arrhythmia and/or conduction abnormality \< 6 months prior to screening (including resting bradycardia, uncontrolled atrial fibrillation or uncontrolled paroxysmal supraventricular tachycardia) * Left ventricular ejection fraction (LVEF) \< 50% as determined by multi-gated acquisition (MUGA) or echocardiogram (ECHO) * Uncontrolled hypertension defined as persistent elevation of systolic blood pressure \>= 160 mmHg or diastolic blood pressure \>= 100 mm Hg, despite current therapy * Known positive serology for HIV (human immunodeficiency virus), active hepatitis B, and/or active hepatitis C infection * Known history of acute or chronic pancreatitis (history of acute pancreatitis with no recurrent events in the prior 24 months are permitted) * Patients with a history of inflammatory bowel disease, including ulcerative colitis and Crohn?s disease, are excluded from this study, as are patients with a history of symptomatic autoimmune disease (e.g., rheumatoid arthritis, systemic progressive sclerosis \[scleroderma\], systemic lupus erythematosus, autoimmune vasculitis \[e.g., Wegener?s granulomatosis\]); central nervous system (CNS) or motor neuropathy considered of autoimmune origin (e.g., Guillain-Barre Syndrome and myasthenia gravis, multiple sclerosis). Patients with Graves? disease will be allowed * Impaired gastrointestinal (GI) function or disease that may significantly alter the absorption of study drug (e.g., ulcerative diseases, uncontrolled vomiting, malabsorption syndrome, small bowel resection with decreased intestinal absorption) * Concurrent neuromuscular disorder that is associated with elevated creatine kinase (CK) (e.g., inflammatory myopathies, muscular dystrophy, amyotrophic lateral sclerosis, spinal muscular atrophy) * History or current evidence of retinal vein occlusion (RVO) or current risk factors for RVO (e.g., uncontrolled glaucoma or ocular hypertension, history of hyperviscosity or hypercoagulability syndromes); history of retinal degenerative disease * History of thromboembolic or cerebrovascular events =\< 12 weeks prior to the first dose of study treatment. Examples include transient ischemic attacks, cerebrovascular accidents, hemodynamically significant (i.e. massive or sub-massive) deep vein thrombosis or pulmonary emboli * Note: Patients with either deep vein thrombosis or pulmonary emboli that does not result in hemodynamic instability are allowed to enroll as long as they are on a stable dose of anticoagulants for at least 4 weeks * Note: Patients with thromboembolic events related to indwelling catheters or other procedures may be enrolled * Any other condition that would, in the investigator?s judgment, contraindicate the patient?s participation in the clinical study due to safety concerns or compliance with clinical study procedures * Major surgery =\< 6 weeks prior to starting study drug or failure to recover from side effects of such procedure at the discretion of the treating investigator * Pregnant or nursing (lactating) females, where pregnancy is defined as the state of a female after conception and until the termination of gestation, confirmed by a positive human chorionic gonadotropin (hCG) laboratory test * Prisoners or persons who are involuntarily incarcerated * Medical, psychiatric, cognitive or other conditions that may compromise the patient's ability to understand the patient information, give informed consent, comply with the study protocol or complete the study