Regorafenib, With Cetuximab or Panitumumab, for the Treatment of Unresectable, Locally Advanced, or Metastatic Colorectal Cancer

Inclusion Criteria: * Histologically proven, unresectable distant metastatic or locally advanced colorectal adenocarcinoma * KRAS, NRAS wild type * BRAF v600E wildtype * Measurable disease * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0, 1, or 2 * Life expectancy of \>= 3 months per estimation of treating physician * Absolute neutrophil count (ANC) \>= 1200/mm\^3 (obtained =\< 7 days prior to randomization) * Platelet count \>= 75,000/mm\^3 (obtained =\< 7 days prior to randomization) * Hemoglobin \>= 9.0 g/dL (obtained =\< 7 days prior to randomization) * Total bilirubin =\< 1.5 x upper limit of normal (ULN) (obtained =\< 7 days prior to randomization) * Alanine aminotransferase (ALT) and aspartate amino-transferase (AST) =\< 2.5 x ULN (=\< 5 x ULN for subjects with liver involvement of their cancer) (obtained =\< 7 days prior to randomization) * Serum creatinine =\< 1.5 x ULN (obtained =\< 7 days prior to randomization) * International normalized ratio (INR)/partial thromboplastin time (PTT) =\< 1.5 x ULN (obtained =\< 7 days prior to randomization) * NOTE: Patients who are therapeutically treated with an agent such as warfarin or heparin will be allowed to participate provided that no prior evidence of underlying abnormality in coagulation parameters exists. Close monitoring of at least weekly evaluations will be performed until INR/PTT is stable based on a measurement that is pre-dose as defined by the local standard of care * Alkaline phosphatase limit =\< 2.5 x ULN (=\< 5 x ULN for patients with liver involvement of their cancer) (obtained =\< 7 days prior to randomization) * Negative serum pregnancy test done =\< 7 days prior to randomization for women of childbearing potential only. * NOTE: Post-menopausal women (defined as no menses for at least 1 year) and surgically sterilized women are not required to undergo a pregnancy test. The definition of adequate contraception will be based on the judgment of the treating physician * Provide informed written consent * Willing to return to enrolling institution for follow-up (during the active monitoring phase of the study) * Disease progression on or intolerable to any of the following: fluoropyrimidine, oxaliplatin and irinotecan * Able to swallow and retain oral medication * Willing to provide tissue and blood samples for correlative research purposes * Willing to allow transfer of tissue and blood samples, clinical information, and outcome data collected from this trial for future research Exclusion Criteria: * Prior treatment with regorafenib, cetuximab or panitumumab * Major surgical procedure, open biopsy, or significant traumatic injury =\< 28 days prior to randomization * Congestive heart failure \> New York Heart Association (NYHA) class 2. * NOTE: Class 3 is defined as marked limitation in activity due to symptoms, even during less-than-ordinary activity, e.g., walking short distances (20-100m). They are comfortable at rest. Class 4 is defined as patients with severe limitations. Experiences symptoms even while at rest. Mostly bed bound * Unstable angina (angina symptoms at rest), new-onset angina (begun =\< 3 months prior to randomization) or myocardial infarction =\< 6 months prior to randomization * Cardiac arrhythmias requiring anti-arrhythmic therapy. Note: Pace makers, beta blockers or digoxin are permitted * Uncontrolled hypertension. (Systolic blood pressure \> 140 mmHg or diastolic pressure \> 90 mmHg despite optimal medical management) * History of or current pheochromocytoma * Arterial or venous thrombotic or embolic events such as cerebrovascular accident (including transient ischemic attacks), deep vein thrombosis or pulmonary embolism =\< 6 months prior to randomization * Ongoing infection \> grade 2 National Cancer Institute (NCI)-Common Terminology Criteria for Adverse Events (CTCAE) version (v)5.0 * Known history of chronic hepatitis B or C * Patients with seizure disorder requiring medication * Symptomatic metastatic brain or meningeal tumors unless the patient is \> 6 months from definitive therapy, has a negative imaging study within 4 weeks of randomization and is clinically stable with respect to the tumor at the time of randomization. Note: Patient must not be undergoing acute steroid therapy or taper (chronic steroid therapy is acceptable provided that the dose is stable for one month prior to and following screening radiographic studies) * History of organ allograft (including corneal transplant) * Evidence or history of bleeding diathesis or any hemorrhage or bleeding event \> CTCAE v5.0 grade 3 =\< 4 weeks prior to randomization * Non-healing wound, ulcer, or bone fracture * Substance abuse, medical, psychological or social conditions that may interfere with the patient?s participation in the study or evaluation of the study results * High-frequency microsatellite instability (MSI-H) patients who have not received prior PD-1 monoclonal antibody (mAb) therapy * Concurrent anti-cancer therapy =\< 3 weeks from randomization (chemotherapy, radiation therapy, surgery, immunotherapy, biologic therapy, or tumor embolization) * Known hypersensitivity to any of the study drugs, study drug classes, or excipients in the formulation * Interstitial lung disease with ongoing signs and symptoms at the time of informed consent * History of known persistent proteinuria of CTCAE v5.0 grade 3 or higher (\>= 3.5 g/24 hrs) * Any malabsorption condition * Unresolved toxicity greater than CTCAE v5.0 grade 1 attributed to any prior therapy/procedure excluding alopecia and oxaliplatin induced neurotoxicity =\< grade 2 * Albumin levels \< 2.5 g/dl * Any of the following because this study involves an agent that has known genotoxic, mutagenic and teratogenic effects: * Pregnant women * Nursing women * Men or women of childbearing potential who are unwilling to employ adequate contraception * NOTE: Men and women of childbearing potential must agree to use adequate contraception beginning at the signing of the informed consent form (ICF) until at least 3 months after the last dose of study drug. The definition of adequate contraception will be based on the judgment of the principal investigator or a designated associate * Co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the treating physician, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens * Known history of human immunodeficiency virus (HIV) infection or active hepatitis B or C infection requiring treatment with antiviral therapy * Receiving any other investigational agent which would be considered as a treatment for the primary neoplasm * Previous or concurrent cancer that is distinct in primary site or histology from colorectal cancer =\< 3 years prior to randomization EXCEPT for cervical cancer in-situ, treated ductal carcinoma in situ of the breast, curatively treated nonmelanoma skin carcinoma, noninvasive aerodigestive neoplasms, or superficial bladder tumor (Ta \[Non-invasive tumor\], Tis \[carcinoma in situ\] and T1 \[Tumor invades lamina propria\]). Note: All cancer treatments for cancers that were distinct in a primary site other than colorectal must be completed at least 3 years prior to randomization (i.e., signature date of the informed consent form) * Pleural effusion or ascites that causes respiratory compromise (\>= CTCAE v5.0 grade 2 dyspnea) * Any condition which, in the treating physician?s opinion, makes the subject unsuitable for trial participation