Panitumumab, Regorafenib, or TAS-102, in Treating Patients With Metastatic and/or Unresectable RAS Wild-Type Colorectal Cancer

Inclusion Criteria: * Most recent Guardant360 blood collection date =\< 90 days prior to randomization. * NOTE: If a patient does not have Guardant360 test results available, enrollment in ACCRU\_GI-1611 (COLOMATE) is strongly encouraged. * Greater than 90 days has elapsed between the most recent treatment with an anti-epidermal growth factor receptor gene (EGFR) therapy (cetuximab or panitumumab) and blood collection for Guardant360 assay. * Age \>= 18 years. * Histologically and/or cytologically confirmed adenocarcinoma of the colon or rectum that is metastatic and/ or unresectable. * Documented wild-type in KRAS and NRAS (codons 12, 13, 59, 61, 117, and 146) and in BRAF codon 600, based on tumor tissue taken from primary or metastatic site prior to receipt of anti EGFR therapy. * Progression, intolerance, or contraindication to: * A fluoropyrimidine (e.g., 5-fluorouracil or capecitabine) * Oxaliplatin * Irinotecan * An anti-vascular endothelial growth factor (VEGF) monoclonal antibody (including but not limited to bevacizumab, ramucirumab, or aflibercept). * If the tumor has deficient mismatch repair proteins (dMMR) or is microsatellite instability (MSI)-high based on tumor tissue testing, an anti-programmed death (PD)-1 monoclonal antibody (including but not limited to nivolumab or pembrolizumab). * Clinical or radiographic progression after treatment with an anti-EGFR monoclonal antibody (cetuximab and/or panitumumab) for at least 3 months (minimum of 6 biweekly treatments or 12 weekly treatments at full or partial dose). * NOTE: Treatments do not need to be administered consecutively. * NOTE: Dose reductions or delays are permitted. * At least one site of disease that is measurable by Response Evaluation Criteria in Solid Tumors (RECIST) criteria that has not been previously irradiated; if the patient has had previous radiation to the target lesion(s), there must be evidence of progression since the radiation. * Life expectancy \>= 3 months per estimation of investigator. * Eastern Cooperative Oncology Group (ECOG) Performance Status (PS) 0, 1, or 2. * Absolute neutrophil count (ANC) \>= 1500/mm\^3 without colony stimulating factor support (obtained =\< 7 days prior to randomization). * Platelet count \>= 75,000 /mm\^3 (obtained =\< 7 days prior to randomization). * Hemoglobin \> 8.0 g/dL (obtained =\< 7 days prior to randomization). * Total bilirubin =\< 1.5 x upper limit of normal (ULN) (obtained =\< 7 days prior to randomization). * Aspartate transaminase (AST) =\< 2.5 x ULN (=\< 5 x ULN for subjects with liver involvement of their cancer) (obtained =\< 7 days prior to randomization). * Alanine aminotransferase (ALT) =\< 2.5 x ULN (=\< 5 x ULN for subjects with liver involvement of their cancer) (obtained =\< 7 days prior to randomization). * Calculated creatinine clearance must be \> 30 ml/min using the Cockcroft-Gault formula or serum creatinine \< 1.5 x ULN (obtained =\< 7 days prior to randomization). * Women of child bearing potential and male partners of women of child bearing potential must agree to use two medically accepted methods of contraception, one of them being a barrier method during the study and for 2 months after the last dose of study drug(s). * Negative serum pregnancy test done =\< 7 days prior to randomization, for women of childbearing potential only. * NOTE: Women of childbearing potential include women who have experienced menarche and who have not undergone successful surgical sterilization (hysterectomy, bilateral tubal ligation, or bilateral oophorectomy) or are not postmenopausal. Postmenopause is defined as amenorrhea \>= 12 consecutive months. * NOTE: Women who have been amenorrheic for 12 or more months are still considered to be of childbearing potential if the amenorrhea is possibly due to prior chemotherapy, anti-estrogens, ovarian suppression or any other reversible treatment. * Ability to complete questionnaire(s) by themselves or with assistance. * Capable of understanding and complying with the protocol requirements and has signed the informed consent document. * Willing to return to enrolling institution for follow-up (during the active monitoring phase of the study). * Willing to provide tissue and blood samples for correlative research purposes. * Willing to allow transfer of tissue and blood samples, clinical information, and outcome data collected from this trial for future research. Exclusion Criteria: * Radiation therapy, hormonal therapy, biologic therapy, experimental therapy, or chemotherapy for cancer \< 21 days prior to randomization. * Therapeutic anticoagulation with Vitamin-K antagonists (e.g., warfarin). * Maximum mutant allele frequency (highest allele frequency reported for any gene mutation) (MAF) less than 0.5% by Guardant360 assay. * Detection of at least one of the following gene mutation(s) or amplification(s) by Guardant360 assay. * BRAFV600E mutation (mutant allele frequency (MAF) \>= 0.5% or amplification. * EGFR mutation (MAF \>= 0.5%). Note: EGFR S492R, K467, and R451C mutations are not an exclusion. * ERBB2 (HER2) amplification. * KRAS mutation (MAF \>= 0.5%) or amplification. * MET amplification. * NRAS mutation (MAF \>= 0.5%) or amplification. * Prior treatment with both TAS-102 and regorafenib (prior treatment with either TAS-102 or regorafenib is permitted). * Unable to swallow oral tablets (crushing of study treatment tablets is not allowed). * Co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the investigator, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens. Note: This includes impaired heart function or clinically significant heart disease. * Not recovered to baseline or Common Terminology for Adverse Events (CTCAE) version (v)5.0 =\< grade 1 from toxicity due to all prior therapies except alopecia, oxaliplatin-related neuropathy, asymptomatic electrolyte abnormalities, and other non-clinically significant adverse events. * Any of the following because this study involves an agent that has known genotoxic, mutagenic and teratogenic effects: * Pregnant women. * Nursing women. * Men or women of childbearing potential who are unwilling to employ adequate contraception. * Patients with known central nervous system (CNS) metastases. Note: Patients with radiated or resected lesions are permitted, provided the lesions are fully treated and inactive (based on repeat imaging \>= 30 days after completion of definitive treatment), patients are asymptomatic, and no steroids to control symptoms related to CNS metastases have been administered for at least 30 days. * Major surgical procedure, open biopsy, or significant traumatic injury =\< 28 days prior to randomization (=\< 56 days for hepatectomy, open thoracotomy, major neurosurgery) or anticipation of need for major surgical procedure during the course of the study. * Serious, non-healing wound, ulcer, or bone fracture. * History of stroke (cerebrovascular accident), transient ischemic attack (TIA), myocardial infarction (MI), unstable angina, cardiac or other vascular stenting, angioplasty, or cardiac surgery =\< 6 months prior to randomization. * History of cardiac arrhythmias requiring anti-arrhythmic therapy other than beta blockers, calcium channel blockers, or digoxin =\< 6 months prior to randomization. * Known history of congestive heart failure - New York Heart Association (NYHA) \>= class II. * Known history of human immunodeficiency virus (HIV) seropositivity, acute or chronic active hepatitis B or C infection, or other serious chronic infection requiring ongoing treatment. * History of interstitial lung disease (e.g., pneumonitis or pulmonary fibrosis) or evidence of interstitial lung disease on baseline chest computed tomography (CT) scan. * Subjects with any previously untreated or concurrent cancer that is distinct in primary site or histology from colorectal cancer except cervical cancer in-situ, treated basal cell carcinoma, or superficial bladder tumor. Note: Subjects surviving a cancer that was curatively treated and without evidence of disease or biochemical relapse (undetectable PSA for prostate cancer) for 3 or more years before randomization are allowed. All cancer treatments must be completed at least 3 years prior to randomization. * Uncontrolled hypertension (systolic pressure \> 150 mm HG or diastolic pressure \> 90 mm Hg \[National Cancer Institute (NCI)-CTCAE v5.0\]) on repeated measurement despite optimal medical management. * Evidence or history of bleeding diathesis or coagulopathy. * Any hemorrhage or bleeding event \>= NCI CTCAE v5.0 grade 3, =\< 4 weeks prior to randomization. * Ongoing active infection \> grade 2 NCI-CTCAE v5.0. * Known or suspected allergy or hypersensitivity to any of the study drugs, study drug classes, or excipients of the formulation given during the course of this trial. * EXCEPTION: Cetuximab * Any known history of malabsorption condition. * Substance abuse, medical, psychological or social conditions that may interfere with the subject's participation in the study or evaluation of the study results. * Use of any herbal remedy (e.g. St. John's wort) =\< 7 days prior to randomization. * Use of strong CYP3A4 inducers or inhibitors =\< 7 days prior to randomization.