FOLFIRI and Panitumumab in Treating Patients With RAS and BRAF Wild-Type Metastatic Colorectal Cancer

Inclusion Criteria: * Participant must have the ability to understand and the willingness to sign a written informed consent document * Participant must be willing to comply with study and/or follow-up procedures * Eastern Cooperative Oncology Group (ECOG) performance status 0-2 * Life expectancy of 3 \>= months * Histologically confirmed colon or rectal cancer with metastatic disease * Extended RAS and BRAF wild type status documented on archival tumor tissue or on fresh biopsy if no archival tissue present * Measurable disease defined by at least 1 lesion \>= 1 cm * Documented objective response or stable disease lasting for 6 months or more to last prior anti-EGFR (cetuximab or panitumumab) in combination with irinotecan or FOLFIRI * Progression within 6 weeks following their last dose of anti-EGFR therapy * Treatment with a non-EGFR targeting regimen following progression on anti-EGFR plus irinotecan-based therapy * At least 4 months from prior anti-EGFR therapy prior to start of study treatment * At least three weeks from any non-anti-EGFR therapy prior to start of study treatment; any number of prior therapies is permitted * Adequate recovery in the investigators opinion from any clinically significant toxicity from prior therapy * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L * Hemoglobin (Hgb) \>= 9 g/dL without transfusions * Platelets (PLT) \>= 100 x 10\^9/L without transfusions * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\]) and/or alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x upper limit of normal (ULN); patient with liver metastases =\< 5 x ULN * Total bilirubin =\< ULN * Creatinine =\< 1.5 mg/dL * Magnesium \>= 1.2mg/dL or 0.5 mmol/L * Negative serum beta-human chorionic gonadotropin (HCG) test (female patient of childbearing potential only), to be performed locally within the screening period * Agreement by females of childbearing potential and sexually active males to use an effective method of contraception (hormonal or barrier method of birth control or abstinence) prior to study entry and for three months following duration of study participation; should a woman become pregnant or suspect that she is pregnant while participating on the trial, she should inform her treating physician immediately Exclusion Criteria: * History of severe anti-EGFR toxicity requiring drug discontinuation or dose-modification within the first 4 months of prior anti-EGFR therapy * History of intolerance to irinotecan at dose-intensity of 125 mg/m\^2/2 weeks or lower * History of intolerance to 5-FU at dose-intensity of 1800 mg/m\^2/2 weeks or lower * Current use (or planned use during the treatment period) of other investigational agents, or biological, chemotherapy, radiation or other anti-tumor therapy * Co-medication that may interfere with study results; e.g. immuno-suppressive agents other than corticosteroids, such as systemic cyclosporine and tacrolimus * No St John's wort supplement or other herbal supplementation is allowed while on trial; patients are not to take grapefruit juice during study treatment * Use of drugs known to inhibit UDP glycosyltransferase 1 family, polypeptide A1 gene (UGT1A1), such as Atazanavir, Gemfibrozil, Indinavir, or Ketoconazole while on study treatment; (patients using these drugs must not take these drugs on the day study treatment begins and for the duration of study treatment) * Planned use of strong cytochrome P450, family 3, subfamily A, polypeptide 4 (CYP3A4) inhibitors or CYP3A4 inducers while on study treatment unless deemed clinically necessary with no reasonable alternatives and with expressed permission from the principal investigator * If on anticoagulation, participant must be on stable therapeutic dose prior to enrollment * Impairment of gastrointestinal function or gastrointestinal disease (e.g., active ulcerative disease, uncontrolled nausea, vomiting, diarrhea, malabsorption syndrome, extensive small bowel resection) * Major surgery =\< 3 weeks prior to starting study drug or who have not recovered from side effects of such procedure * Unstable pulmonary embolism, deep vein thrombosis, or other significant arterial/venous thromboembolic event =\< 30 days before enrollment * Clinically significant cardiovascular disease (including myocardial infarction, unstable angina, symptomatic congestive heart failure, serious uncontrolled cardiac arrhythmia) =\< 6 months prior to enrollment * History of interstitial lung disease (ILD) eg, interstitial pneumonitis, pulmonary fibrosis or evidence of ILD on baseline chest computed tomography (CT) or magnetic resonance imaging (MRI) * Other active malignancies except cervical carcinomas in situ or clinically insignificant non-melanoma skin cancers * Clinically significant uncontrolled illness or active infections * History of allergic reactions attributed to compounds of similar chemical or biologic composition to irinotecan, 5-FU, leucovorin or any of the products to be administered during dosing * Pregnant women and women who are lactating; breastfeeding should be discontinued if the mother is enrolled on this study * Any other condition that would, in the Investigator's judgment, contraindicate the patient's participation in the clinical study due to safety concerns or compliance with clinical study procedures, e.g., infection/inflammation, intestinal obstruction, unable to swallow medication, social/psychological issues, etc * Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics)