Pemigatinib for the Treatment of Metastatic or Unresectable Colorectal Cancer Harboring FGFR Alterations

Inclusion Criteria: * Registered to Colorectal and Liquid Biopsy Molecularly Assigned Therapy (COLOMATE) Academic and Community Cancer Research United (ACCRU)-GI-1611 and: * COLOMATE Companion Trial Recommendation Form indicates patient qualifies to be screened for a COLOMATE companion trial * COLOMATE Companion Trial Recommendation Form date of completion is =\< 30 days prior to registration * Histologically or cytologically confirmed diagnosis of metastatic or unresectable colorectal cancer (mCRC), based on documentation from local or outside review of pathology according to each site?s established institutional procedure * Documentation of an activating genomic alteration(s) in FGFR1-3 (gain of function mutations, translocations, and amplifications allowed) * Provide informed written consent * Patient must have received and progressed on, or be intolerant to, each of the following treatments for mCRC (or have contraindication to these treatments): * Fluoropyrimidine * Oxaliplatin * Irinotecan * Anti-VEGF (vascular endothelial growth factor) monoclonal antibody, if eligible for this therapy * Anti-EGFR (epidermal growth factor receptor) monoclonal antibody, if eligible for this therapy * Measurable disease * Eastern Cooperative Oncology Group (ECOG) performance status (PS) of 0, 1, or 2 * Absolute neutrophil count (ANC) \>= 1500/mm\^3 (obtained =\< 28 days prior to registration) * Platelet count \>= 100,000/mm\^3 (obtained =\< 28 days prior to registration) * Hemoglobin \>= 9.0 g/dL (obtained =\< 28 days prior to registration) * Total bilirubin =\< 1.5x upper limit of normal (ULN), or =\< 2.5x ULN if patient has Gilbert syndrome or disease involving the liver (obtained =\< 28 days prior to registration) * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5x ULN (or =\< 5x ULN in presence of suspected liver metastases) (obtained =\< 28 days prior to registration) * Serum phosphate \< institutional ULN (obtained =\< 28 days prior to registration) * Serum calcium within institutional normal range, or serum albumin-corrected calcium within institutional normal range (if serum albumin is outside of the institutional normal range) (obtained =\< 28 days prior to registration) * Potassium levels \> institutional lower limit of normal (supplementation can be used to correct potassium level during screening) (obtained =\< 28 days prior to registration) * Serum creatinine =\< 1.5x ULN, or calculated creatinine clearance \> 30 mL/min using the Cockcroft-Gault formula or 24-hours urine collection analysis (obtained =\< 28 days prior to registration) * Corrected QT interval (QTc) by Fridericia?s method (QTcF) assessed by electrocardiogram (ECG) completed =\< 28 days prior to registration, and resulted as: * QTcF =\< 450 msec in men, or * QTcF =\< 470 msec in women * Negative serum pregnancy test completed =\< 7 days prior to registration, for women of childbearing potential only * Willing to provide tissue and blood samples for correlative research purposes * Willing to allow transfer of tissue and blood samples, clinical information, and outcome data collected from this trial for future research Exclusion Criteria: * Prior treatment with pemigatinib * Prior treatment with a selective FGFR inhibitor =\< 180 days (6 months) prior to registration * Known hypersensitivity or severe reaction to an FGFR inhibitor, or to the excipients of pemigatinib (i.e. microcrystalline cellulose, sodium starch glycolate, and magnesium stearate) * Current evidence of clinically significant corneal or retinal disorder confirmed by ophthalmologic examination * Treatment with other investigational study drug for any indication for any reason, or receipt of anticancer medications =\< 14 days prior to registration * Major surgery =\< 28 days prior to registration * External beam radiation therapy =\< 28 days prior to registration, or palliative radiation for non-central nervous system (CNS) disease =\< 14 days prior to registration * Brain metastases, central nervous system (CNS) metastases, leptomeningeal disease, or spinal cord compression * NOTE: Patients who are asymptomatic or previously treated and stable, without evidence of progression for \>= 28 days prior to registration are eligible * NOTE: Patients taking concomitant corticosteroids and/or anticonvulsants are allowed if patient is on a stable or decreasing dose of such treatment for \>= 28 days prior to registration * History or presence of significant cardiovascular disease or condition including: * Uncontrolled angina pectoris (Canadian Cardiovascular Society grade II-IV despite medical therapy) * Congestive heart failure (New York Heart Association class III or IV) * Uncontrolled arrhythmia requiring therapy. Note: Patients with a pacemaker and well-controlled rhythm for \>= 28 days prior to registration are not excluded * Any of the following occurring =\< 6 months prior to registration: myocardial infarction, angioplasty, cardiac stenting, coronary/peripheral artery bypass graft, cerebrovascular accident or transient ischemic attack * Failure to adequately recover (i.e. to =\< grade 1 \[according to Common Terminology Criteria for Adverse Events (CTCAE) version (v.)5\] or to pre-treatment baseline) from adverse events (AEs) deemed by the investigator as clinically significant and attributed to prior therapy. Exception: alopecia * Current use of prohibited medication * Use of any potent CYP3A4 inhibitors or inducers or moderate CYP3A4 inducers =\< 14 days or 5 half-lives (whichever is shorter) prior to registration. Note: topical ketoconazole will be allowed * History of hypovitaminosis D requiring supraphysiologic doses to replenish the deficiency. Note: patients receiving vitamin D food supplements are allowed * History and/or current evidence of ectopic mineralization/calcification, including but not limited to soft tissue, kidneys, intestine, myocardia, or lung; with the exception of calcified lymph nodes and asymptomatic arterial or cartilage/tendon calcification * Unable or unwilling to swallow pemigatinib and keep a medication diary, or significant gastrointestinal disorder(s) that could interfere with absorption, metabolism or excretion of pemigatinib per the discretion of the investigator * Any of the following because this study involves an investigational agent whose genotoxic, mutagenic and teratogenic effects on the developing fetus and newborn are unknown: * Pregnant women * Nursing women * Women of childbearing potential or men able to father children who have a female partner of childbearing potential, who are unwilling to employ acceptable contraception * Known history of human immunodeficiency (HIV) infection or positivity on immunoassay confirmed per local standards * Note: HIV test is not required for screening, but patients with a known history of HIV infection will be excluded * Evidence of active hepatitis B virus (HBV) or hepatitis C virus (HCV) infection * Other known active malignancy =\< 5 years prior to registration * EXCEPTIONS: Non-melanotic skin cancer or carcinoma in situ of the cervix, provided there is no known active disease and no additional therapy for the condition is ongoing or required during the trial period * NOTE: anti-estrogen/androgen therapy or bisphosphonates allowed * Co-morbid systemic illness, other severe concurrent disease, or psychiatric illness/social situation which, in the judgment of the investigator, would make the patient inappropriate for entry into this study, limit compliance with study requirements, or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimen