Pan FGFR Kinase Inhibitor BGJ398 and Combination Chemotherapy in Treating Patients With Untreated Metastatic Pancreatic Cancer

Inclusion Criteria: * Phase Ib: Histologically or cytologically confirmed adenocarcinoma of the pancreas, colon or rectum which disease is advanced (defined as not surgically curable) or metastatic in whom combination treatment using fluorouracil, oxaliplatin and irinotecan is a rational option * All patients must consent to provide archival tumor samples; non-availability of evaluable tumor samples does not exclude patient from the study * Phase II: Patients with histologically or cytologically confirmed metastatic pancreatic ductal adenocarcinoma that have not received systemic chemotherapy for advanced or metastatic disease; the definitive diagnosis of metastatic pancreatic adenocarcinoma will be made by integrating the histopathological data within the context of the clinical and radiographic data * Phase II: The initial 20 patients must have evaluable baseline tumor samples; evaluable samples are defined as those obtained by core biopsy or surgical resection and amendable to histological analysis; samples obtained by fine needle aspiration biopsy are not considered evaluable; patients who do not have evaluable archival tumor samples must consent to a tumor core biopsy prior to starting study treatment and, patients who consent to the baseline tumor biopsy will be eligible to receive study treatment irrespective of whether the samples obtained are evaluable * Have an Eastern Cooperative Oncology Group (ECOG) performance status of =\< 1 * Phase Ib: Have evaluable disease per Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria * Phase II: Have at least 1 metastatic lesion other than the primary pancreatic tumor that is evaluable per RECIST 1.1 criteria; lesions previously irradiated are not considered evaluable * Absolute neutrophil count (ANC) \>= 1,500 cells/mm\^3 (International System of Units \[SI\] units 1.5 x 10\^9/L) * Platelets \>= 100,000 cells/mm\^3 (SI units 100 x 10\^9/L) * Hemoglobin \>= 9.0 g/dL (SI units 90 g/L) * Total bilirubin =\< 1.5 x upper limit of normal (ULN) * Aspartate aminotransferase (AST)/serum glutamic oxaloacetic transaminase (SGOT) and alanine aminotransferase (ALT)/serum glutamate pyruvate transaminase (SGPT) =\< 3 x ULN; in patients with known hepatic involvement, AST and ALT =\< 5 x ULN are allowed * Serum creatinine =\< 1.5 x ULN or estimated creatinine clearance \>= 60 mL/min by the Cockcroft-Gault equation * International normalized ratio (INR) =\< 1.5 * Inorganic phosphorus =\< ULN * Ionized calcium =\< ULN * Ability to swallow and retain oral medication * Participant or legal representative must understand the investigational nature of this study and sign an Independent Ethics Committee/Institutional Review Board approved written informed consent form prior to receiving any study related procedure Exclusion Criteria: * Phase II: Patient who recurs with metastatic disease =\< 6 months of completing adjuvant chemotherapy after curative-intent surgical resection is not eligible; patients who recur with metastatic disease \> 6 months after completion of adjuvant chemotherapy are eligible * Phase II: Patients that previously received systemic chemotherapy for metastatic or advanced pancreatic adenocarcinoma are not eligible * Phase II: Diagnosis of any second malignancy within the last 3 years except for adequately treated basal cell carcinoma, or squamous cell skin carcinoma or in-situ cervical carcinoma * Patients with clinically significant ascites requiring paracentesis on 2 or more occasions within 4 weeks prior to start of study treatment * Known hypersensitivity to BGJ398, fluorouracil, oxaliplatin, irinotecan or to any of the excipients * Homozygous UDP glucuronosyltransferase 1 family, polypeptide A1 (UGT1A1)\*28 (i.e. 7 TA repeats) gene alleles; the UGT1A1 test should be conducted per local institutional practice * Palliative radiation treatment (e.g., pain control, bony lesions at risk of fracture) completed =\< 2 weeks of starting study treatment; patient will be eligible if palliative radiotherapy is completed \> 2 weeks from the start of study treatment and has recovered from radiotherapy toxicities * Pulmonary embolus or thrombosis of the deep venous system (deep vein thrombosis \[DVT\]) within 2 weeks of starting study treatment; patients who had a history of thromboembolic disease should be stable on therapeutic anticoagulation using low molecular weight (LMW) heparin for at least 2 weeks prior to the start of study treatment; use of warfarin (or derivatives) is not allowed at the start of study treatment * Patients with known symptomatic brain metastases requiring steroids and/or antiepileptic therapy; patients with previously diagnosed brain metastases are eligible if they have completed their treatment and have recovered from the acute effects of radiation therapy or surgery prior to the start of study treatment, and have discontinued corticosteroid treatment for these metastases for at least 4 weeks and are neurologically stable; patients with leptomeningeal involvement are excluded * Current evidence of corneal or retinal disorder/keratopathy including but not limited to bullous/band keratopathy, corneal abrasion, inflammation/ulceration, keratoconjunctivitis, confirmed by ophthalmologic examination * Prior FGFR inhibitor therapy * History or current evidence of: * Tissue calcification including, but not limited to, the soft tissue, kidneys, intestines, myocardium and lung with the exception of calcified lymph nodes and asymptomatic coronary calcification * Endocrine alterations of calcium/phosphate homeostasis, e.g., parathyroid disorders, history of parathyroidectomy, tumor lysis, tumoral calcinosis etc * Patients who are currently receiving or consuming * Medications that are known strong inducers or inhibitors of cytochrome P450, family 3, subfamily A, polypeptide 4 (CYP3A4) =\< 14 days of starting study treatment • Medications with a known risk of prolonging the QT interval or inducing Torsades de Pointes =\< 7 days prior to start of study treatment * Amiodarone =\< 180 days prior to start of study treatment * Grapefruit, grapefruit juice, pomegranates, star fruits, Seville oranges or products =\< 7 days prior to start of study treatment * Warfarin (or derivatives) * Uncontrolled intercurrent illness including, but not limited to, * Ongoing or active infection * Psychiatric illness/social situations that would limit compliance with study requirements * Clinically significant cardiac disease including any of the following: * Congestive heart failure requiring treatment (New York Heart Association grade \>= 2), left ventricular ejection fraction (LVEF) \< 50% as determined by multi gated acquisition scan (MUGA) scan or echocardiogram * Uncontrolled hypertension (defined as systolic blood pressure \[BP\] \>= 160 mmHg OR diastolic BP \>= 100 mmHg despite maximal anti-hypertensive medications: refer to World Health Organization \[WHO\]-International Society of Hypertension \[ISH\] guidelines) * History or presence of clinically significant ventricular arrhythmias, atrial fibrillation, resting bradycardia, or conduction abnormality * Unstable angina pectoris or acute myocardial infarction \< 3 months prior to starting study treatment * Corrected QT using the Fredericia's formula (QTcF) \> 450 msec (males); \> 470 msec (females) * History of congenital long QT syndrome * Known (historical) positive human immunodeficiency virus (HIV) serology, active hepatitis B, or active hepatitis C infection * Major surgery (e.g. intra-thoracic, intra-abdominal or intra-pelvic), open biopsy or significant traumatic injury =\< 4 weeks and minor procedures (including percutaneous biopsies, placement of vascular access device, laparoscopy +/- biopsy) =\< 1 week prior to starting study treatment; patients who have minor procedure(s) \> 1 week prior to starting study treatment and have recovered from side effects of such procedure are eligible * Unwilling or unable to follow protocol requirements * Any condition which in the Investigator's opinion deems the participant an unsuitable candidate to receive study drug * Received an investigational agent within 30 days prior to enrollment * Pregnant or nursing women; a negative pregnancy test (serum or urine) =\< 3 days prior to starting study treatment * Women of child-bearing potential (defined as all women physiologically capable of becoming pregnant) unless they are using highly effective methods of contraception during study treatment and for 3 months following the discontinuation of study treatment; highly effective contraception methods include: * Total abstinence (when this is in line with the preferred and usual lifestyle of the subject); periodic abstinence (e.g., calendar, ovulation, symptothermal, post-ovulation methods) and withdrawal are not acceptable methods of contraception * Female sterilization (have had surgical bilateral oophorectomy with or without hysterectomy) or tubal ligation at least six weeks before taking study treatment; in case of oophorectomy alone, only when the reproductive status of the woman has been confirmed by follow up hormone level assessment * Male sterilization (at least 6 months prior to screening); for female patients on the study the vasectomized male partner should be the sole partner for that patient * Combination of the following (a+b or a+c, or b+c): * Use of oral, injected or implanted hormonal methods of contraception or other forms of hormonal contraception that have comparable efficacy (failure rate \< 1%), for example hormone vaginal ring or transdermal hormone contraception * Placement of an intrauterine device (IUD) or intrauterine system (IUS). * Barrier methods of contraception: condom or occlusive cap (diaphragm or cervical/vault caps) with spermicidal foam/gel/film/cream/vaginal suppository; oral contraceptives (OC), injected or implanted hormonal methods are not allowed as the sole method of contraception * Post-menopausal women are allowed to participate in this study; women are considered post-menopausal and not of child bearing potential if they have had 12 months of natural (spontaneous) amenorrhea with an appropriate clinical profile (e.g. age appropriate, history of vasomotor symptoms) or have had surgical bilateral oophorectomy (with or without hysterectomy) or tubal ligation at least six weeks ago; in the case of oophorectomy alone, only when the reproductive status of the woman has been confirmed by follow up hormone level assessment is she considered not of child bearing potential * Sexually active males unless they use a condom during intercourse during and for 3 months after the last dose of the study treatment and should not father a child in this period; a condom is required to be used also by vasectomized men in order to prevent delivery of the drug via seminal fluid