Liposomal Irinotecan, Fluorouracil, Leucovorin Calcium, and Rucaparib in Treating Patients With Metastatic Pancreatic, Colorectal, Gastroesophageal, or Biliary Cancer

Inclusion Criteria: * Phase I only: Histologic confirmation of pancreatic, colorectal, gastroesophageal or biliary adenocarcinoma, as follows: * Patients with metastatic disease from pancreatic cancer who received no more than 2 lines of prior therapy in the metastatic setting * Patients with metastatic disease from colorectal cancer who received no more than 3 lines of prior therapy in the metastatic setting * Patients with metastatic disease from gastroesophageal cancer who received no more than 1 line of prior therapy in the metastatic setting * Patients with metastatic disease from biliary tract cancer who received no more than 1 line of prior therapy in the metastatic setting * NOTE: No prior exposure to irinotecan in the metastatic setting will be allowed except in the phase I dose escalation portion and in colon cancer patients only; in pancreas cancer, exposure to irinotecan is only allowed in the neoadjuvant setting and no progressive disease \< 3 months from last dose of irinotecan * Phase Ib only: Patients with metastatic adenocarcinoma of the pancreas who have who received no more than 1 line of prior therapy in the metastatic setting * NOTE: Exposure to irinotecan is only allowed in the neoadjuvant setting and no progressive disease \< 3 months from last dose of irinotecan * Phase II only: Patients with metastatic adenocarcinoma of the pancreas with genomic markers (signature) of homologous recombination deficiency (HRD) or BRCA1 or BRCA2 or PALB2 mutation, or HRD (non-BRCA, non-PALB) who have not received any systemic therapy in the metastatic setting * NOTE: Exposure to irinotecan is only allowed in the neoadjuvant setting and no progressive disease \< 3 months from last dose of irinotecan * Measurable disease * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 * Absolute neutrophil count (ANC) \>= 1500/mm\^3 (obtained =\< 21 days prior to registration) * Platelet count \>= 100,000/mm\^3 (obtained =\< 21 days prior to registration) * Hemoglobin \> 9.0 g/dL (obtained =\< 21 days prior to registration) * Total bilirubin =\< institutional upper limit of normal (ULN) (obtained =\< 21 days prior to registration) * Aspartate transaminase (AST) =\< 3 x ULN, =\< 5.0 x ULN for patients with metastatic disease to the liver (obtained =\< 21 days prior to registration) * Aminotransferase (ALT) =\< 3.0 x ULN, =\< 5.0 x ULN for patients with metastatic disease to the liver (obtained =\< 21 days prior to registration) * Creatinine =\< 1.0 mg/dL or creatinine clearance \>= 45 ml/min using the Cockcroft-Gault formula (obtained =\< 21 days prior to registration) * Negative serum or urine pregnancy test done =\< 7 days prior to registration and repeated prior to dosing on day 1 of each cycle, for individuals of childbearing potential only; NOTE: Individuals are considered to be of childbearing potential unless one of the following applies: * Is postmenopausal, defined as no menses for at least 12 months without an alternative medical cause; a high follicle-stimulating hormone (FSH) level consistently in the postmenopausal range (30 mIU/mL or higher) may be used to confirm a postmenopausal state in women not using hormonal contraception or hormonal replacement therapy; however, in the absence of 12 months of amenorrhea, a single FSH measurement is insufficient to confirm a postmenopausal state: or * Considered to be permanently sterile; permanent sterilization includes hysterectomy, bilateral salpingectomy, and/or bilateral oophorectomy * Provide informed written consent * Willing to return to enrolling institution for follow-up (during the Active Monitoring Phase of the study) * Note: During the active monitoring phase of a study (i.e., active treatment and observation), participants must be willing to return to the consenting institution for follow-up * Willing to provide tissue and blood samples for mandatory correlative research purposes * Individuals of reproductive potential and their partners willing to practice total abstinence or use a highly effective method of contraception (failure rate \< 1% per year) during treatment and for 6 months following the last dose of rucaparib; the following are allowable only: * Ongoing use of progesterone-only injectable or implantable contraceptives (eg, Depo Provera, Implanon, Nexplanon) * Placement of an intrauterine device or intrauterine system * Bilateral tubal occlusion * Sterilization, with appropriate post-vasectomy documentation of absence of sperm in ejaculate * True, complete (as opposed to periodic) abstinence * Patients must discontinue prior chemotherapy \>= 28 days before registration Exclusion Criteria: * Any of the following because this study involves an investigational agent whose genotoxic, mutagenic and teratogenic effects on the developing fetus and newborn are unknown: * Pregnant individuals * Nursing individuals * Persons of childbearing potential who are unwilling to employ adequate contraception * Co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the investigator, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens * Immunocompromised patients and patients known to be human immunodeficiency virus (HIV) positive and currently receiving antiretroviral therapy * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Receiving any other investigational agent which would be considered as a treatment for the primary neoplasm * Previous or concurrent cancer that is distinct in primary site or histology from cancer of primary site =\< 3 years prior to registration EXCEPT for curatively treated cervical cancer in situ, melanoma in situ, non-melanoma skin cancer and superficial bladder tumors \[Ta (non-invasive tumor), Tis (carcinoma in situ) and T1 (tumor invades lamina propria)\]; Note: All cancer treatments for those distinct in a primary site other than cancer of origin must be completed \>= 3 years prior to registration * Received any prior poly ADP-ribose polymerase inhibitor (PARPi) treatment. Patients who received prior PARPi treatment in the adjuvant setting with the last dose received more than 12 months prior to registration are allowed to enroll. * Corrected QT interval (QTc) prolongation \> 480 msec, as calculated by either the Bazett or Fridericia formula, as per institutional standard * Inability to swallow