Panobinostat and Fluorouracil Followed By Leucovorin Calcium in Treating Patients With Stage IV Colorectal Cancer Who Did Not Respond to Previous Fluorouracil-Based Chemotherapy

Inclusion Criteria: * Patients must have histologically or cytologically confirmed advanced/metastatic colorectal cancer * Must have measurable disease * Must have received prior therapy (in any setting) with 5-FU, CPT-11, and oxaliplatin; (may have received prior erbitux and bevacizumab, but it is not required) * Must have received at least one prior chemotherapy regimen for advanced disease * Tumor must be accessible for core biopsy at the beginning of treatment and patients have a high intratumoral TS expression level prior to the beginning of treatment * Life expectancy of \> 12 weeks * ECOG performance status 0-2 (Karnofsky \>= 50%) * Normal organ and marrow function as defined below: * Serum albumin \>= 3g/dL * AST/SGOT and ALT/SGPT =\< 2.5 x upper limit of normal(ULN)or =\< 5.0 x ULN if the transaminase elevation is due to liver metastasis * Serum bilirubin =\< 1.5 x ULN * Serum creatinine =\< 1.5 x ULN or 24-hr creatinine clearance \>= 50 ml/min * Serum potassium \>= LLN * Serum phosphorous \>= LLN * Serum total calcium (corrected for serum albumin) or serum ionized calcium \>= LLN * Serum magnesium \>= LLN * TSH and free T4 within normal limits(WNL)(patients may be on thyroid hormone replacement) * Leukocytes \>= 3,000/μL * Absolute neutrophil count \>= 1,500/μL * Platelets \>= 100,000/μL * Hemoglobin \>= 9 mg/dL * Eligibility of patients receiving any medications or substances known to affect or with the potential to affect the activity or pharmacokinetics of LBH589 will be determined following review by the Principal Investigator * Ability to understand and willing to sign a written informed consent document * INR is =\< 1.5 times ULN unless receiving therapeutic anticoagulation * Patient is highly unlikely to conceive as indicated by at least one "yes" answer to the following questions: 1. patient is a male and agrees to use an adequate method of contraception for the duration of the study, and for 30 days after the last dose of study medication; 2. surgically sterilized female; 3. postmenopausal female \>= 45 years of age with \> 2 years since last menses; 4. non-sterilized premenopausal female and agrees to use 2 adequate barrier methods of contraception to prevent pregnancy or agrees to abstain from heterosexual activity throughout the study and for 30 days after the last dose of study medication * Baseline MUGA must demonstrate LVEF \>= the lower limit of the institutional normal * Clinically euthyroid; Note: Patients are permitted to receive thyroid hormone supplements to treat underlying hypothyroidism Exclusion Criteria: * Male patients whose sexual partners are WOCBP not using effective birth control * May not have received any other investigational agents within 28 days of study entry (chemotherapy, any investigational drug or undergone major surgery \< 4 weeks prior to starting study drug or who have not recovered from side effects of such therapy). * May not receive other anti-cancer therapy (cytotoxic, biologic, radiation, or hormonal other than for replacement) while on this study * Patients with known brain metastases are excluded from this clinical trial * History of allergic reactions attributed to compounds of similar chemical or biologic composition to LBH589; including sodium butyrate, trichostatin A (TSA), trapoxin (TPX), MS-27-275 and depsipeptide * Severe and/or uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Pregnant or breast feeding or patients of reproductive potential not using two effective methods of birth control; women of childbearing potential (WOCBP) must have a negative serum pregnancy test within 7 days of the first administration of oral LBH589 * History of another primary malignancy within 5 years other than curatively treated CIS of the cervix, or basal or squamous cell carcinoma of the skin * Unresolved diarrhea \> CTCAE grade 1 * Acute infection requiring intravenous antibiotic, antiviral, or antifungal medications within 2 weeks prior to the start of study drugs * Known positivity for human immunodeficiency virus (HIV) or hepatitis C * Screening ECG with a QTc \> 450 msec * Patients with congenital long QT syndrome * History of sustained ventricular tachycardia * Any history of ventricular fibrillation or torsades de pointes * Bradycardia defined as heart rate \< 50 beats per minute; patients with a pacemaker and heart rate \>= 50 beats per minute are eligible * Myocardial infarction or unstable angina within 6 months of study entry * Congestive heart failure (NY Heart Association class III or IV) * Right bundle branch block and left anterior hemiblock (bifascicular block) * Prior cancer treatment with an HDAC inhibitor (e.g., vorinostat, Depsipeptide, MS-275, LAQ-824, PXD-101, and valproic acid) * Uncontrolled hypertension * Concomitant use of drugs with a risk of causing torsades de pointes * Any significant history of non-compliance to medical regimens or with inability to grant a reliable informed consent * Impairment of gastrointestinal (GI) function or GI disease that may significantly alter the absorption of oral panobinostat (e.g., ulcerative disease, uncontrolled nausea, vomiting, diarrhea, malabsorption syndrome, obstruction, or stomach and/or small bowel resection) * Patients needing valproic acid for any medical condition during the study or within 5 days prior to first LBH589