TAS-102 and Radiation Therapy in Treating Patients With Rectal Cancer That Is Locally Recurrent, Metastatic, or Cannot Be Removed by Surgery

Inclusion Criteria: * Histological or cytological confirmation of locally recurrent or metastatic rectal adenocarcinoma * Note: Patients with locally recurrent/persistent disease within the pelvis after primary therapy (chemotherapy, surgery, and/or radiotherapy) are eligible * Note: Patients who have had prior pelvic radiotherapy with a total dose of =\< 54 Gy are eligible * Note: Patients with or without metastatic disease (excluding untreated central nervous system \[CNS\] metastasis), with primary pelvic disease or pelvic recurrence are eligible * Note: Patients with pelvic disease that is potentially resectable or unresectable are eligible * Measurable disease * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0, 1 or 2 * Expected life expectancy \>= 12 weeks * Obtained =\< 14 days prior to registration: Absolute neutrophil count (ANC) \>= 1500/mm\^3 * Obtained =\< 14 days prior to registration: Platelet count \>= 100,000/mm\^3 * Obtained =\< 14 days prior to registration: Hemoglobin \>= 9.0 g/dL * Obtained =\< 14 days prior to registration: Total bilirubin =\< 1.5 x upper limit of normal (ULN) (in patients with well-documented Gilbert's syndrome and the total bilirubin is grade 1, then direct bilirubin value must be =\< 1.0 mg/dL) * Obtained =\< 14 days prior to registration: Aspartate transaminase (AST) =\< 2 x ULN (=\< 5 x ULN for patients with liver involvement) * Obtained =\< 14 days prior to registration: Alanine aminotransferase (ALT) =\< 2 x ULN (=\< 5 x ULN for patients with liver involvement) * Obtained =\< 14 days prior to registration: Alkaline phosphatase =\< 3 x ULN * Obtained =\< 14 days prior to registration: Creatinine =\< 1.5 x ULN OR * Obtained =\< 14 days prior to registration: Calculated creatinine clearance must be \>= 45 ml/min using the Cockcroft-Gault formula * Negative pregnancy test done =\< 7 days prior to registration, for women of childbearing potential only; Note: patients must agree to adequate birth control during the study and for up to 6 months after discontinuation of study medication * Patients must be able to take medications orally (i.e. no feeding tube) * Provide written informed consent * Willing to return to enrolling institution for follow-up (during the Active Monitoring Phase of the study) * Able to complete questionnaires by themselves or with assistance Exclusion Criteria: * Primary resectable rectal cancer * Prior treatment with TAS-102 * Chemotherapy or immunotherapy =\< 28 days prior to registration * Radiation therapy =\< 28 days prior to registration; Note: patients with prior pelvic radiation therapy \> 54 Gy are ineligible * Failure to fully recover from acute, reversible effects of prior chemotherapy regardless of interval since last treatment * Other concurrent chemotherapy, immunotherapy, or any ancillary antitumor therapy considered investigational (utilized for a non-Food and Drug Administration \[FDA\]-approved indication and in the context of a research investigation) * Untreated CNS or leptomeningeal metastasis * Note: CNS or leptomeningeal disease must be stable for \>= 3 months prior to registration * History of seizure disorder * Co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the investigator, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Ascites, pleural effusion, or pericardial fluid requiring drainage in the last 4 weeks prior to registration * Intestinal obstruction, uncontrolled gastrointestinal hemorrhage, pulmonary fibrosis, renal failure, liver failure, or cerebrovascular disorder * Known human immunodeficiency virus (HIV) or acquired immunodeficiency syndrome (AIDS)-related illness, or hepatitis B or C * Patients with autoimmune disorders or history of organ transplantation who require immunosuppressive therapy * History of myocardial infarction =\< 12 months prior to registration, severe/unstable angina, systematic congestive heart failure (CHF) New York Heart Association classification III or IV or CHF requiring use of ongoing maintenance therapy for life-threatening ventricular arrhythmias * Major surgery =\< 4 weeks prior to registration (the surgical incision should be fully healed prior to drug administration or radiation therapy) * Any of the following: * Pregnant women * Nursing women * Men or women of childbearing potential who are unwilling to employ adequate contraception