Talimogene Laherparepvec, Chemotherapy, and Radiation Therapy Before Surgery in Treating Patients With Locally Advanced or Metastatic Rectal Cancer

Inclusion Criteria: * For dose escalation: Patients must have A) histologically or cytologically confirmed low lying (up to 6 cm of anal verge) rectal adenocarcinoma eligible for radiation therapy to rectal tumor, B) if the treatment is palliative in the metastatic setting, no additional requirements for tumor size or nodal involvement is needed; C) if the treatment is in the neoadjuvant setting, the tumor must ALSO be high-risk locally advanced rectal cancer defined as T3-4, N+, and/or at risk for a positive radial margin (as determined by the surgeon) * For dose expansion: Patients must have A) histologically or cytologically confirmed rectal adenocarcinoma eligible for radiation therapy to rectal tumor irrespective of location from anal verge, B) if the treatment is palliative in the metastatic setting, no additional requirements for tumor size or nodal involvement is needed; C) if the treatment is in the neoadjuvant setting, the tumor must ALSO be high-risk locally advanced rectal cancer defined as T3-4, N+, and/or at risk for a positive radial margin (as determined by the surgeon) * Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of talimogene laherparepvec in combination with chemotherapy in patients \< 18 years of age * Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%) * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9 /L * Hemoglobin \>= 9 g/dL * Platelets \>= 100,000 x 10\^9 /L * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (except patients with Gilbert's syndrome, who can have total bilirubin \< 3 mg/DL) * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x institutional ULN * Serum creatinine =\< 1.5 mg/dL OR calculated creatinine clearance (Cockcroft-Gault formula) \>= 50 mL/min OR 24-hour urine creatinine clearance \>= 50 mL/min * Prothrombin time (PT)/international normalized ratio (INR) and partial thromboplastin time (PTT) =\< 1.5 x institutional unless the subject is on anticoagulant therapy (if the subject is receiving anticoagulant therapy, PT, and activated partial thromboplastin time \[aPTT\] must be within therapeutic range of intended use of anticoagulants) * Patients must have signed informed consent indicating that they are aware of the investigational nature of the study, and are aware that participation is voluntary; patients must be made aware of their other treatment options * Talimogene laherparepvec, as well as other therapeutic agents used in this trial including radiation and capecitabine, may cause fetal harm when administered to a pregnant woman; women of childbearing potential (WOCBP) is defined as any female who has experienced menarche and who has not undergone surgical sterilization (hysterectomy or bilateral oophorectomy) or who is not postmenopausal; menopause is defined clinically as 12 months of amenorrhea in a woman over 45 in the absence of other biological or physiological causes; WOCBP and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, during the study participation, and for four months after the last dose of the drug; WOCBP must have a negative serum pregnancy test within 72 hours prior to enrollment and agree to use effective contraception throughout the treatment period and for 4 months after the last dose of study treatment; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately * Ability to understand and the willingness to sign a written informed consent document Exclusion Criteria: * Patients who have had radiotherapy within \< 4 weeks are ineligible * Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1) except alopecia are ineligible * Use of other investigational, chemotherapeutic or targeted drugs within 28 days (or five half-lives, whichever is shorter; with a minimum of 14 days from the last dose) preceding the first dose of talimogene laherparepvec and during the study are ineligible * Patients who have previously been treated with talimogene laherparepvec, any other oncolytic virus or pelvic radiation are ineligible * Patients with known active central nervous system (CNS) metastases are ineligible; subjects with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least four weeks prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases * Patients with a known immediate or delayed hypersensitivity reaction or idiosyncrasy to talimogene laherparepvec or any of its components, capecitabine, fluorouracil (5-FU) and / or oxaliplatin are ineligible * Patients with a history or evidence of active autoimmune disease (e.g., pneumonitis, glomerulonephritis, vasculitis, or other); or history of active autoimmune disease that has required systemic treatment (i.e., use of corticosteroids, immunosuppressive drugs or biological agents used for treatment of autoimmune diseases) within 2 months of enrollment are ineligible; (replacement therapy \[e.g., thyroxine for hypothyroidism, insulin for diabetes or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency\] is not considered a form of systemic treatment for autoimmune disease) * Patients with evidence of clinically significant immunosuppression such as the following are ineligible: * Primary immunodeficiency state such as severe combined immunodeficiency disease * Concurrent opportunistic infection * Receiving systemic immunosuppressive therapy (\> 2 weeks) including oral steroid doses \> 10 mg/day of prednisone or equivalent within 2 months prior to enrollment * Patients with active herpetic skin lesions or prior complications of herpetic infection (e.g., herpetic keratitis or encephalitis) are ineligible * Patients with viral infections requiring intermittent or chronic systemic (intravenous or oral) treatment with an antiherpetic drug (e.g., acyclovir), other than intermittent topical use are ineligible * Patients with other viral infections are ineligible: * Known to have acute or chronic active hepatitis B or hepatitis C infection * Known to have human immunodeficiency virus (HIV) infection * Prior therapy with viral-based tumor vaccine * Received live vaccine within 28 days prior to enrollment * Subject who is unwilling to minimize exposure with his/her blood or other body fluids to individuals who are at higher risks for herpes simplex virus (HSV)-1 induced complications such as immunosuppressed individuals, individuals known to have HIV infection, pregnant women, or children under the age of 1 year, during talimogene laherparepvec treatment and through 30 days after the last dose of talimogene laherparepvec are ineligible * Patients with uncontrolled intercurrent illness including, but not limited to, active, non-colorectal malignancies requiring systemic therapy, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements are ineligible * Although no effects on embryo-fetal development have been observed in animal studies, adequate and well-controlled studies with talimogene laherparepvec have not been conducted in pregnant women; in addition, given the high risk of detrimental effects of other study agents including capecitabine, 5-FU, oxaliplatin and radiation on embryo-fetal development, the study treatment must be excluded in the following patients: * Female subject is pregnant or breast-feeding, or planning to become pregnant during study treatment and through 4 months after the last dose of talimogene laherparepvec * Female subject of childbearing potential who is unwilling to use acceptable method(s) of effective contraception during study treatment and through 4 months after the last dose of talimogene laherparepvec * Sexually active subjects and their partners unwilling to use male or female latex condom to avoid potential viral transmission during sexual contact while on treatment and within 30 days after treatment with talimogene laherparepvec * Patients that are unable to swallow oral medications are ineligible * Patients with a major surgical procedure, open biopsy, or significant traumatic injury within 28 days prior to initiation of therapy, or anticipation of need for major surgical procedure during the course of the study other than that defined by protocol are ineligible * If patients require anticoagulation while on protocol, they should be switched from warfarin to another alternative anticoagulant such as low molecular weight heparin or oral direct factor Xa inhibitors as deemed appropriate by the treating physician for the duration they are on capecitabine (must be switched at least 1 week prior to starting capecitabine) to avoid drug-drug interaction of warfarin with capecitabine * Patients with a known dihydropyrimidine dehydrogenase (DPD) deficiency are ineligible * Patients who are unable to get MRIs due to any reason including pacemakers or automatic implantable cardioverter-defibrillator (AICD) are ineligible