Radiation Therapy and Fluorouracil With or Without Combination Chemotherapy Followed by Surgery in Treating Patients With Stage II or Stage III Rectal Cancer

Inclusion Criteria: * Patients must have an Eastern Cooperative Oncology Group (ECOG) Status of 0 or 1, or comparable Karnofsky performance status * Patients must have histologically confirmed invasive adenocarcinoma of the rectum Distal border of the tumor must be within 12 cm from the anal verge as measured on rigid proctoscopic exam * Patients must have Stage II (uT3-4, uN0) or Stage III (any T, uN1-2) tumors, as confirmed by ERUS or MRI; females with anterior tumors invading the posterior vaginal wall (uT4) and males with anterior tumors that invade the seminal vesicles or adjacent organs (uT4) will also be eligible provided they undergo an extended resection including the organs involved * Patients with high grade obstruction that impedes the ERUS exam are eligible for the study provided they can be staged by MRI * Patients with synchronous or metachronous colorectal cancer are eligible for the study on condition that they are treated for rectal cancer in accordance with the protocol * Patients with the following are NOT allowed on study: * Metastatic disease or other primaries * Locally recurrent rectal cancer * Previously documented history of Familial Adenomatous Polyposis * History of Inflammatory Bowel Disease * History of prior radiation treatments to pelvis * History of clinically significant cardiac disease (i.e., Class 3-4 congestive heart failure, symptomatic coronary artery disease, uncontrolled arrhythmia, and/or myocardial infarction within the previous 6 months * History of uncontrolled seizures or clinically significant central nervous system disorders * History of psychiatric conditions or diminished capacity that could compromise the giving of informed consent, or interfere with study compliance * History of allergy and/or hypersensitivity to 5-fluorouracil (fluorouracil), leucovorin (leucovorin calcium), and/or oxaliplatin * History of difficulty or inability to take or absorb oral medications * Patients must have adequate bone marrow, hepatic and renal function within 7 days prior to registration * White blood cells (WBC) \>= 3,000 mm\^3 * Absolute neutrophil count (ANC) \> 1,500 mm\^3 * Hemoglobin \> 9.5 mg/dl * Platelet count \>= 100,000 mm\^3 * Total bilirubin =\< 1.5 mg/dl * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) =\< 2.0 times institutional upper limit of normal (ULN) * Alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) =\< 2.0 times ULN * Alkaline phosphatase =\< 2.0 times ULN * Serum creatinine =\< 1.5 times ULN * Patients with hereditary non-polyposis colorectal cancer are eligible for the study provided they meet the rest of the eligibility criteria * Patients who have experienced a prior malignancy should have received potentially curative therapy for that malignancy, and should be cancer-free for at least five years from the date of initial diagnosis (Exceptions: patients treated for basal cell carcinoma, or carcinoma in-situ of the cervix) * Patients of reproductive potential should agree to use an effective method of birth control when undergoing treatments with known or possible mutagenic or teratogenic effects; all female participants of childbearing potential must have a negative urine or serum pregnancy test within two weeks prior to study registration * Patients or the patient's legally acceptable representative must provide written authorization to allow the use and disclosure of protected health information; NOTE: this may be obtained in either the study-specific informed consent or in a separate authorization form and must be obtained from the patient prior to study registration or the initiation of any study-specific procedures