Testing Nivolumab and Ipilimumab With Short-Course Radiation in Locally Advanced Rectal Cancer

Inclusion Criteria: * Patient must be \>= 18 years of age * Patient must have histologically confirmed adenocarcinoma of the rectum with the inferior margin within 15 cm from the anal verge based on colonoscopy and/or flexible sigmoidoscopy * Patient must have T3-4Nx or TxN+ disease (stage II or III) based on magnetic resonance imaging of the pelvis and computed tomography of the chest and abdomen. These baseline scans must be done within 28 days prior to registration * Patient must have MSI-H (microsatellite instability-high) or dMMR (deficient mismatch repair) tumors based on immunohistochemistry or PCR (polymerase chain reaction) * Patient must have Eastern Cooperative Oncology Group (ECOG) Performance status 0-2 * Patient must not have previously received chemotherapy or immunotherapy for rectal cancer * Patient must not have previously received radiotherapy to the pelvis * Patient must have the ability to understand and the willingness to sign a written informed consent document. Patients with impaired decision-making capacity (IDMC) who have a legally authorized representative (LAR) or caregiver and/or family member available will also be considered eligible * Patient must not have had major surgery performed within 28 days prior to registration * Patient must not have a history of interstitial lung disease (e.g., pneumonitis or pulmonary fibrosis) or evidence of interstitial lung disease on baseline chest CT scan * Patient must not have a serious active infection requiring IV antibiotics at time of registration * Patient must agree to not receive live vaccines while on this study * Patient must not have active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including chronic prolonged systemic corticosteroids (defined as corticosteroid use of duration one month or greater). These include but are not limited to patients with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome, myasthenia gravis; systemic autoimmune disease such as systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, and patients with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or anti-phospholipid syndrome. Patients with any of these are ineligible for this study because of the risk of recurrence or exacerbation of disease * Patient must not have a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days prior to registration. Inhaled or topical steroids and adrenal replacement doses \< 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease. Patients are permitted to use topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption). Physiologic replacement doses of systemic corticosteroids are permitted, even if \< 10 mg/day prednisone equivalents. A brief course of corticosteroids for prophylaxis (e.g., contrast dye allergy) or for treatment of non-autoimmune conditions (e.g., delayed-type hypersensitivity reaction caused by contact allergen) is permitted * Patient must not be pregnant or breast-feeding due to the potential harm to an unborn fetus and possible risk for adverse events in nursing infants with the treatment regimens being used. All patients of childbearing potential must have a blood test or urine study within 2 weeks prior to registration to rule out pregnancy. A repeat pregnancy test must be done within 72 hours prior to first dose of treatment if the baseline test was done outside the 72 hour window. A patient of childbearing potential is defined as anyone, regardless of sexual orientation or whether they have undergone tubal ligation, who meets the following criteria: 1) has achieved menarche at some point, 2) has not undergone a hysterectomy or bilateral oophorectomy; or 3) has not been naturally postmenopausal (amenorrhea following cancer therapy does not rule out childbearing potential) for at least 24 consecutive months (i.e., has had menses at any time in the preceding 24 consecutive months) * Patients of childbearing potential and sexually active patients must not expect to conceive or father children by using accepted and effective method(s) of contraception or by abstaining from sexual intercourse for at least one month (female patients) or one week (male patients) prior to the start of study drug and continue for 5 months after the last dose of study drug (for female patients). Investigators must counsel patients on the importance of pregnancy prevention and the implications of an unexpected pregnancy * Leukocytes \>= 3,000/mcL (must be obtained =\< 14 days prior to protocol registration) * Absolute neutrophil count (ANC) \>= 1,500/mcL (must be obtained =\< 14 days prior to protocol registration) * Platelets \>= 100,000/mcL (must be obtained =\< 14 days prior to protocol registration) * Total bilirubin =\< institutional upper limit of normal (ULN) (must be obtained =\< 14 days prior to protocol registration) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) =\< 2.5 x institutional ULN (must be obtained =\< 14 days prior to protocol registration) * Creatinine =\< 1.5 x institutional ULN (must be obtained =\< 14 days prior to protocol registration) * Patients should have urine dipstick with proteinuria \< 1. If urine dipstick \> 2, proteinuria must be less than 1 g in 24 hours urine collection * Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months of registration are eligible for this trial * For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load * Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial * Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should have a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, patients should be class 2B or better * Patient must not have had live vaccines within 30 days prior to registration. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, chicken pox, yellow fever, rabies, Bacillus Calmette-Guerin (BCG), and typhoid (oral) vaccine. Patients are permitted to receive inactivated vaccines and any non-live vaccines including those for the seasonal influenza and COVID-19 (Note: intranasal influenza vaccines, such as Flu-Mist \[registered trademark\] are live attenuated vaccines and are not allowed). If possible, it is recommended to separate study drug administration from vaccine administration by about a week (primarily, in order to minimize an overlap of adverse events)