INNATE: Immunotherapy During Neoadjuvant Therapy for Rectal Cancer

Inclusion Criteria: 1. At least 18 years of age. Both men and women and members of all races and ethnic groups will be included. 2. Willing and able to provide written informed consent 3. Pathologic diagnosis of rectal adenocarcinoma 4. Stage III or Stage II with at least 1 of the following high-risk features: * Distal (\<1cm from anal ring) * cT4 or within 3mm of MR fascia * Not candidate for sphincter preservation * Extramural venous invasion 5. No prior treatment for rectal adenocarcinoma 6. Eastern Cooperative Group (ECOG) performance status of 0-1. 7. Laboratory values supporting acceptable organ and marrow function within 21 days of eligibility confirmation. Defined as follows: * WBC ≥ 3,000/mL; * ANC WBC ≥ 1,500/mL; * PLT ≥ 100,000/mL; * T Bili ≤ 1.5 x upper limit of normal (ULN); * AST/ALT ≤ 2.5 x ULN; * Creatinine ≤ 1.5 times upper limit of normal or calculated creatinine clearance \> 45 mL/min per Cockcroft-Gault equation. 8. Female participants of childbearing potential (FOCBP) must have a negative serum or urine pregnancy test (per institutional standards) within 72 hours prior to the start of study drug. FOCBP must agree to use highly-effective method(s) of contraception (Appendix A) during the study and for 90 days after the last dose of study drugs. FOCBP are those who have not been surgically sterilized (hysterectomy, bilateral oophorectomy, or bilateral salpingectomy) or have not been free from menses for \>1 year without an alternative medical cause. 9. Male participants must agree to use an adequate method of contraception (Appendix A) starting with the first dose of study therapy through 90 days after the last dose of study drugs. Exclusion Criteria: 1. Distant nodal disease (retroperitoneal nodes) including inguinal nodes, or any metastatic disease by CT or PET 2. Prior RT to the pelvis. 3. Uncontrolled comorbid illness or condition including an active infection, congestive heart failure, unstable angina, cardiac arrhythmia, or psychiatric illness that would limit compliance with the study requirements. 4. Prior treatment with any anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CD137, or anti-CTLA-4 antibody, or any other antibody or drug specifically targeting T-cell co-stimulation or checkpoint pathways. 5. Any positive history for HIV/AIDS, HTLV, hepatitis B or hepatitis C virus indicating acute or chronic infection. 6. Any active known or suspected autoimmune disease. Participants with vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll. 7. Any condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalent) or other immunosuppressive medications within 14 days prior to the first dose of study drug. Inhaled steroids and adrenal replacement steroid doses up to 10 mg daily prednisone equivalent are permitted (although not encouraged) in the absence of active autoimmune disease. 8. Malignancy in the past 3 years that required active treatment except locally curable cancers or cancers deemed by the treating physicians to not impact the subject's survival duration. 9. Participants receiving any other investigational agent, standard antineoplastic agents, or immunosuppressive agents. 10. Known history of interstitial lung disease. 11. Received live vaccine within 6 weeks prior to randomization. 12. Psychiatric illness/social situations that would limit consenting and compliance with study requirements. 13. Participants who are pregnant or nursing due to the potential for congenital abnormalities and the potential of this regimen to harm nursing infants. 14. Patient is not a candidate for the full treatment regimen.