Safety and Feasibility of PD-1 Blockade in the Treatment of dMMR or MSI-H Rectal Cancer

Inclusion Criteria: * Must have confirmed rectal adenocarcinoma Defined as, MRI based clinical stage II (T3-4, N0), stage III (T1-4, N+), or oligometastatic locally advanced stage IV that are candidates for curative surgery * Tumor location at and/or below the peritoneal reflection on MRI. * Review and discussion at multidisciplinary tumor board with consensus recommendation for neoadjuvant chemoradiation followed by curative-intent surgery. Documented in EPIC tumor board. * MMR-deficiency confirmed on immunohistochemistry or MSI status confirmed by PCR. * ECOG Performance status 0-1 * Life expectancy of ≥ 6 months, in the opinion of and as documented by the treating physician. * Must have normal organ and marrow function as defined below: * Hemoglobin ≥ 8.0 g/dL * Leukocytes ≥ 3,000/k/uL * Absolute neutrophil count ≥ 1,500/k/uL * Platelet count ≥ 100,000/k/uL * Total bilirubin ≤ 1.3 x institutional upper limit of normal (ULN) * AST (SGOT) ≤ 2.5 x institutional upper limit of normal (ULN) * ALT (SGPT) ≤ 2.5 x institutional upper limit of normal (ULN) * Must have the ability to understand and the willingness to sign a written informed consent document. Exclusion Criteria: * Prior treatment for rectal cancer or prior radiation for other diagnoses to the expected rectal cancer treatment fields. * Participants receiving any other investigational agents. * Unresectable primary tumor or unresectable metastatic disease as determined by imaging. * History of allergic reactions attributed to compounds of similar chemical or biologic composition to Pembrolizumab or other agents used in this study. * Participants with uncontrolled intercurrent illness including, but not limited to: * Ongoing or active infection * Known history of pneumonitis * Symptomatic congestive heart failure * Unstable angina pectoris * Cardiac arrhythmia * Psychiatric illness/social situations that would limit compliance with study requirements. * Pregnant or lactating females. * Female participants who: * Are postmenopausal for at least 1 year before the screening visit, OR * Are surgically sterile, OR * Agree to practice true abstinence from heterosexual contact or agree to use effective contraception without interruption during the study therapy and 90 days after the last dose * Male participants who: Are surgically sterile, OR Agree to practice true abstinence from heterosexual contact or agree to use effective contraception without interruption during the study therapy and 90 days after the last dose * HIV-positive participants on combination antiretroviral therapy, participants with active Hepatitis B or C, active tuberculosis, or administration of live vaccine within 30 days of planned start of study therapy will be excluded. * Participants with a diagnosis of immunodeficiency, active autoimmune disease (including inflammatory bowel disease) or those receiving immunosuppressive therapy within 7 days (other than Prednisone ≤ 5mg daily) prior to the planned start of study treatment will be excluded.