Nivolumab and Ipilimumab and Radiation Therapy in MSS and MSI High Colorectal and Pancreatic Cancer

Inclusion Criteria: * Participants must have histologically or cytologically confirmed adenocarcinoma of colorectal or pancreatic origin * Age \>18 years. * Eastern Cooperative Oncology Group (ECOG) performance status ≤1 * Life expectancy of greater than 3 months * Participants must have normal organ and marrow function as defined in Table 1, all screening labs should be performed within 14 days of protocol registration. Table 1 Adequate Organ Function Laboratory Values System Laboratory Value * Hematological * Absolute neutrophil count (ANC) ≥1500 /mcL * White blood count (WBC) ≥2000 /mcL * Platelets ≥100,000 / mcL * Hemoglobin ≥9 g/dL * Renal * Serum creatinine OR Measured or calculated creatinine clearance (GFR can also be used in place of creatinine or CrCl) ≤ Serum creatinine ≤ 1.5 x ULN or creatinine clearance (CrCl) ≥ 40 mL/min (if using the Cockcroft-Gault formula below): * Female CrCl = (140 - age in years) x weight in kg x 0.85 72 x serum creatinine in mg/dL * Male CrCl = (140 - age in years) x weight in kg x 1.00 72 x serum creatinine in mg/dL * Hepatic * Serum total bilirubin ≤ 1.5 X ULN (upper limit of normal) (subjects with Gilbert Syndrome can have a total bilirubin \<3 mg/dL * aspartate aminotransferase (AST) serum glutamic oxaloacetic transaminase (SGOT) and Alanine Aminotransferase ALT (SGPT) ≤ 3 X ULN OR ≤ 5 X ULN for subjects with liver metastases * Coagulation * International Normalized Ratio (INR) or Prothrombin Time (PT) * Activated Partial Thromboplastin Time (aPTT) ≤1.5 X ULN unless subject is receiving anticoagulant therapy * as long as PT or PTT is within therapeutic range of intended use of anticoagulants ≤1.5 X ULN unless subject is receiving anticoagulant therapy * as long as PT or PTT is within therapeutic range of intended use of anticoagulants * Creatinine clearance should be calculated per institutional standard. * Women of childbearing potential (WOCBP) must use appropriate method(s) of contraception. WOCBP should use an adequate method to avoid pregnancy for 5 months (30 days plus the time required for nivolumab to undergo five half-lives) after the last dose of investigational drug. * Women of childbearing potential must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of HCG) * Women must not be breastfeeding * Men who are sexually active with WOCBP must use any contraceptive method with a failure rate of less than 1% per year. Men receiving nivolumab and who are sexually active with WOCBP will be instructed to adhere to contraception for a period of 7 months after the last dose of investigational product. Women who are not of childbearing potential, ie, who are postmenopausal or surgically sterile as well as azoospermic men do not require contraception * Ability to understand and the willingness to sign a written informed consent document. * Stable dose of dexamethasone 2 mg or less for 7 days prior to initiation of treatment * One previously unirradiated lesion amenable to radiotherapy 8 Gy x 3 and can meet dose constraints, and another unirradiated measurable lesion \> 1 cm in size outside the radiation field that can be used as measurable disease * Colorectal patients must have documentation of microsatellite status. Immunohistochemistry (IHC) is acceptable. * Colorectal patients must have received prior Fluorouracil (5FU), Irinotecan and Oxaliplatin (any combination) or have a contraindication to receiving these agents. * Pancreas patients must have progressed on at least 1 prior line of chemotherapy Exclusion Criteria: * Participants who have had chemotherapy, targeted small molecule therapy or study therapy within 14 days of protocol treatment, or those who have not recovered (i.e., ≤ Grade 1 or at baseline) from adverse events due to agents administered more than 2 weeks earlier. Subjects with ≤ Grade 2 neuropathy are an exception to this criterion and may qualify for the study. If subject received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy. * Participants who are receiving any other investigational agents. * Patients are excluded if they have an active, known or suspected autoimmune disease other than those listed below. Subjects are permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger * Patients are excluded if they have a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration. Inhaled or topical steroids and adrenal replacement doses \> 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease. Subjects are permitted to use topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption). Physiologic replacement doses of systemic corticosteroids are permitted, even if \> 10 mg/day prednisone equivalents. A brief course of corticosteroids for prophylaxis (eg, contrast dye allergy) or for treatment of non-autoimmune conditions (eg, delayed-type hypersensitivity reaction caused by contact allergen) is permitted. * Colorectal patients are excluded if they have had prior systemic treatment with an anti-CTLA4, anti-PD1 (Programmed cell death protein 1) or PDL1 (Programmed death-ligand 1) antibody. Pancreatic patients are excluded if they have previously received anti-CTLA-4 therapy. Prior PD-1 or PDL1 therapy will be permitted for pancreas patients * Has a known history of active TB (Bacillus Tuberculosis) * Patients are excluded if they are positive test for hepatitis B virus surface antigen (HBV sAg) or hepatitis C virus ribonucleic acid (HCV antibody) indicating acute or chronic infection * Patients are excluded if they have known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS). These participants are at increased risk of lethal infections when treated with marrow-suppressive therapy. Appropriate studies will be undertaken in participants receiving combination antiretroviral therapy when indicated. * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements. * Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial. * Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 5 months for woman and 7 months for men, after the last dose of trial treatment. * Has a known additional malignancy that is progressing or requires active treatment. Exceptions include basal cell carcinoma of the skin and squamous cell carcinoma of the skin that has undergone potentially curative therapy or in situ cervical cancer. * Has known history of, or any evidence of active, non-infectious pneumonitis. * Has an active infection requiring systemic therapy. * Has received a live vaccine within 30 days of planned start of study therapy. Note: Seasonal influenza vaccines for injection are generally inactivated flu vaccines and are allowed; however intranasal influenza vaccines (e.g., Flu-Mist®) are live attenuated vaccines, and are not allowed. * History of allergy to study drug components * History of severe hypersensitivity reaction to any monoclonal antibody * Uncontrolled brain metastases. Patients treated with radiation \> 4 weeks prior with follow up imaging showing control are eligible