Phase II Trial of Combination Immunotherapy in Subjects With Advanced Small Bowel and Colorectal Cancers

-INCLUSION CRITERIA: 1. Subjects with cytologically or histologically confirmed locally advanced or metastatic small bowel or colorectal adenocarcinoma 2. Subjects must have received two prior lines of systemic therapy unless the subject is not eligible to receive standard therapy or declines standard treatment 3. Subjects must have measurable disease 4. Eastern Cooperative Oncology Group (ECOG) performance status \<= 2 5. Adequate hematologic function at screening, as follows: * Absolute neutrophil count (ANC) \>=1 x 10\^9/L * Hemoglobin \>= 9 g/dL * Platelets \>= 75,000/microliter 6. Adequate renal and hepatic function at screening, as follows: \- Serum creatinine \<= 1.5 x upper limit of normal (ULN) OR Measured or calculated creatinine clearance \>= 40 mL/min for participant with creatinine levels \> 1.5 X institutional ULN (glomerular filtration rate (GFR) can also be used in place of creatinine or creatinine clearance (CrCl); * Bilirubin \<= 1.5 x ULN OR in subjects with Gilbert's syndrome, a total bilirubin \<= 3.0 x ULN * Alanine aminotransferase (ALT) and aspartate aminotransferase (AST) \<= 2.5 x ULN, unless liver metastases are present, then values must be \<= 3 x ULN) 7. The effects of the immunotherapies on the developing human fetus are unknown; thus, women of childbearing potential and men must agree to use highly effective contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study treatment and up to two months after the last dose of study drug. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. 8. Participants serologically positive for human immunodeficiency virus (HIV), Hep B, Hep C are eligible as long as the viral loads are undetectable by quantitative polymerase chain reaction (PCR). HIV positive participants must have cluster of differentiation 4 (CD4) count \>= 200 cells per cubic millimeter at enrollment, be on stable antiretroviral therapy for at least 4 weeks and have no reported opportunistic infections or Castleman s disease within 12 months prior to enrollment. 9. Ability of subject to understand and the willingness to sign a written informed consent document. EXCLUSION CRITERIA: 1. Participants with prior investigational drug, chemotherapy, immunotherapy or any prior radiotherapy (except for palliative bone directed therapy) within the past 28 days prior to the first drug administration except if the investigator has assessed that all residual treatment-related toxicities have resolved or are minimal and feel the participant is otherwise suitable for enrollment. Additionally, current therapies (e.g., maintenance capecitabine) may be continued where in the opinion of the investigator stopping such therapies may increase the risk of disease progression. Also, participants may continue adjuvant hormonal therapy in the setting of a definitively treated cancer (e.g., breast). 2. Participants with microsatellite unstable or mismatch repair deficient disease. 3. Major surgery within 28 days prior to the first drug administration (minimally invasive procedures such as diagnostic biopsies are permitted). 4. Known life-threatening side effects resulting from prior checkpoint inhibitor therapy (e.g., colitis, pneumonitis, fulminant hepatitis which led to permanent discontinuation of prior checkpoint therapy). Autoimmune toxicity which was not life threatening (e.g., arthritis) or did not lead to discontinuation of prior checkpoint therapy is allowed. 5. Known active brain or central nervous system metastasis (less than a month out from definitive radiotherapy or surgery), seizures requiring anticonvulsant treatment (\<3 months) or clinically significant cerebrovascular accident (\<3 months). In order to be eligible participants must have repeat central nervous system (CNS) imaging at least a month after definitive treatment showing stable CNS disease. Participants with evidence of intra-tumoral or peritumoral hemorrhage on baseline imaging are also excluded unless the hemorrhage is grade \<= 1 and has been shown to be stable on two consecutive imaging scans. 6. Pregnant women are excluded from this study because these drugs have not been tested in pregnant women and there is potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with these immunotherapies, breastfeeding should be discontinued if the mother is treated on this protocol. 7. Active autoimmune disease that might deteriorate when receiving an immunostimulatory agent with exception of: * Diabetes type I, eczema, vitiligo, alopecia, psoriasis, hypo- or hyperthyroid disease or other mild autoimmune disorders not requiring immunosuppressive treatment; * Subjects requiring hormone replacement with corticosteroids are eligible if the steroids are administered only for the purpose of hormonal replacement and at doses \<= 10 mg of prednisone or equivalent per day; * Administration of steroids for other conditions through a route known to result in a minimal systemic exposure (topical, intranasal, intro-ocular, or inhalation) is acceptable; * Subjects on systemic intravenous or oral corticosteroid therapy with the exception of physiologic doses of corticosteroids (\<= the equivalent of prednisone 10 mg/day) or other immunosuppressive agents such as azathioprine or cyclosporin A are excluded on the basis of potential immune suppression. For these subjects these excluded treatments must be discontinued at least 1 weeks prior to enrollment for recent short course use (\<= 14 days) or discontinued at least 4 weeks prior to enrollment for long term use (\> 14 days). In addition, the use of corticosteroids as premedication for contrast-enhanced studies is allowed prior to enrollment and on study. 8. Subjects with a history of serious intercurrent chronic or acute illness, such as cardiac or pulmonary disease, hepatic disease, bleeding diathesis or recent (within 3 months) clinically significant bleeding events or other illness considered by the Investigator as high risk for investigational drug treatment. 9. Subjects unwilling to accept blood products as medically indicated. 10. History of second malignancy within 3 years of enrollment except for the following: adequately treated localized skin cancer, cervical carcinoma in situ, superficial bladder cancer, or other localized malignancy which has been adequately treated or malignancy which does not require active systemic treatment (e.g., low risk chronic lymphocytic leukemia (CLL). 11. Subjects with a known severe hypersensitivity reaction to a monoclonal antibody (grade \>= 3 National Cancer Institute (NCI)-Common Terminology Criteria for Adverse Events (CTCAE v5) will be evaluated by the allergy/immunology team prior to enrollment. 12. Receipt of any organ transplantation requiring ongoing immunosuppression. 13. Receipt of prior lymphodepleting chemotherapy (e.g., cyclophosphamide or fludarabine at standard lymphodepleting doses).