PI Pembro in Combination With Stereotactic Body Radiotherapy for Liver Metastatic Colorectal Cancer

Inclusion Criteria: * Willing and able to provide written informed consent/assent for the trial * Be \>/= 18 years of age on day of signing consent. * Have a diagnosis of histologically confirmed metastatic colorectal cancer to the liver (no other sites of metastatic disease) \* Histologic confirmation of a colorectal primary tumor is acceptable if accompanied by radiographic evidence of metastatic disease * Tumor must be mismatch repair (MMR) proficient as determined by microsatellite instability or immunohistochemistry for for MMR proteins * Microsatellite instability testing must be MSI-stable or MSI-low * Or IHC for MMR proteins must demonstrate intact MMR proteins * Participant must be candidate for SBRT to at least one intrahepatic lesion. There is no limit on the number of intrahepatic lesions the patient may have * Participant must be a surgical candidate with therapeutic goal of eradicating all known disease with one additional surgery. Portal venous embolization is permitted to ensure resectability. * Prior resection of extra-hepatic metastatic disease allowed if completed more than 12 months previous to study enrollment and now new extra-hepatic disease has been found * Have measurable disease based on RECIST 1.1 * Fresh or archived colorectal cancer tissue, preferably from a hepatic metastatic site. Archival tissue is acceptable for enrolled into this study. Participants who have no archival tissue available do not need to undergo a new biopsy solely for the purpose of this study * Participants must have received at least one prior line or chemotherapy including an irinotecan or oxaliplatin-fluoropyrimidine-based systemic treatment for colorectal cancer * Have performance status of 0 or 1 on the ECOG Performance Scale * Demonstrate an adequate organ function as defined in Table 1. These labs should be repeated if not completed within 10 days of SBRT treatment initiation * Female participants of childbearing potential should have a negative urine or serum pregnancy test within 10 days of initiating SBRT. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Female participants of childbearing potential should be willing to use 2 methods of birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study through 120 days after the last dose of study medication (Reference Section 5.7.2). Participants of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \>1 year * Male participant should agree to use an adequate method of contraception starting with the first dose of study therapy through 120 days after the last dose of study therapy Exclusion Criteria: * Current participation and receiving study therapy or previous participation in a study of an investigational agent and received study therapy or used an investigational device within 4 weeks of the initiation of SBRT * Prior anti-cancer monoclonal antibody (mAb) within 4 weeks prior to study Day 1 (first day of SBRT treatment) or who has not recovered (i.e. \< Grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier * Prior chemotherapy, targeted small molecule therapy, or radiation therapy within 2 weeks prior to study Day 1 or who has not recovered (i.e. \< Grade 1 or at baseline) from adverse events due to a previously administered agent. Prior radiotherapy to the liver is not allowed * Participants with \< Grade 2 neuropathy are an exception to this criterion and may qualify for the study * If the participant received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy * Participant has a diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the initiation of SBRT * Participant has a known history of active TB (Bacillus Tuberculosis) * Hypersensitivity to pembrolizumab or any of its excipients * Participant has had a prior anti-cancer monoclonal antibody (mAb) within 4 weeks prior to study Day 1 (first day or SBRT treatment) or who has not recovered (i.e., ≤ Grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier * Participant has had prior chemotherapy, targeted small molecule therapy, or radiation therapy within 2 weeks prior to study Day 1 or who has not recovered (i.e., ≤ Grade 1 or at baseline) from adverse events due to a previously administered agent. Prior radiotherapy to the liver is not allowed. (Notes: Participants with ≤ Grade 2 neuropathy are an exception to this criterion and may qualify for the study. If participant received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy.) * Participant has a known additional malignancy that is progressing or requires active treatment. Exceptions include basal cell carcinoma of the skin or squamous cell carcinoma of the skin that has undergone potentially curative therapy or in situ cervical cancer. * Participant has known active central nervous system (CNS) metastases and/or carcinomatous meningitis. Participants with previously resected brain metastases may participate provided it has been at least 6 months and no CNS progression has been identified. * Participant has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (eg., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. * Participant has known history of, or any evidence of active, non-infectious pneumonitis. * Participant has an active infection requiring systemic therapy. * Participant has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the participant's participation for the full duration of the trial, or is not in the best interest of the participant to participate, in the opinion of the treating investigator. * Participant has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial. * Participant is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment. * Participant has received prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent. * Participant has a known history of Human Immunodeficiency Virus (HIV) (HIV 1/2 antibodies). * Participant has known active Hepatitis B (e.g., HBsAg reactive) or Hepatitis C (e.g., HCV RNA \[qualitative or quantitative\] is detected). * Participant has received a live vaccine within 30 days of planned start of study therapy.