A Phase I/II Study of Pexa-Vec Oncolytic Virus in Combination With Immune Checkpoint Inhibition in Refractory Colorectal Cancer

* INCLUSION CRITERIA: * Patients must have histopathological confirmation of Colorectal Carcinoma (CRC) by the Laboratory of Pathology of the National Cancer Institute (NCI) prior to entering this study. * Patients must have progressed on, been intolerant of or refused prior oxaliplatin- and irinotecan-containing, fluorouracil-based, chemotherapeutic regimen and have disease that is not amenable to potentially curative resection. Patients who have a known Kirsten rat sarcoma viral oncogene homolog (KRAS) wild type tumor must have progressed, been intolerant of or refused cetuximab or panitumumab-based chemotherapy. * Patient's tumors must be documented to be microsatellite-stable (MSS) either by genetic analysis or immunohistochemistry OR microsatellite-high with documented disease progression following anti-Programmed cell death protein 1 (PD1)/Programmed death-ligand 1 (PDL1) therapy. * Patients must have at least one focus of metastatic disease that is amenable to pre- and on-treatment biopsy and be willing to undergo this. Ideally the biopsied lesion should not be one of the target measurable lesions, although this can be up to the discretion of the investigators. * All patients enrolled will be required to have measurable disease by Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria. * Age greater than or equal to 18 years. Because no dosing or adverse event data are currently available on the use of Pexa-Vec in combination with tremelimumab and/or durvalumab in patients \<18 years of age, children are excluded from this study, but will be eligible for future pediatric trials. * Eastern Cooperative Oncology Group (ECOG) performance status 0-1 * Patients must have acceptable organ and marrow function as defined below: * Leukocytes greater than or equal to 3,000/mcL * absolute neutrophil count greater than or equal to 1,500/mcL * Platelets greater than or equal to 100,000/mcL * total bilirubin less than or equal to 1.5X institution upper limit of normal * Hemoglobin (Hb) \> 9g/dl * Aspartate aminotransferase (AST) serum glutamic-oxaloacetic transaminase (SGOT)/Alanine aminotransferase (ALT) serum glutamate-pyruvate transaminase (SGPT) less than or equal to 2.5 x institutional upper limit of normal unless liver metastases are present, in which case it must be less than or equal to 5x upper limit of normal (ULN) * Creatinine \<1.5X institution upper limit of normal, OR * creatinine clearance greater than or equal to 45 mL/min/1.73 m(2), as calculated below, for patients with creatinine levels above institutional normal * Patient must be able to understand and willing to sign a written informed consent document * The effects of Pexa-Vec, durvalumab and tremelimumab on the developing human fetus are unknown. For this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study participation and up to 180 days after the last dose of durvalumab + tremelimumab combination therapy or 90 days after the last dose of durvalumab monotherapy, whichever is the longer time period. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. * Evidence of post-menopausal status or negative urinary or serum pregnancy test for female pre-menopausal patients. Women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause. The following age-specific requirements apply: * Women \<50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy). * Women greater than or equal to 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \>1 year ago, had chemotherapy-induced menopause with last menses \>1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy. Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up. * Body weight \>35kg EXCLUSION CRITERIA: * Patients who have had anti-cancer therapy (chemotherapy, immunotherapy, endocrine therapy, targeted therapy, biologic therapy, tumor embolization, monoclonal antibodies or other investigation agents), large field radiotherapy, or major surgery must wait 4 weeks after completing treatment prior to entering the study. * No prior exposure to immune-mediated therapy including, but not limited to, other anti-cytotoxic T-lymphocyte-associated protein 4 (CTLA-4), anti-PD-1, anti-PD-L1, and anti-programmed cell death ligand 2 (anti-PD-L2) antibodies, including therapeutic anticancer vaccines. The exception to this is those whose tumors are MSI-hi and are refractory to anti-PD1 monotherapy. * Involvement in the planning and/or conduct of the study * Previous IP assignment in the present study * Patients who are receiving any other investigational agents. * Inability to suspend treatment with anti-hypertensive medication (including but not limited to: diuretics, beta-blockers, angiotensin converting enzyme \[ACE\] inhibitors, aldosterone antagonists, etc.) for 48 hours pre and post each Pexa-Vec administration. * Patients with severe hypertension who in the opinion of the investigator cannot withhold antihypertensive medication for 48 hours pre and post Pexa-Vec administration. * Any unresolved toxicity NCI Common Terminology Criteria for Adverse Events (CTCAE) Grade greater than or equal to 2 from previous anticancer therapy with the exception of alopecia, vitiligo, and the laboratory values defined in the inclusion criteria * Patients with Grade greater than or equal to 2 neuropathy will be evaluated on a case-by-case basis * Patients with known brain metastases will be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events. * Uncontrolled intercurrent illness including, but not limited to, hypertension (systolic blood pressure (BP) \> 160, diastolic BP \> 100), ongoing or active systemic infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, uncontrolled diabetes or psychiatric illness/social situations that would limit compliance with study requirements. For patients with a history of cardiovascular disease, cardiology consultation. Echocardiogram, troponin and creatinine clearance must be obtained prior to enrollment. NOTE: Patients with active cardiac disease (e.g. myocarditis and myocardial infarction) within 12 months of study entry are excluded from study participation. * Human immunodeficiency virus (HIV)-positive patients receiving anti-retroviral therapy are excluded from this study due to the possibility of pharmacokinetic interactions between antiretroviral medications and the investigational agent. HIV positive patients not receiving antiretroviral therapy are excluded due to the possibility that Durvalumab or Tremelimumab may worsen their condition and the likelihood that the underlying condition may obscure the attribution of adverse events. * Known significant immunodeficiency due to underlying illness (e.g. HIV/Acquired immunodeficiency syndrome (AIDS) and/or immune-suppressive medication including high-dose corticosteroids (defined as greater than or equal to 20 mg/day prednisone or equivalent which is ongoing at the time of enrollment and/or was taken for more than 4 weeks within the preceding 2 months of enrollment) * History of chronic autoimmune disease (e.g. systemic lupus erythematosus or Wegener's granulomatosis, Addison's disease, multiple sclerosis, Graves' disease, Hashimoto's thyroiditis, hypophysitis, etc.) with symptomatic disease within the 3 years before enrollment. Note: Active vitiligo or a history of vitiligo will not be a basis for exclusion. In addition, a past history of certain autoimmunity e.g. rheumatoid arthritis or thyroiditis may be allowed per principal investigator (PI) discretion provided it has been quiescent for a minimum of three years. The following are exceptions to this criterion: 1. Patients with vitiligo or alopecia 2. Patients with hypothyroidism (e.g. following Hashimoto syndrome) stable on hormone replacement 3. Any chronic skin condition that does not require systemic therapy 4. Patients without active disease in the last 5 years may be included 5. Patients with celiac disease controlled by diet alone 6. Active or history of inflammatory bowel disease (colitis, Crohn's), irritable bowel disease, celiac disease, or other serious, chronic, gastrointestinal conditions associated with diarrhea. * History of active primary immunodeficiency * Active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and tuberculosis (TB) testing (if clinically indicated), hepatitis B (known positive HBV surface antigen (HBsAg) result), hepatitis C. Patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible. Patients positive for hepatitis C (HCV) antibody are eligible only if polymerase chain reaction is negative for hepatitis C virus (HCV) ribonucleic acid (RNA). * Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab or tremelimumab. The following are exceptions to this criterion: * Intranasal, inhaled, topical steroids, or local steroid injections (e.g. intra articular injection) * Systemic corticosteroids at physiologic doses not to exceed \<\<10 mg/day\>\> of prednisone or its equivalent * Steroids as premedication for hypersensitivity reactions (e.g. computed tomography (CT) scan premedication) * Receipt of live attenuated vaccine within 30 days prior to the first dose of IP. Note: Patients, if enrolled, should not receive live vaccine whilst receiving IP and up to 30 days after the last dose of IP. * Female patients who are breastfeeding. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with Pexa-Vec, breastfeeding should be discontinued if the mother is treated with Pexa-Vec. These potential risks may also apply to other agents used in this study. * Known allergy or hypersensitivity to IP * Prior randomization or treatment in a previous durvalumab and/or tremelimumab clinical study regardless of treatment arm assignment. * History of sarcoidosis syndrome * Mean QT interval corrected for heart rate (QTc) greater than or equal to 470 ms calculated from 3 electrocardiograms (ECGs) using Fredericia's Correction * Patients with a history of Interstitial lung disease or pneumonitis * Subjects with uncontrolled seizures * History of leptomeningeal carcinomatosis * History of hypersensitivity reaction to human or mouse antibody products. * Patients with unhealed surgical wounds for more than 30 days * Ongoing severe inflammatory skin condition (as determined by the Investigator) requiring medical treatment * History of severe eczema (as determined by the Investigator) requiring medical treatment * Patients with tumor(s) invading a major vascular structure (e.g. carotid artery) or other key anatomical structure (e.g. pulmonary airway) in the event of post treatment tumor swelling and/or necrosis (hepatic and portal vein involvement allowed) * Patients with liver tumors in a location that would potentially result in significant clinical adverse effects in the opinion of investigator if post-treatment tumor swelling were to occur, including at the site of the common bile duct * Clinically significant and/or rapidly accumulating ascites, pericardial and/or pleural effusions. Mild ascites that does not preclude safe tumor biopsy as protocol specified is allowed at the discretion of the treating physician. * Medical conditions, per the investigator's judgment, that predispose the patient to untoward medical risk in the event of volume loading (e.g. intravenous \[IV\] fluid bolus infusion), tachycardia, or hypotension during or following treatment with Pexa-Vec * Any prior or planned organ transplant (e.g. liver transplant) * Patients who experienced a severe systemic reaction or side-effect as a result of a previous vaccination with vaccinia * Pulse oximetry oxygen (O2) saturation \<90% at rest on room air