Immune Response Activation for the Treatment of Unresectable Metastatic Colorectal Cancer or CEA Positive Metastatic Breast Cancer

Inclusion Criteria: * Patients should have a diagnosis of metastatic colon or rectal or breast cancer that is pathology proven * Patients should have a CEA producing colorectal cancer or breast cancer defined as a baseline CEA or prior documented CEA level exceeding 5 ng/ml or evidence of CEA staining by Immunohistochemistry (IHC) * Patients should 18 years of age or older * Patients are willing and capable to consent to study and to adhere with all elements of the study * Patients who have failed to respond to standard systemic therapy, or for whom standard or curative systemic therapy does not exist, is not tolerable or was refused * Patients should be at least 4 weeks from last receipt of a cytotoxic or biological agent prior to start of SBRT, with the exception of mitomycin C which requires a 6-week washout * Patients should have unresectable disease or not be a candidate for surgical resection * Patients must have a minimum of 1 and a maximum of 5 separate metastatic lesions planned for SBRT. (Patients may have \> 5 metastatic lesions overall, however only up to 5 lesions will be treated with SBRT.) SBRT sites must be equal to or less than 5 cm in greatest dimension. SBRT treated sites must be measurable per RECIST 1.1 and can include metastatic sites in the lung, liver, or soft tissue. Sites that are intracranial or in the bone are excluded. Sites deemed not appropriate for SBRT by the treating radiation oncologist are also excluded * Patients should be at least 4 weeks from last radiation therapy prior to starting SBRT * Patients should be at least 4 weeks from any investigational therapy prior to starting SBRT, with the exception of prior immunotherapy which would require a 3 month washout * Patients should have an Eastern Cooperative Oncology Group (ECOG) performance status of 0-1 * Patients should be considered clinically stable with an estimated overall survival of at least 3 months * Neutrophil count \> 1500/mm\^3 * Lymphocyte count \> 500/mm\^3 * Hemoglobin \> 9 gm/dl * Platelets count \> 100,000/mm\^3 * Aspartate transaminase (AST)/alanine transaminase (ALT) \< 2.5 x upper limit of normal (ULN) * Bilirubin ≤ ULN * Patients should have adequate kidney function defined as a serum creatinine \< ULN or calculated creatinine clearance of \> 60ml/min (Cockroft-Gault formula) * Patients should have adequate cardiac function defined as: * No history of acute coronary syndromes (including myocardial infarction, unstable angina, Coronary artery bypass grafting (CABG), coronary angioplasty, or stenting) \< 12 months prior to screening * No impaired cardiovascular function or clinically significant cardiovascular diseases, including any of the following: * Symptomatic chronic heart failure; * Evidence of clinically significant cardiac arrhythmias and/or conduction abnormalities * No uncontrolled arterial hypertension despite appropriate medical therapy (defined as systolic blood pressure \> 160 or diastolic blood pressure \>100) * Electrocardiogram (EKG) showing normal sinus rhythm and a corrected QT (QTc) ≤ 450 ms for male and ≤ 470 ms for female patients * Patients should have adequate pulmonary function defined as: * Lack of uncontrolled pleural effusion requiring recurrent draining procedures (more than once per month) * Lack of oxygen supplementation dependence * All subjects must have the ability to understand and the willingness to sign a written informed consent * Screening 2-dimensional (2-D) echocardiogram (echo) shows a left ventricular ejection fraction (LVEF) \> 40% * Urinalysis shows lack of proteinuria or a maximum of 1+ proteinuria * Women of childbearing potential should use highly effective contraception while receiving the trial regimen and for at least 5 half-lives of M5A-IL2 from the last dose of M5A-IL2 Exclusion Criteria: * Patients on immunosuppressive treatments including supra-physiological doses of corticosteroids * Patients with history of auto-immune disease including history of inflammatory bowel disease * Patients with active brain metastases * Patients in the child-bearing ages who refuse to use adequate birth control measures (example: contraceptives, barrier method, or abstinence) * Lactating females who do not agree to stop breastfeeding * Known active hepatitis B or C * Major surgical procedure within 4 weeks prior to SBRT * Non-healed wound or surgical incisions * Radiographic evidence of bowel obstruction * Electrolyte disturbances (sodium, potassium, magnesium, calcium, and phosphorous) that are not correctable to at least CTCAE grade 1 with replacement therapy * Known hypersensitivity of any of the study drug agents or components * Patients should not have any uncontrolled illness including ongoing or active infection * History of allergic reactions attributed to compounds of similar chemical or biologic composition to the study agents * Pregnant women are excluded from this study because the investigational agents on this study are highly likely to exert teratogenic or abortifacient effects * Patients with other active malignancies are ineligible for this study * Subjects, who in the opinion of the investigator, may not be able to comply with the safety monitoring requirements of the study