A Study of Radiation Therapy With Pembrolizumab and Olaparib or Other Radiosensitizers in Women Who Have Triple-Negative or Hormone-Receptor Positive/Her2 Negative Breast Cancer

Inclusion Criteria: * Female participants who are at least 18 years of age on the day of signing informed consent with histologically confirmed diagnosis of triple negative breast cancer or ER+/Her2 will be enrolled in this study. * Have an Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 1. Evaluation of ECOG is to be performed within 30 days prior to the date of allocation/randomization * Histologically or cytologically-confirmed TNBC (defined as ER \<5%, PR \<5%, HER- 2-neu 0-1+ by IHC or FISH-negative or per MD discretion). * Histologically of cytologically-confirmed ER+ breast cancer, defined as ER 1-100% and HER-2/neu 0-1+ by IHC or FISH-negative * Metastatic or recurrent TNBC. * Metastatic or recurrent ER+ breast cancer * For mTNBC patients, prior receipt of ICI with progression and/or PDL1-negative. PDL-1 status is not used to determine eligibility among mER+BC patients * Note: PDL1-status may be determined on tissues from either primary or mTNBC. Determination of PD-L1 status by any prototype assays is acceptable. PD-L1 status is required or archival tissue must be readily available for testing during screening if status was not previously determined. * The participant (or legally acceptable representative if applicable) provides written informed consent for the trial. * Have measurable disease based on RECIST 1.1. There should be at least one radiographically-confirmed non-bone metastatic lesion that will not undergo RT and is measurable based on RECIST and suitable for repeated measurements. * Lesions situated in a previously irradiated area are considered measurable if progression has been demonstrated in such lesions. * No more than 3 prior lines of systemic therapy (including conventional cytotoxics, targeted therapies, biologics, or other invesigational systemic treatments) for inoperable/recurrent or metastatic disease in the TNBC cohort. A line of treatment in this instance refers to any systemic therapy directed at metastatic TNBC and which was discontinued due to disease progression. Treatment discontinued due to toxicity will not be counted. Systemic therapy previously delivered for hormone-receptor positive or Her2+ breast cancer (that has now switched to TNBC subtype) will not count as a prior line of treatment. Any number of prior lines of treatment for mER+BC are allowed. * At least one tumor site for which palliative RT is considered clinically appropriate. The site under consideration can be a metastatic site, or uncontrolled primary/locally recurrent disease in the breast/chest wall or in the nodes. Prior radiotherapy to the target site is allowed. Investigators should remain within departmental radiotherapy for normal tissue; exceptions should be discussed with the PI. * Have provided archival tumor tissue sample or newly obtained core or excisional biopsy of a tumor lesion. Formalin- fixed, paraffin embedded (FFPE) tissue blocks are preferred to slides. Newly obtained biopsies are preferred to archived tissue. Newly-obtained is defined as a specimen obtained up to 30 days prior to initiation of treatment on Day 1 of pembro. (Note: PI can waive this requirement at his discretion if specimen collection is deemed unfeasible). * A female participant is eligible to participate if she is not pregnant (see Appendix C), not breastfeeding, and at least one of the following conditions applies: 1. Not a woman of childbearing potential (WOCBP) as defined in Appendix C OR 2. A WOCBP who agrees to follow the contraceptive guidance in Appendix C during the treatment period and for at least one month after the last dose of study treatment. * Have adequate organ function as defined in the following table (Table 2). Specimens must be collected within 30 days prior to the start of study treatment. Table 2 Adequate Organ Function Laboratory Values System: Laboratory Value Hematological Absolute neutrophil count (ANC): ≥ 1500/µL Platelets: ≥ 100 000/µL Hemoglobin: ≥ 9.0 g/dL with no blood transfusion in the past 28 days Renal Creatinine: 1.5 x ULN OR Measured or calculatedb creatinine clearance (GFR can also be used in place of creatinine or CrCl): ≥ 51 mL/min Hepatic Total bilirubin: ≤ 1.5 x ULN OR direct bilirubin ≤ ULN for participants with total bilirubin levels \>1.5 x ULN AST (SGOT) and ALT (SGPT): ≤ 2.5 x ULN (≤ 5 x ULN for participants with liver metastases) Coagulation International normalized ratio (INR) OR prothrombin time (PT); Activated partial thromboplastin time (aPTT): ≤ 1.5 x ULN unless participant is receiving anticoagulant therapy as long as PT or aPTT is within therapeutic range of intended use of anticoagulants ALT (SGPT)=alanine aminotransferase (serum glutamic pyruvic transaminase); AST (SGOT)=aspartate aminotransferase (serum glutamic oxaloacetic transaminase); GFR=glomerular filtration rate; ULN=upper limit of normal. a - Criteria must be met without erythropoietin dependency and without packed red blood cell (pRBC) transfusion within last 2 weeks. b - Creatinine clearance (CrCl) should be calculated per institutional standard. Note: This table includes eligibility-defining laboratory value requirements for treatment; laboratory value requirements should be adapted according to local regulations and guidelines for the administration of specific chemotherapies. * Participant must agree not to breastfeed during the study or for 180 days after the last dose of study treatment. * Participant receiving corticosteroids may continue as long as their dose is stable for at least 4 weeks prior to initiating protocol therapy. * Ability to swallow (whole) and retain oral medications. Exclusion Criteria: * Receipt of \> 3 lines of systemic therapy in mTNBC patients * A WOCBP who has a positive urine pregnancy test within 72 hours or a serum pregnancy test within 14 days prior to treatment (see Appendix C). If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required. * A minimum 2-week washout required for all anti-cancer agents, including cytotoxic chemotherapeutic agents, immunotherapy, biologic therapy, and targeted therapies. A 3-4 week washout is preferred when reasonable. * Note: in the event that 72 hours have elapsed between the screening pregnancy test and the first dose of study treatment, another pregnancy test (urine or serum) must be performed and must be negative in order for subject to start receiving study medication. * Has received prior systemic anti-cancer therapy including investigational agents within 4 weeks (preferred) or a minimum of 2 weeks prior to the start of study treatment. * Note: Participants must have recovered from all AEs due to previous therapies to ≤ Grade 1 or baseline. Participants with ≤ Grade 2 neuropathy may be eligible. * Note: If participant received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting study treatment. * Has received prior radiotherapy within 2 weeks of start of study treatment. Participants must have recovered from all radiation-related toxicities, not require corticosteroids, and not have had radiation pneumonitis. A 1-week washout is permitted for palliative radiation (≤ 2 weeks of radiotherapy) to non-CNS disease. * Has previously experienced grade 3 or higher immune-mediated adverse events from prior courses of immunotherapy. * Has received a live vaccine within 30 days prior to the first dose of study drug. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster (chicken pox), yellow fever, rabies, Bacillus Calmette-Guérin (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (eg, FluMist®) are live attenuated vaccines and are not allowed. ° Note: Seasonal influenza vaccines for injection are generally inactivated flu vaccines and are allowed; however intranasal influenza vaccines (e.g., Flu-Mist®) are live attenuated vaccines, and are not allowed. * Is currently participating in or has participated in a study of an investigational agent or has used an investigational device within 4 (preferred, or 2 weeks minimum) weeks prior to the first dose of study treatment. * Note: Participants who have entered the follow-up phase of an investigational study may participate as long as it has been at least 2 weeks (prefer 4 weeks) after the last dose of the previous investigational agent. * Has a diagnosis of immunodeficiency or is receiving chronic systemic steroid therapy (in dosing exceeding 10 mg daily of prednisone equivalent) or any other form of immunosuppressive therapy within 7 days prior to the first dose of study drug. * Has a known additional malignancy that is progressing or has required active treatment within the past 2 years. Note: Participants with basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or carcinoma in situ (e.g. breast carcinoma, cervical cancer in situ) that have undergone potentially curative therapy are not excluded. * Has severe hypersensitivity (≥ Grade 3) to pembrolizumab and/or olaparib and/or any of their excipients. * Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (eg., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. * Has a history of (non-infectious) pneumonitis that required steroids or has current pneumonitis. * Has an active infection requiring systemic therapy. * Has a known history of Human Immunodeficiency Virus (HIV). * Note: No HIV testing is required. * Has a known history of Hepatitis B (defined as Hepatitis B surface antigen \[HBsAg\] reactive) or known active Hepatitis C virus (defined as HCV RNA is detected) infection. Note: no testing for Hepatitis B and Hepatitis C is required unless mandated by local health authority. * Has a known history of active TB (Bacillus Tuberculosis). * Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the study, interfere with the subject's participation for the full duration of the study, or is not in the best interest of the subject to participate, in the opinion of the treating investigator. * Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial. * Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the study, starting with the screening visit through 120 days after the last dose of trial treatment * Has had an allogenic tissue/solid organ transplant * Resting ECG indicating uncontrolled, potentially reversible cardiac conditions, as judged by the investigator (eg., unstable ischemia, uncontrolled symptomatic arrhythmia, congestive heart failure, QTcF prolongation \> 500 ms, electrolyte disturbances, etc.), or patients with congenital long QT syndrome. * Persistent toxicities (\>Common Terminology Criteria for Adverse Event (CTCAE 5.0) grade 2) caused by previous cancer therapy, excluding alopecia. * Patients with myelodysplastic syndrome/acute myeloid leukemia or with features suggestive of MDS/AML. * Patients with symptomatic uncontrolled brain metastases. A scan to confirm the absence of brain metastases is not required. The patient can receive a stable dose of corticosteroids before and during the study as long as these were started at least 4 weeks prior to treatment. Patients with spinal cord compression unless considered to have received definitive treatment for this and evidence of clinically stable disease for 28 days. * Patients considered a poor medical risk due to a serious, uncontrolled medical disorder, nonmalignant systemic disease or active, uncontrolled infection. Examples include, but are not limited to, uncontrolled ventricular arrhythmia, recent (within 3 months) myocardial infarction, uncontrolled major seizure disorder, unstable spinal cord compression, superior vena cava syndrome, extensive interstitial bilateral lung disease on High Resolution Computed Tomography (HRCT) scan or any psychiatric disorder that prohibits obtaining informed consent. * Patients unable to swallow orally administered medication and patients with gastrointestinal disorders likely to interfere with absorption of the study medication. * Concomitant use of known strong CYP3A inhibitors (eg. itraconazole, telithromycin, clarithromycin, protease inhibitors boosted with ritonavir or cobicistat, indinavir, saquinavir, nelfinavir, boceprevir, telaprevir) or moderate CYP3A inhibitors (eg. ciprofloxacin, erythromycin, diltiazem, fluconazole, verapamil). The required washout period prior to starting olaparib is 2 weeks. * Concomitant use of known strong (eg. phenobarbital, enzalutamide, phenytoin, rifampicin, rifabutin, rifapentine, carbamazepine, nevirapine and St John's Wort ) or moderate CYP3A inducers (eg. bosentan, efavirenz, modafinil). The required washout period prior to starting olaparib is 5 weeks for enzalutamide or phenobarbital and 3 weeks for other agents. * Major surgery within 2 weeks of starting study treatment and patients must have recovered from any effects of any major surgery. * Previous allogenic bone marrow transplant or double umbilical cord blood transplantation (dUCBT). * Judgment by the investigator that the patient should not participate in the study if the patient is unlikely to comply with study procedures, restrictions and requirements. * Previous enrollment in the present study.