Breast Cancer Study of Preoperative Pembrolizumab + Radiation

Inclusion Criteria: * Confirmed histologic diagnosis of invasive adenocarcinoma of the breast, and * ER, PR and HER2 testing (on outside or Cedars-Sinai biopsy report), and * High-risk, ER-positive and HER2-negative breast cancer patients. ER-positive disease is defined as ER\>10%, any PR and HER2-negative by ASCO CAP guidelines. High-risk disease will be defined by the presence of at least 2 of the following 3 criteria: histologic grade II-III, Ki-67 \> 20%, ER expression \< 75% by IHC) * TNBC patients (defined as ER\<10%, PR\<10%, HER2-neu 0-1+ by IHC or FISH-negative; or as per MD discretion) * Operable tumor measuring ≥2 cm in maximal diameter as measured by any available standard of care imaging (mammogram, breast ultrasound, breast MRI) * Any nodal status * Multifocal disease is permitted; largest focus must measure ≥2 cm * Synchronous bilateral invasive breast cancer is permitted * No indication of distant metastases * Breast-conserving therapy is planned * Tumor amenable to preoperative radiation therapy boost as determined by radiation oncologist * ECOG performance status score of 0 or 1 * Screening laboratory values must meet the following criteria: i. White blood cells (WBCs) ≥ 2000/μL ii. Absolute neutrophil count (ANC) ≥ 1500/μL iii. Platelets ≥ 100 x 103/μL iv. Hemoglobin ≥ 11.0 g/dL v. Serum creatinine ≤ 2 mg/dL (or glomerular filtration rate ≥ 40 ml/min) vi. AST ≤ 2.5 x upper limit of normal (ULN) vii. ALT ≤ 2.5 x ULN viii. Total bilirubin within normal limits (except subjects with Gilbert's syndrome, who must have total bilirubin \< 3.0 mg/dL) ix. INR ≤ 1.5 x ULN unless participant is receiving anticoagulant therapy as long as PT or aPTT is within therapeutic range of intended use of anticoagulant(s) x. Negative HIV screening test xi. Negative screening tests for Hepatitis B and Hepatitis C. Patients with positive results that do not indicate true active or chronic infection may enroll after discussion and consensus agreement by the treating physician and principal investigator. * Women of childbearing potential (WOCBP) must be using an acceptable method of contraception to avoid pregnancy throughout the study and for at least 4 months after the last dose of pembrolizumab in such a manner that the risk of pregnancy is minimized. * WOCBP must have a negative serum pregnancy test within 14 days prior to the first dose of pembrolizumab. * Women must not be breastfeeding. * Willingness to adhere to the study visit schedule and the prohibitions and restrictions specified in this protocol. * Willingness to undergo mandatory Week 4 research biopsy * Written informed consent obtained from subject and ability for subject to comply with the requirements of the study. Exclusion Criteria: * HER2-positive breast cancer defined as IHC3+ or IHC2+ with FISH\>2 AND copy number \>4 OR FISH \<2 AND copy number \>6 * Inflammatory breast cancer * Contraindication(s) to breast-conserving therapy * Contraindication to radiation therapy or planned partial breast irradiation * Patients with cosmetic breast augmentations, specifically sub glandular implants with altered breast tissue, at the time of diagnosis * Evidence of metastatic disease. * Diagnosis of immunodeficiency or is receiving chronic systemic steroid therapy (in dosing exceeding 10 mg daily of prednisone equivalent) or any other form of immunosuppressive therapy within 7 days prior to the first dose of study drug. * Known additional malignancy that is progressing or has required active treatment within the past 3 years. Note: Participants with basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or carcinoma in situ (e.g. breast carcinoma, cervical cancer in situ) that have undergone potentially curative therapy are not excluded. * Medical history and concurrent diseases * Active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (eg., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. * History of (non-infectious) pneumonitis that required steroids or has current pneumonitis. * Active infection requiring systemic therapy. * Any serious or uncontrolled medical disorder that, in the opinion of the investigator, may increase the risk associated with study participation or study drug administration, impair the ability of the subject to receive protocol therapy, or interfere with the interpretation of study results. * Known history of Human Immunodeficiency Virus (HIV) (HIV 1/2 antibodies). * Known history of Hepatitis B (e.g., HBsAg reactive) or known active Hepatitis C (e.g., HCV RNA \[qualitative\] is detected). * Prohibited Treatments and/or Therapies * Chronic use of immunosuppressants and/or systemic corticosteroids (used in the management of cancer or non-cancer-related illnesses). However, use of corticosteroids is allowed for the treatment of immune related Adverse Events (irAEs), or adrenal insufficiency. * Live vaccines within 30 days prior to the first dose of study treatment and while participating in the study. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster, yellow fever, rabies, BCG, and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (eg, FluMist®) are live attenuated vaccines and are not allowed. * Prior treatment with an anti-PD-1, anti-PD-L1, or anti PD L2 agent or with an agent directed to another stimulatory or co-inhibitory T-cell receptor (eg, CTLA-4, OX 40, CD137) * Prior systemic anti-cancer therapy including investigational agents within 4 weeks prior to study start. Note: Participants must have recovered from all AEs due to previous therapies to ≤Grade 1 or baseline. Participants with ≤Grade 2 neuropathy may be eligible. Note: If participant received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting study treatment. * Prior radiotherapy within 2 weeks of start of study treatment to sites outside the breast. Participants must have recovered from all radiation-related toxicities, not require corticosteroids, and not have had radiation pneumonitis. * Currently participating in or has participated in a study of an investigational agent or has used an investigational device within 4 weeks prior to the first dose of study treatment. Note: Participants who have entered the follow-up phase of an investigational study may participate as long as it has been 4 weeks after the last dose of the previous investigational agent. * For subjects who agree to the research breast MRI sub-study: Four or more previous gadolinium contrast scans due to the risk of brain deposits following repeated use of gadolinium-based contrast agents.