Pembrolizumab in Treating Patients With Hormone Receptor Positive, Localized Inflammatory Breast Cancer Who Are Receiving Hormone Therapy and Did Not Achieve a Pathological Complete Response to Chemotherapy

Inclusion Criteria: 1. Is willing and able to provide written informed consent for the trial. 2. Is a female or male and \>/= 18 years of age 3. Has histological confirmation of breast carcinoma. 4. Has confirmed inflammatory breast cancer by using international consensus criteria: * Onset: Rapid onset of breast erythema, edema and/or peau d'orange, and/or warm breast, with/without an underlying breast mass * Duration: History of such findings no more than 6 months * Extent: Erythema occupying at least 1/3 of whole breast * Pathology: Pathologic confirmation of invasive carcinoma 5. Did not achieve pathological complete response (pCR) to any chemotherapy that was given with the intention to induce best response prior surgery. pCR is defined as the current American Joint Committee on Cancer (AJCC) breast cancer staging. 6. Is HER2 normal, defined as HER2 0 or 1+ by IHC and negative by FISH if performed; or HER2 is 2+ by IHC and negative by FISH; or HER2 negative by FISH if IHC is not performed. 7. Has positive ER or PR status. ER or PR \>/= 10% 8. Has a performance status of 0-1 on the ECOG Performance Scale. 9. Has adequate organ function as determined by the following laboratory values: ANC \>/= 1,500 /mcL, Platelets \>/=100,000 /mcL, Hgb \>/= 9 g/dL, creatinine levels \< 1.5 x ULN, Total bilirubin \</= 1.5 x ULN, ALT and AST \</= 2.5 x ULN 10. Participants of reproductive potential must agree to avoid becoming pregnant or impregnating a partner, respectively, while receiving study drug and for 120 days after the last dose of study drug by complying with one of the following: (1) practice abstinence† from heterosexual activity; OR (2) use (or have their partner use) acceptable contraception during heterosexual activity. Acceptable methods of contraception are: Single method (one of the following is acceptable): (1) intrauterine device (IUD); (2) vasectomy of a female participant's male partner; (3) contraceptive rod implanted into the skin. Combination method (requires use of two of the following): (1) diaphragm with spermicide (cannot be used in conjunction with cervical cap/spermicide); (2) cervical cap with spermicide (nulliparous women only); (3) contraceptive sponge (nulliparous women only); (4) male condom or female condom (cannot be used together); (5) hormonal contraceptive: oral contraceptive pill (estrogen/progestin pill or progestin-only pill), contraceptive skin patch, vaginal contraceptive ring, or subcutaneous contraceptive injection. Female participants will be considered of non-reproductive potential if they are either: (1) postmenopausal (defined as at least 12 months with no menses without an alternative medical cause; in women \< 45 years of age a high follicle stimulating hormone (FSH) level in the postmenopausal range may be used to confirm a post-menopausal state in women not using hormonal contraception or hormonal replacement therapy. In the absence of 12 months of amenorrhea, a single FSH measurement is insufficient.); OR (2) have had a hysterectomy and/or bilateral oophorectomy, bilateral salpingectomy or bilateral tubal ligation/occlusion, at least 6 weeks prior to screening; OR (3) has a congenital or acquired condition that prevents childbearing. Male participants will be considered to be of non-reproductive potential if they have azoospermia (whether due to having had a vasectomy or due to an underlying medical condition). 11\. Has negative serum or urine pregnancy test for subjects of childbearing potential within 10 days before first dose. 12\. Have completed radiation (if candidate for post-mastectomy radiation) or plans to begin radiation and endocrine therapy within 28 days. 13\. If the participant has already started hormonal blockade therapy after radiation as adjuvant therapy, the patient is eligible as long as the hormonal therapy was initiated no more than 6 months before the screening day and the participant can start the study drug within 4 weeks since the completion of screening. Exclusion Criteria: 1. Is currently participating in a study of an investigational anti-cancer agent. 2. Has a diagnosis of immunodeficiency or any other form of immunosuppressive therapy. 3. Has not recovered from adverse events due to prior therapies, i.e. monoclonal antibody, chemotherapy, targeted small molecule therapy, radiation therapy, or surgery. \- Note: Participants with ≤ Grade 2 neuropathy, alopecia and general disorders and administration site conditions (per CTCAE version 4.0) are an exception to this criterion and may qualify for the study. 4. Has a known history of prior malignancy with the exception of basal cell carcinoma of the skin, superficial bladder cancer, squamous cell carcinoma of the skin, or in situ cervical cancer, and has undergone potentially curative therapy and has no evidence of recurrence over the last 1 year since completion of curative therapy. 5. Has an active autoimmune disease requiring systemic treatment within the past 3 months or a documented history of clinically severe autoimmune disease, or immunosuppressive agents. Participants with vitiligo or resolved childhood asthma/atopy would be an exception to this rule. Participants that require intermittent use of bronchodilators, inhaled steroid or local steroid injections to the skin would not be excluded from the study. Participants with hypothyroidism stable on hormone replacement or Sjögren's syndrome will not be excluded from the study. 6. Has a history of (non-infectious) pneumonitis that required steroids or current pneumonitis. 7. Has an active infection requiring systemic therapy. 8. Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial. 9. Has received prior therapy with an anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CD137, or anti-Cytotoxic T-lymphocyte-associated antigen-4 (CTLA-4) antibody (including ipilimumab or any other antibody or drug specifically targeting T-cell co-stimulation or checkpoint pathways). 10. Has a known history of Human Immunodeficiency Virus (HIV). 11. Has a known active Hepatitis B or Hepatitis C 12. Have received a live vaccine within 30 days prior to the first dose of trial treatment. 13. Gastrointestinal tract disease or defect or previous history of colitis. 14. Has proven or suspected distant metastasis that involves occurrence of breast cancer outside of loco-regional breast and lymph nodes area. 15. Participants requiring daily corticosteroids either via po or infusion 16. Myocardial infarction within 6 months before starting therapy, symptomatic congestive heart failure (New York Heart Association \> class II), unstable angina, or unstable cardiac arrhythmia requiring medication.