Study of Pembrolizumab and Mifepristone in Patients With Advanced HER2-negative Breast Cancer

Inclusion Criteria: Common to both cohorts * Patients must have histologically confirmed breast cancer that is metastatic or locally advanced and unresectable. * Patients must have evaluable disease as defined by RECIST 1.1 with tumor lesion \> 10 mm by CT scan or caliper measurement on clinical exam or lymph node ≥ 15mm in short axis. * ECOG performance status of 0 or 1. * Age ≥ 18 years. Because no dosing or adverse event data are currently available on the use of pembrolizumab in combination with mifepristone in patients \< 18 years of age, children are excluded from this study, but will be eligible for future pediatric trials. * Patients must have normal organ and marrow function as defined below: * absolute neutrophil count ≥ 1,000/mcL * platelets ≥ 80,000/mcL * total bilirubin within normal institutional limits * AST(SGOT)/ALT(SGPT) ≤ 2.5 × institutional upper limit of normal * creatinine clearance ≥ 60 mL/min/1.73 m2 * INR ≤ 1.5 x ULN unless subject is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use on anticoagulant. * Females of child-bearing potential (defined as a sexually mature woman who has not undergone hysterectomy, bilateral oophorectomy, or who has not been naturally postmenopausal for at least 24 consecutive months prior to study enrollment) must: * Commit to abstinence from heterosexual contact or agree to use effective contraception without interruption beginning at least 28 days prior to starting protocol therapy, while on study medication, and for 6 months following the last dose of therapy. * Have a negative serum pregnancy test (β -hCG) result at screening. * Male subjects must practice abstinence or agree to use a condom during sexual contact with a pregnant female or a female of childbearing potential while participating in the study, during dose interruptions and for 6 months following protocol discontinuation, even if he has undergone a successful vasectomy. * Ability to understand and the willingness to sign a written informed consent document. * Any number of prior therapies is allowed. * Patients must consent to pre and on treatment research biopsies * Patients must have measurable disease. Cohort 1 (TNBC): * Have diagnosis of triple negative breast cancer (defined as ER \< 1% by IHC, PR \< 1% by IHC, Her 2 negative by ASCO CAP guidelines). * Patients with TNBC may be enrolled in dose expansion cohort A with any number of prior lines of therapy (patients will be allowed to enroll in frontline setting). Cohort 2 (HR+): * Have a diagnosis of ER+ and/or PR+ breast cancer (defined as ER/PR \> 1% by IHC). * Patients with ER+ and/or PR+ breast cancer must have progressed on at least 2 lines of endocrine therapy, unless ER/PR expression is low (\< 10% positive) or weakly expressed, and the treating physician does not feel endocrine therapy is appropriate. Exclusion Criteria: * Has a known additional malignancy that is progressing or requires active treatment. Exceptions include basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or in situ cervical cancer that has undergone potentially curative therapy. * Patients who have had chemotherapy or radiotherapy prior to entering the study must have recovered from adverse events to Grade 1. Radiation within 3 months of study treatment initiation is prohibited, as serious rashes have been observed in patients who have recently received radiation therapy. * Is currently participating and receiving study therapy or has participated in a study of an investigational agent and received study therapy or used an investigational device within 4 weeks of the first dose of treatment. * Prior treatment with an anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CTLA-4 agent, or mifepristone * Known active central nervous system (CNS) metastases and/or carcinomatous meningitis. Subjects with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging at least 4 weeks prior to first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 7 days prior to treatment. * History of allergic reactions attributed to compounds of similar chemical or biologic composition to pembrolizumab or mifepristone * Uncontrolled intercurrent medical or psychiatric illness that would limit compliance with study requirements. * Pregnant women are excluded from this study because Mifepristone is an abortifacient agent with the potential for teratogenic effects. * Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with Mifepristone, breastfeeding should be discontinued if the mother wishes to participate in this study. * HIV-positive patients on combination antiretroviral therapy are ineligible because of the potential for pharmacokinetic interactions with pembrolizumab and/or mifepristone. In addition, these patients are at increased risk of lethal infections when treated with marrow-suppressive therapy. Appropriate studies will be undertaken in patients receiving combination antiretroviral therapy when indicated. * Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (eg., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. * Has evidence of active, non-infectious pneumonitis * Has an active infection requiring intravenous systemic therapy * Has received a live vaccine or live-attenuated vaccine within 30 days prior to first dose of therapy. Administration of killed vaccines is allowed. * Has a known human immunodeficiency virus (HIV), known active Hepatitis B (e.g. HBsAg reactive), or Hepatitis C (e.g. HCV RNA \[qualitative\] is detected) infection. * Long term or concurrent use of corticosteroid therapy other than for premedication or supportive care use as detailed in the protocol. Physiologic doses of steroids (e.g. equivalent to or less than oral prednisone 10mg daily) are allowed and do not require approval. * Mifepristone can both inhibit CYP3A4 and induce CYP3A4. Addition of mifepristone to a pre-existing drug regimen may cause a mild and temporary increase in plasma drug concentration of drugs with significant CYP3A4 metabolism. Medications that are strong inducers of CYP3A4 such as carbamazepine, oxcarbazepine, phenobarbital, phenytoin, primidone, rifabutin, rifampin, rifapentine, St. John's Wort may decrease plasma mifepristone levels. Strong CYP3A4 inhibitor medications are expected to cause the largest increases in plasma mifepristone concentrations. * Mifepristone may increase the plasma drug concentration of concomitant medications with metabolism mediated by CYP2C9/CYP2C8. Drugs with the largest increases will be those whose metabolism is largely or solely mediated by CYP2C9/2C8 and include: Non-steroidal Anti-inflammatory drugs (NSAIDs) and warfarin. * Has evidence of active, non-infectious pneumonitis/interstitial lung disease that required steroids or has current pneumonitis/interstitial lung disease.