A Study of Olaparib and Pembrolizumab in People With Triple Negative Breast Cancer (TNBC) or Hormone Receptor-positive HER2-negative Breast Cancer

Inclusion Criteria: * Signed Informed Consent Form * Ability to comply with protocol, in the investigator's judgment * Newly diagnosed histologically confirmed Stage T1c-3N0-3: 1. TNBC (ER/PgR negativity will be defined using IHC per ASCO/CAP criteria/HER2-negativity will be defined using ISH or IHC assays per ASCO/CAP criteria) OR 2. Hormone receptor-positive, HER2-negative breast cancer defined as per ASCO/CAP criteria * Measurable disease per RECIST v1.1 * Minimum tumor size of 1.5 cm * All patients must have a germline mutation in BRCA1, BRCA2, PALB2, RAD51C, or RAD51D. * ECOG Performance Status of 0 or 1 * Patient agreement to undergo appropriate surgical management including, if indicated, axillary lymph node surgery and partial or total mastectomy after completion of neoadjuvant treatment. Note: consideration of neoadjuvant treatment will be determined by disease management team based on the subtypes and minimum size the tumor has to be. * Baseline LVEF ≥ 53% measured by echocardiogram (ECHO) or multiple-gated acquisition (MUGA) scans * Have adequate organ function as defined by the following laboratory results obtained within 14 days prior to the first study treatment * Hematological * Absolute neutrophil count (ANC) ≥1500/μL * Platelets ≥100 000/μL * Hemoglobin ≥9.0 g/dL or ≥5.6 mmol/La * Renal * Creatinine OR ≤1.5 × ULN OR * Measured or calculated creatinine clearance (GFR can also be used in place of creatinine or CrCl) ≥30 mL/min for participant with creatinine levels \>1.5 × institutional ULN * Hepatic * Total bilirubin ≤1.5 ×ULN OR direct bilirubin ≤ULN for participants with total bilirubin levels \>1.5 × ULN Patients with known Gilbert syndrome who have serum bilirubin level ≤ 3 X ULN may be enrolled. * AST (SGOT) and ALT (SGPT) ≤2.5 × ULN * A female participant is eligible to participate if she is not pregnant, not breastfeeding, and at least one of the following conditions applies: 1. Not a woman of childbearing potential (WOCBP) as defined by institutional guidelines OR 2. A WOCBP who agrees to follow the contraceptive guidance starting at the time of informed consent, during the treatment period and for at least 120 days after the last dose of study treatment (pembrolizumab, olaparib) according to local standard of care. * Male participants must agree to use contraception starting at the time of informed consent, during the treatment period and for at least 120 days after the last dose of study treatment (pembrolizumab, olaparib) and refrain from donating sperm during this period. Exclusion Criteria: * Prior history of invasive breast cancer * Stage IV (metastatic) breast cancer * Prior systemic therapy for treatment and prevention of breast cancer * Contraindication to MRI scan and/or allergy to intravenous contrast-gadolinium * Has received prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent or with an agent directed to another stimulatory or co-inhibitory T-cell receptor (eg, CTLA-4, OX-40, CD137). * Has received prior systemic anti-cancer therapy including investigational agents within 4 weeks. Note: Participants must have recovered from all AEs due to previous therapies to ≤Grade 1 or baseline. Participants with ≤Grade 2 neuropathy may be eligible. Note: If participant received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting study treatment. * Has received prior radiotherapy within 2 weeks of start of study treatment. Participants must have recovered from all radiation-related toxicities, not require corticosteroids, and not have had radiation pneumonitis. * Has received a live vaccine within 30 days prior to the first dose of study drug. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster (chicken pox), yellow fever, rabies, Bacillus Calmette-Guérin (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (eg, FluMist®) are live attenuated vaccines and are not allowed. * Has a diagnosis of immunodeficiency or is receiving chronic systemic steroid therapy (in dosing exceeding 10 mg daily of prednisone equivalent) or any other form of immunosuppressive therapy within 7 days prior to the first dose of study drug. * Has a known additional malignancy that is progressing or has required active treatment within the past X 3 years prior to screening. Note: Participants with basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or cervical carcinoma in situ are not excluded. Patients with prior ductal/lobular carcinoma in situ are not excluded if they were treated exclusively with mastectomy \> 3 years prior to diagnosis of current breast cancer. * Has severe hypersensitivity (≥Grade 3) to pembrolizumab or olaparib and/or any of its excipients. * Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. * Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. * Has a history of (non-infectious) pneumonitis that required steroids or has current pneumonitis * Severe infections within 4 weeks prior to initiation of study treatment, including but not limited to hospitalization for complications of infection, bacteremia, or severe pneumonia. * Treatment with therapeutic oral or IV antibiotics within 2 weeks prior to initiation of study treatment. a. Patients receiving prophylactic antibiotics (e.g., for prevention of a urinary tract infection or to prevent chronic obstructive pulmonary disease exacerbation) are eligible for the study. * Has a known history of Human Immunodeficiency Virus (HIV). * Has a known history of Hepatitis B (defined as Hepatitis B surface antigen \[HBsAg\] reactive) or known active Hepatitis C virus (defined as HCV RNA \[qualitative\] is detected) infection. * Has a known history of active TB Bacillus Tuberculosis. * Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the study, interfere with the subject's participation for the full duration of the study, or is not in the best interest of the subject to participate, in the opinion of the treating investigator. * Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial. * Is pregnant or breastfeeding or expecting to conceive or father children within the projected duration of the study, starting with the screening visit through 120 days after the last dose of trial treatment. * A WOCBP who has a positive urine pregnancy test within 72 hours prior to study start. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required. * Note: If 72 hours have elapsed between the screening pregnancy test and the performed and must be negative in order for subject to start receiving study medication. * Patients who have undergone hCG-stimulation for egg harvest as part of fertility preservation within 7 days before treatment initiation are permitted to receive treatment despite a positive pregnancy test.