Phase Ib Study of Axatilimab in Combination With Olaparib in BRCA1/2 and PALB2- Associated Metastatic HER2-negative Breast Cancer

Inclusion Criteria: * Participants must have histologically or cytologically confirmed metastatic or unresectable HER2 negative breast cancer, including HER2 low (IHC 2+/ISH-, IHC 1+). Any ER and PR expressions are permitted but must be known. Patients with hormone receptor (HR) positive disease, defined by either ER and/or PR positivity, must have progressed or are intolerant to all available endocrine therapy regimens, or not candidates to further endocrine therapy-based approaches. * Documented germline or somatic mutation in BRCA1, BRCA2, or germline mutation in PALB2 that is predicted to be deleterious or suspected deleterious (known or predicted to be detrimental/lead to loss of function). Testing may be completed by any CLIA-certified laboratory. * Participants must have evaluable or measurable disease per RECIST 1.1 criteria. NOTE: If the only site of measurable of disease has been previously irradiated, there must be evidence of post-radiation progression. * Patients must have received no more than 2 prior lines of cytotoxic chemotherapy for metastatic disease. Antibody drug conjugates will count towards prior lines of cytotoxic chemotherapy, as well as checkpoint inhibitors. For the purposes of this study, prior treatment with hormonal therapy and non-hormonal targeted therapy, as well as the combination of an aromatase inhibitor and everolimus, are not counted as a prior cytotoxic therapy. * Age ≥18 years. * ECOG performance ≤ 2. * Participants must meet the following organ and marrow function as defined below: * leukocytes ≥ 3000/mcL * absolute neutrophil count ≥ 1.5 x 109/L * platelets ≥ 100 x 109/L * total bilirubin ≤ 1.5x institutional upper limit of normal (ULN) * AST(SGOT)/ALT(SGPT) ≤ 2.5x × institutional ULN or ≤5 × institutional ULN for participants with documented liver metastases * Creatinine clearance ≥ 51 mL/min (using Cockcroft-Gault equation) * Participants who are HBsAg positive are eligible if they have received HBV anti-viral therapy for at least 4 weeks and have undetectable HBV viral load prior to registration. * Note: Participants should remain on anti-viral therapy throughout study intervention and follow local guidelines for HBV anti-viral therapy post completion of study intervention. * Hepatitis B screening tests are not required unless: * Known history of HBV infection * As mandated by local health authority * Participants with a history of HCV infection are eligible if HCV viral load is undetectable at screening. * Note: Participants must have completed curative anti-viral therapy at least 4 weeks prior to registration. * Hepatitis C screening tests are not required unless: * Known history of HCV infection * As mandated by local health authority * HIV-infected participants must have well-controlled HIV on ART, defined as: * a. Participants on ART must have a CD4+ T-cell count ≥350 cells/mm3 at the time of screening. * b. Participants on ART must have achieved and maintained virologic suppression defined as confirmed HIV RNA level below 50 or the LLOQ (below the limit of detection) using the locally available assay at the time of screening and for at least 12 weeks before screening. * c. It is advised that participants must not have had any AIDS-defining opportunistic infections within the past 12 months. * d. Participants on ART must have been on a stable regimen, without changes in drugs or dose modification, for at least 4 weeks before study entry (Day 1) and agree to continue ART throughout the study. * e. The combination ART regimen must not contain any antiretroviral medications that interact with CYP3A4 inhibitors/inducers/substrates (https://www.fda.gov/drugs/drug-interactions-labeling/drug-development-and- drug-interactions-table-substrates-inhibitors-and-inducers) * Ability to swallow and retain oral study medication * Patients with a history of treated central nervous system (CNS) metastases are eligible, provided they meet all of the following criteria: * Disease outside the CNS is present * No clinical evidence of progression in the CNS since completion of CNS-directed therapy * Minimum of 2 weeks between completion of radiotherapy and Cycle 1 Day 1 * Recovery from significant (≥ Grade 3) acute toxicity with no requirement for escalating doses of corticosteroid over the 7 days prior to treatment start. * Participants with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial. * Female participants must be postmenopausal or must have a negative serum pregnancy test performed during screening. Postmenopausal is defined as: * Amenorrhoeic for 1 year or more following cessation of exogenous hormonal treatments * Luteinizing hormone (LH) and follicle stimulating hormone (FSH) levels in the postmenopausal range for women under 50. * Radiation-induced oophorectomy with last menses \>1 year ago * Chemotherapy-induced menopause with \>1 year interval since last menses * Bilateral oophorectomy (with or without hysterectomy) or tubal ligation at least six weeks ago * The effects of axatilimab and olaparib on the developing human fetus are unknown. For this reason, women of child-bearing potential and men who are sexually active with WOCBP must agree to use adequate contraception for the duration of study participation and for 4 months after discontinuation of treatment. * Participants must be willing to undergo 3 research biopsies: at baseline, after 2 weeks of olaparib monotherapy, and after 2 cycles of combination therapy. If biopsy is not feasible or safe, OR if the only area accessible to biopsy is also the only site of measurable disease per RECIST 1.1 criteria, permission must be obtained from the DFCI sponsor- investigator to forgo the mandatory research biopsies. Formal eligibility exception would not be required in these circumstances. * Ability to understand and the willingness to sign a written informed consent document. Exclusion Criteria: * Clinical progression on a PARP inhibitor, or within 12 months of receipt of a PARP inhibitor, including but not limited to olaparib. * Any previous treatment with CSF1R antibody. * Patients who have had prior systemic chemotherapy, immune therapy, or investigational therapy within three weeks of initiation of protocol therapy. Endocrine therapy must have been discontinued at least 7 days prior to initiation of protocol therapy. Patients may receive bisphosphonates or denosumab during the study. * Participants who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> Grade 1) with the exception of alopecia, are excluded. * History of allergic reactions attributed to compounds of similar chemical or biologic composition to axatilimab or olaparib. * Participants receiving any medications or substances that are strong or moderate inhibitors or inducers of CYP450 enzyme(s) are ineligible. This also includes strong or moderate CYP3A inhibitors and inducers. Because the lists of these agents are constantly changing, it is important to regularly consult a frequently-updated medical reference. * Participants with a QTcF of \>470msec on screening ECG * Patients with a history of myelodysplastic syndrome (MDS)/acute myeloid leukemia (AML) or with features suggestive of MDS/AML * Participants unable to swallow orally administered medication and participants with gastrointestinal disorders that are likely to interfere with absorption of the study medications in the opinion of the treating investigator (e.g. malabsorption syndrome or major stomach or bowel resections). * Uncontrolled intercurrent illness including, but not limited to, uncontrolled hypertension, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements. * History of pneumonitis or ILD, or evidence of pneumonitis/ILD on baseline imaging * Known active or latent tuberculosis * Patients with major surgical procedure within 28 days prior to initiation of protocol therapy.