TRUDI: TDXD+Durva in HER2+/Low IBC

Inclusion Criteria: * Participants must have a histological or cytological diagnosis of invasive breast cancer. * All histologic subtypes are eligible. * Participants must have a clinical diagnosis of stage III inflammatory breast cancer within the past 6 months * HER2-positive status as determined locally by the current ASCO/CAP guidelines or HER2-low tumor expression (IHC 2+/ISH-, IHC 1+/ISH-, or IHC 1+/ISH untested) (note: ISH may be determined by either fluorescence in situ hybridization \[FISH\] or dual in situ hybridization \[DISH\]) * Any ER and PR expressions are permitted but must be known * Participants must be treatment-naïve * Participants must agree to undergo two research biopsies of the tumor (if safely accessible, as determined by the treating investigator): at baseline (prior to the first treatment) and after the first week of treatment on C1D8. Previously collected archival tissue will also be obtained on all participants. For participants for whom the tumor is not safely accessible, this archival tissue needs to be located and availability confirmed at time of registration. * Pre- and postmenopausal women or male patients ≥ 18 years of age * ECOG performance status 0-1 (Karnofsky \> 60%, see Appendix A). * LVEF ≥ 50% within 28 days prior to enrollment * Participants must have normal organ and marrow function prior to enrollment as defined below: * Absolute neutrophil count ≥2,000/mcL * Platelets ≥100,000/mcL * Hemoglobin ≥ 9.0 g/dl * INR/PT/aPTT ≤1.5 × ULN unless participant is receiving anticoagulant therapy as long as PT or aPTT is in therapeutic range of anticoagulant * Total bilirubin ≤1.5 × institutional upper limit of normal (ULN)(or ≤2.0 x ULN in patients with documented Gilbert's Syndrome) * AST(SGOT)/ALT(SGPT) ≤2.5 × institutional ULN * Serum creatinine ≤1.5 × institutional ULN OR creatinine clearance ≥ 30 mL/min/ 1.73m2 for participants with creatinine levels above institutional ULN. * Serum albumin ≥2.5 g/dL * International normalized ratio (INR)/prothrombin time (PT) and either partial thromboplastin or activated partial thromboplastin time (aPTT) ≤1.5 × ULN * Women of childbearing potential (WOCBP) and WOCBP who are partners of male participants must agree to use one highly effective non-hormonal form of contraception or two effective forms of non-hormonal contraception for the duration of study treatment with durvalumab and 7 months after the last dose of study treatment * Males who are sexually active with WOCBP must agree to follow instructions for method(s) of contraception for the duration of study treatment with durvalumab and 4 months after the last dose of study treatment * Must have a life expectancy of at least 12 weeks * Body weight \>30 kg * The participant must be capable of understanding and complying with the protocol and willing to sign a written informed consent document Exclusion Criteria: * Has received prior systemic anti-cancer therapy for the current diagnosis of inflammatory breast cancer, including chemotherapy, immunotherapy, or targeted therapy * Has received any radiotherapy or surgery for the current diagnosis of inflammatory breast cancer. Tumor biopsies are not considered surgery for the purpose of enrollment. * Prior hypersensitivity to durvalumab or the excipients of durvalumab or trastuzumab deruxtecan or history of severe hypersensitivity reactions to other monoclonal antibodies. * Major surgery within 4 weeks prior to study treatment initiation. Patients must have recovered from any effects of any major surgery. * Uncontrolled intercurrent illness, including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, uncontrolled cardiac arrhythmia, serious chronic gastrointestinal conditions associated with diarrhea, or psychiatric illness/social situations that would limit compliance with study requirements, substantially increase risk of incurring AEs, or compromise the ability of the patient to give written informed consent. * Participant has a medical condition that requires chronic systemic steroid therapy (\> 10 mg of prednisone daily or equivalent) or any other form of immunosuppressive medication (including disease modifying agents) and has required such therapy in the last 2 years. Replacement therapy (e.g. thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic therapy. * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[eg, colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, sarcoidosis syndrome, Wegener syndrome \[granulomatosis with polyangiitis\], Graves' disease, and rheumatoid arthritis, hypophysitis, uveitis, etc). The following are exceptions to this criterion: \-- Patients with vitiligo or alopecia, Patients with hypothyroidism (eg, following Hashimoto syndrome) stable on hormone replacement, Patients with any chronic skin condition that does not require systemic therapy, Patients with celiac disease controlled by diet alone, who may also be included, but only after consultation with the Principal Investigator, Patients without active disease in the last 5 years, who may also be included but only after consultation with the Principal Investigator * History of (non-infectious) ILD/pneumonitis that required steroids, has current ILD/pneumonitis, or where suspected ILD/pneumonitis cannot be ruled out by imaging at screening. * Lung-specific intercurrent clinically significant illnesses including, but not limited to, any underlying pulmonary disorder (ie, pulmonary emboli within 3 months of the study enrollment, severe asthma, severe chronic obstructive pulmonary disease (COPD), restrictive lung disease, pleural effusion etc.), and any autoimmune, connective tissue or inflammatory disorders with pulmonary involvement (ie, rheumatoid arthritis, Sjogren's syndrome, sarcoidosis etc.), and prior pneumonectomy * Corrected QT interval (QTcF) prolongation to \> 470 msec on screening EKG * Any factors that increase the risk of QTc prolongation or risk of arrhythmic events, such as heart failure, hypokalemia, potential for torsades de pointes, congenital long QT syndrome, family history of long QT syndrome or unexplained sudden death under 40 years of age, or any concomitant medication known to prolong the QT interval * Any of the following procedures or conditions in the 6 months prior to enrollment: coronary artery bypass graft, angioplasty, vascular stent, myocardial infarction, angina pectoris, congestive heart failure (New York Heart Association Functional Classification Grade ≥2), and stroke. Subjects with troponin levels above ULN at screening (as defined by the manufacturer), and without any myocardial infarction related symptoms, should have a cardiology consultation before enrollment to rule out myocardial infarction. * Cardiac ejection fraction outside institutional range of normal or \<50% (whichever is higher) as measured by echocardiogram (or multiple-gated acquisition \[MUGA\] scan if an echocardiogram cannot be performed or is inconclusive). * History of another primary malignancy, except for Malignancy treated with curative intent and with no known active disease ≥5 years before the first dose of study treatment and of low potential risk for recurrence. Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease, Adequately treated carcinoma in situ without evidence of disease * History of venous thromboembolism in the past 3 months. * History of active primary immunodeficiency. * Active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and tuberculosis testing in line with local practice), hepatitis B (known positive hepatitis B virus \[HBV\] surface antigen \[HBsAg\] result) or hepatitis C virus (HCV). Patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible. Patients positive for hepatitis C antibody are eligible only if the polymerase chain reaction is negative for HCV RNA. Some medications used for these conditions have drug-drug interactions with the study treatment. * Known to have previously tested positive for human immunodeficiency virus (HIV) (positive HIV 1/2 antibodies)(HIV testing is not required for participation on this study) * Receipt of a live vaccine within 30 days prior to study treatment initiation. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster, yellow fever, rabies, BCG, and typhoid vaccine. The use of the inactivated seasonal influenza vaccine is allowed. * Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the study, interfere with the subject's participation for the full duration of the study, or is not in the best interest of the subject to participate, in the opinion of the treating investigator. * Use of potent inhibitors or inducers or substrates of CYP3A4 or substrates of CYP2C9 or CYP2D6 within 2 weeks before the first dose of study treatment (3 weeks for St John's Wort) * Use of hydroxychloroquine in \<14 days prior to Day 1 of trastuzumab deruxtecan treatment * History of leptomeningeal carcinomatosis * Female patients who are pregnant, breastfeeding or of reproductive potential who are not willing to employ effective birth control from screening to 6 months after the last dose of durvalumab and at least 7 months after the final administration of trastuzumab deruxtecan due to the potential for teratogenic effects. And unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother