Trastuzumab Deruxtecan Alone or in Combination With Anastrozole for the Treatment of Early Stage HER2 Low, Hormone Receptor Positive Breast Cancer

Inclusion Criteria: * Previously untreated operable invasive carcinoma of the breast greater than 2.0 cm (cT2) in size based on physical exam or imaging. Patients with clinical node negative disease or clinical node (cN1/cN2) positive are allowed provided they are deemed to have operable disease at study entry * Participants with clinically involved lymph nodes should not have radiological evidence of distant disease per standard of care staging prior to patient informed consent form (PICF) signature * In the United States * Tumor is HER2-low by immunohistochemistry (IHC), defined as 1+ or 2+, confirmed by central testing (central testing results not required for enrollment, unless no local results available). If HER2 is 2+ by IHC, fluorescence in situ hybridization (FISH) must be performed (per standard of care) and the FISH result must be HER2 non-amplified per 2018 American Society of Clinical Oncology College of American Pathologists (ASCO CAP) guidelines * Tumor is HR positive (HR+) per ASCO CAP guidelines with known estrogen and progesterone receptor status, locally defined * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 * Normal cardiac function (left ventricular ejection fraction \[LVEF\] \>= 50%) based on echocardiogram (ECHO) or multigated acquisition (MUGA) scan within 28 days before randomization/enrollment * Platelet count \>= 100 000/mm\^3 (Platelet transfusion is not allowed within 1 week prior to screening assessment) (within 14 days before randomization/enrollment) * Hemoglobin \>= 9.0 g/dL (red blood cell transfusion is not allowed within 1 week prior to screening assessment) (within 14 days before randomization/enrollment) * Absolute neutrophil count (ANC) \>=1500/mm\^3 (Granulocyte colony-stimulating factor (G-CSF) administration is not allowed within 1 week prior to screening assessment) (within 14 days before randomization/enrollment) * Creatinine clearance \>= 30 mL/min as calculated using the Cockcroft-Gault equation or serum creatinine =\< 1.5 x upper limit of normal (ULN) (within 14 days before randomization/enrollment) * Alanine aminotransferase (ALT), aspartate aminotransferase (AST) =\< 3 x ULN (within 14 days before randomization/enrollment) * Total bilirubin =\< 1.5 x ULN (within 14 days of randomization/enrollment). Participants with Gilbert's syndrome with a total bilirubin =\< 2.0 times ULN and direct bilirubin within normal limits are permitted * Serum albumin \>= 2.5 g/dL (within 14 days before randomization/enrollment) * International normalized ratio (INR)/prothrombin time (PT) and activated partial thromboplastin time (aPTT) =\< 1.5 x ULN (within 14 days before randomization/enrollment) * Has adequate treatment washout period before randomization/enrollment, defined as: * Major surgery \>= 4 weeks * Chloroquine/hydroxychloroquine \> 14 days * Negative pregnancy test (serum) for women of child bearing potential (CBP): * Women are considered of CBP unless: they have had 12 months of natural (spontaneous) amenorrhea with an appropriate clinical profile (i.e. age appropriate, history of vasomotor symptoms) or have had surgical bilateral oophorectomy (with or without hysterectomy), total hysterectomy, or tubal ligation at least six weeks prior to randomization. In the case of oophorectomy alone, only when the reproductive status of the woman has been confirmed by follow-up hormone level assessment she is considered not of CBP * Male and female participants of reproductive/childbearing potential must agree to use a highly effective form of contraception or avoid intercourse during and upon completion of the study and for at least 7 months for females and 4 months for males after the last dose of study drug. Highly effective contraception methods include: * Total abstinence (when this is in line with the preferred and usual lifestyle of the patient). Periodic abstinence (e.g. calendar, ovulation, symptothermal, post-ovulation methods) and withdrawal are not acceptable methods of contraception * Female sterilization (have had surgical bilateral oophorectomy with or without hysterectomy), total hysterectomy or tubal ligation at least 6 weeks before taking trial treatment. In case of oophorectomy alone, only when the reproductive status of the woman has been confirmed by follow up hormone level assessment * Male partner sterilization (at least 6 months prior to randomization). For female patients on the trial the vasectomized male partner should be the sole partner for that patient. If vasectomy of the male partner is the highly effective method of contraception chosen, the success of the vasectomy should be medically confirmed according to local practice * Placement of an intrauterine device (IUD) * Male participants must not freeze or donate sperm starting at screening and throughout the study period, and at least 4 months after the final study drug administration. Preservation of sperm should be considered prior to enrollment in this study * Female participants must not donate, or retrieve for their own use, ova from the time of screening and throughout the study treatment period, and for at least 7 months after the final study drug administration * Estradiol level must be in post-menopausal range per local lab interpretation prior to baseline biopsy * Postmenopausal status is defined as: * Patient underwent bilateral oophorectomy, or * Age \>= 60 years, or * Age \< 60 years and amenorrhea for 12 or more months (in the absence of chemotherapy, tamoxifen, toremifene or ovarian suppression) and follicle-stimulating hormone (FSH) and plasma estradiol are in the postmenopausal ranges per local normal ranges * Note: for women with therapy-induced amenorrhea, serial measurements of FSH and/or estradiol per local clinical guidelines are required for determination of postmenopausal status. All women who do not meet the criteria for postmenopausal status are considered premenopausal for the purpose of this trial * Pre- or peri-menopausal and amenable to being treated with ovarian function suppression drugs (goserelin, leuprolide, or triptorelin) per standard of care. Patients must have started treatment with ovarian function suppression at least 28 days prior to first dose of study treatment Exclusion Criteria: * Recurrent or metastatic breast cancer * Bilateral breast cancer (multifocal or multicentric breast cancer is allowed provided that all biopsied lesions are HER2 1+ or 2+, not FISH amplified and are HR positive per ASCO guidelines) * Inflammatory breast cancer * Prior systemic therapy for invasive cancer * Prior tamoxifen for history of ductal breast carcinoma in situ (DCIS) allowed, but no prior aromatase inhibitor, no prior chemotherapy and no prior HER2-targeted therapy * Prior ipsilateral chest wall radiation * Major surgery \< 4 weeks prior to enrollment * Medical history of myocardial infarction within 6 months before randomization/enrollment, symptomatic congestive heart failure (CHF) (New York Heart Association Class II to IV), troponin levels consistent with myocardial infarction as defined according to the manufacturer 28 days prior to randomization * Unable to swallow oral medications * Is pregnant or lactating, or planning to become pregnant * Corrected QT interval prolongation to \> 470 ms (females) or \> 450 ms (males) based on average of the screening triplicate 12-lead electrocardiogram * Known hypercoaguable disorder requiring use of anticoagulant * Significant gastrointestinal disorders limiting absorption or tolerance of oral medications (for example, history of major surgical resection involving the stomach or small bowel, or preexisting Crohn's disease or ulcerative colitis or a preexisting chronic condition resulting in baseline grade 2 or higher diarrhea) * History of (non-infectious) interstitial lung disease (ILD)/pneumonitis that required steroids, has current ILD/pneumonitis, or where suspected ILD/pneumonitis cannot be ruled out by imaging at screening * Has multiple primary malignancies within 3 years, except: * Adequately resected non-melanoma skin cancer * Curatively treated non breast in-situ disease, and other solid non-breast tumors curatively treated are allowed if \> 3 years from diagnosis and no evidence of recurrence in that time * Prior history of DCIS is allowed as long as patient has not received an aromatase inhibitor, has not received ipsilateral breast/chest radiation * Prior history of contralateral invasive breast cancer (diagnosed by biopsy \> 2 years prior to current diagnosis) is allowed provided patient has not received prior aromatase inhibitor, CDK4/6 inhibitor (CDK4/6i), HER2-targeted therapy or chemotherapy and has not experienced any recurrence and has no evidence of recurrence (based on standard clinical evaluation) * Other concurrent anti-cancer therapy. Note: ovarian function suppression drugs (goserelin, leuprolide, or triptorelin) and/or bone modifying agents (bisphosphonates, denosumab) do not count as anti-cancer therapy for this criteria. If taking bisphosphonates or denosumab, must have been on these agents prior to signing consent * Has substance abuse or any other medical conditions such as clinically significant cardiac or psychological conditions, that may, in the opinion of the investigator, interfere with the subject's participation in the clinical study or evaluation of the clinical study results * Has known human immunodeficiency virus (HIV) infection, or active hepatitis B or C infection. Patients positive for hepatitis C (HCV) antibody are eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA). Subjects should be tested for HIV prior to randomization/enrollment if required by local regulations or Institutional Review Board (IRB)/ethics committee (EC) * Have personal history within the last 12 months of any of the following conditions: syncope of cardiovascular etiology, ventricular tachycardia, ventricular fibrillation, or sudden cardiac arrest * Have received an autologous or allogeneic stem-cell transplant * Has active systemic bacterial infection (requiring intravenous \[IV\] antibiotics at time of initiating study treatment), fungal infection, or detectable viral infection (such as known human immunodeficiency virus positivity or with known active hepatitis B or C \[for example, hepatitis B surface antigen positive\]). Screening is not required for enrollment * Concurrent treatment with ovarian hormonal replacement therapy. Prior treatment must be stopped prior to first baseline biopsy * Has history of severe hypersensitivity reactions to other monoclonal antibodies and/or to either the drug substances or inactive ingredients in the drug product * Clinically severe pulmonary compromise resulting from intercurrent pulmonary illnesses including, but not limited to, any underlying pulmonary disorder (i.e. pulmonary emboli within three months of the study enrollment, severe asthma, severe chronic obstructive pulmonary disease \[COPD\], restrictive lung disease, pleural effusion etc.), and any autoimmune, connective tissue or inflammatory disorders with pulmonary involvement (i.e. rheumatoid arthritis, Sjogren's, sarcoidosis etc.), or prior pneumonectomy * Life expectancy \< 3 months