TVB-2640 and Trastuzumab With Paclitaxel or Endocrine Therapy for Treatment of HER2 Positive Metastatic Breast Cancer

Inclusion Criteria: * PRE-REGISTRATION INCLUSION CRITERIA * Age \>=18 years * Measurable disease as defined by Response Evaluation Criteria in Solid Tumors (RECIST) criteria that is: * A non-nodal lesion is considered measurable if its longest diameter can be accurately measured as \>= 1.0 cm with computed tomography (CT) scan, CT component of a positron emission tomography (PET)/CT, or magnetic resonance imaging (MRI) and/or * A malignant lymph node is considered measurable if its short axis is \> 1.5 cm when assessed by CT scan (CT scan slice thickness recommended to be no greater than 5 mm) * Note: tumor lesions in a previously irradiated area are not considered measurable disease; disease that is measurable by physical examination only is not eligible * Received =\< six (6) prior chemotherapy regimens in the metastatic setting * Cohort A one of the following must be true: * Distant disease progression during administration of combination therapy with taxane based chemotherapy and anti-HER2 therapy (trastuzumab or pertuzumab) for metastatic disease * Note: patients who began treatment with this combination and discontinued taxane-based chemotherapy due to intolerability before distant disease progression are eligible * Distant disease progression during administration or within 180 days of discontinuing combination therapy with taxane based chemotherapy and anti-HER2 therapy (trastuzumab or pertuzumab) in the adjuvant disease * Note: patients who began treatment with this combination and discontinued taxane-based chemotherapy due to intolerability before distant disease progression are eligible * For patient who received taxane based chemotherapy and anti-HER2 therapy (trastuzumab or pertuzumab) in the neo-adjuvant setting and underwent surgical resection of primary breast disease: distant disease progression during or within 180 days of discontinuing anti-HER2 therapy (trastuzumab or pertuzumab) in the adjuvant setting * Cohort B (one of the following must be true): * Distant disease progression during administration of combination therapy with endocrine therapy and anti-HER2 therapy (trastuzumab or pertuzumab) for metastatic disease; permissible endocrine therapies include an aromatase inhibitor or fulvestrant * NOTE: Tamoxifen is not permissible * Distant disease progression during administration of combination therapy with endocrine therapy and anti-HER2 therapy (trastuzumab or pertuzumab) in the adjuvant setting; permissible endocrine therapies include an aromatase inhibitor or fulvestrant * NOTE: Tamoxifen is not permissible * Willingness to provide mandatory tumor tissue specimens for correlative research * NOTE: If insufficient or no tissue is obtained by the pre-registration biopsy, an archival tissue specimen (preferably from a metastatic site) from procedure performed =\< 5 years prior to pre-registration must be available to submit for Central Laboratory review prior to registration * Exception: If there is no medically safe site for biopsy, Study Chair (Dr. Haddad) may waive this requirement * REGISTRATION INCLUSION CRITERIA * Registration must be completed =\< 28 days of pre-registration * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 * Histological confirmation of HER2-expression (immunohistochemistry \[IHC\] 1-3+) on pre-registration biopsy. (HER2+ is defined by 2018 American Society of Clinical Oncology/College of American Pathologists \[ASCO/CAP\] guidelines) * NOTE: If pre-registration biopsy is HER2- by ASCO/CAP guidelines then both of the following must be true: * The patient's most recent prior biopsy (prior to the pre-registration biopsy) must be HER2+, AND * The managing provider must be planning to treat the patient with anti-HER2 therapy * For Cohort B only: Histologic confirmation of ERalpha positive disease (\>= 1% expression) * Hemoglobin \>= 9.0 g/dL (obtained =\< 14 days prior to registration) * Absolute neutrophil count (ANC) \>= 1500/mm\^3 (obtained =\< 14 days prior to registration) * Platelet count \>= 100,000/mm\^3 (obtained =\< 14 days prior to registration) * Direct bilirubin =\< 1.5 x upper limit of normal (ULN) (obtained =\< 14 days prior to registration) * Aspartate transaminase (AST) =\< 3 x ULN (=\< 5 x ULN for patients with liver involvement) (obtained =\< 14 days prior to registration) * Calculated creatinine clearance \>= 45 ml/min using the Cockcroft-Gault formula (obtained =\< 14 days prior to registration) * Cardiac ejection fraction (left ventricular ejection fraction \[LVEF\]) \>= 50% by echocardiogram =\< 28 days prior to registration * Provide written informed consent * Willing to return to enrolling institution for follow-up (during the active monitoring phase of the study) * Negative urine pregnancy test done =\< 7 days prior to registration, for women of childbearing potential only * NOTE: If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Patient and his/her partner agree to use adequate contraception after providing written informed consent through 3 months after the last dose of TVB-2640, as follows: * For women: Compliant with a medically-approved contraceptive regimen during and for 3 months after the treatment period or documented to be surgically sterile or postmenopausal * For men: Compliant with a medically-approved contraceptive regimen during and for 3 months after the treatment period or documented to be surgically sterile; men whose sexual partners are of child-bearing potential must agree to use 2 methods of contraception prior to study entry, during the study, and for 3 months after the treatment period * Willingness to provide mandatory tumor tissue and/or blood specimens for correlative research Exclusion Criteria: * PRE-REGISTRATION EXCLUSION CRITERIA * Patients who previously discontinued trastuzumab due to unacceptable cardiac toxicity * Patients with a history of LVEF decline to below 50% during or after prior trastuzumab or other HER2 directed therapy =\< 6 months prior to pre-registration * Patients with any class of New York Heart Association (NYHA) congestive heart failure (CHF) or heart failure with preserved ejection fraction (HFPEF) * Patients with a history of known coronary artery disease or a myocardial infarction within 12 months prior to pre-registration * Patients with persistently uncontrolled hypertension (systolic blood pressure \[BP\] \> 160 mm Hg or diastolic BP \> 100 mm Hg) despite optimal medical therapy * Patients with known unstable angina pectoris * Patients with a known history of serious cardiac arrhythmias requiring treatment (exception: controlled atrial fibrillation, paroxysmal supraventricular tachycardia) * Patients with a prolonged corrected QT interval (QTc) interval (\>= 450 ms) * Leptomeningeal disease or uncontrolled brain metastasis * NOTE: Metastases treated by surgery and/or radiotherapy such that patient is neurologically stable and off steroids \>= 4 weeks prior to preregistration are eligible * Failure to recover from acute, reversible effects of prior therapy regardless of interval since last treatment * EXCEPTION: Grade 1 peripheral (sensory) neuropathy that has been stable for at least 3 months since completion of prior treatment * Tumors involving spinal cord or heart * Visceral crisis or lymphangitic spread * NOTE: Visceral crisis is not the mere presence of visceral metastases, but implies severe organ dysfunction as assessed by symptoms and signs, laboratory studies, and rapid progression of disease * Uncontrolled intercurrent non-cardiac illness including, but not limited to, * Ongoing or active infection * Psychiatric illness/social situations * Dyspnea at rest due to complications of advanced malignancy or other disease that requires continuous oxygen therapy * Or any other conditions that would limit compliance with study requirements * Immunocompromised patients and patients known to be human immunodeficiency virus (HIV) positive and currently receiving antiretroviral therapy * NOTE: Patients known to be HIV positive, but without clinical evidence of an immunocompromised state, are eligible for this trial * Receiving any other investigational agent which would be considered as a treatment for the primary neoplasm * History of myocardial infarction =\< 6 months, or congestive heart failure requiring use of ongoing maintenance therapy for life-threatening ventricular arrhythmias * Prior history of hypersensitivity, drug or radiation-induced, or other immune-mediated pneumonitis * Patient is unable to swallow oral medications or has impairment of gastrointestinal (GI) function or GI disease that may significantly alter drug absorption (e.g. active inflammatory bowel disease, uncontrolled nausea, vomiting, diarrhea, or malabsorption syndrome); note: concomitant therapy with proton pump inhibitors and/or H2-receptor antagonists is permissible * Patient has a history of clinically significant dry eye (xerophthalmia) or other corneal abnormality, or if a contact lens wearer, does not agree to abstain from contact lens use from baseline through the last TVB-2640 dose * Patients with a history of intolerance to trastuzumab (i.e. a grade 3 or 4 infusion reaction) are excluded; Note: patients with a history of mild infusion reaction to trastuzumab who have previously been successfully re-challenged after an infusion reaction with or without prophylactic medication are allowed * Other invasive malignancy =\< 3 years prior to pre-registration * EXCEPTIONS: Non-melanoma skin cancer, papillary thyroid cancer, or carcinoma-in-situ of the cervix which has been adequately treated * NOTE: If there is a history of prior malignancy, patients must not be receiving other treatment for their cancer and the disease must be inactive/stable * REGISTRATION EXCLUSION CRITERIA * Any of the following: * Pregnant women * Nursing women * Men or women of childbearing potential who are unwilling to employ adequate contraception * Any of the following therapies prior to registration: * Chemotherapy =\< 3 weeks * Immunotherapy =\< 3 weeks * Biologic therapy =\< 3 weeks * Monoclonal antibodies =\< 3 weeks * Radiation therapy =\< 2 weeks * CDK 4/6 inhibitors =\< 4 weeks * mTOR inhibitors =\< 4 weeks