Neratinib and Paclitaxel With or Without Pertuzumab and Trastuzumab Before Combination Chemotherapy in Treating Patients With Metastatic or Locally Advanced Breast Cancer

Inclusion Criteria: * Histological confirmation of breast cancer * Able to provide written informed consent for the trial * Performance status of =\< 1 on the Eastern Cooperative Oncology Group (ECOG) performance scale * Able to swallow oral medication * Left ventricular ejection fraction (LVEF) assessment by 2-dimensional (D) echocardiogram or multi-gated acquisition (MUGA) scan performed within 90 days prior to registration must be \>= 50% * Absolute neutrophil count \>= 1,500 /uL * Platelets \>= 100,000 /uL * Hemoglobin \>= 9 g/dL * Creatinine clearance \>= 50 ml/min * Total bilirubin =\< 1.5 X upper limit of normal (ULN), for patients with congenital unconjugated hyperbilirubinemia (Crigler-Najjar syndrome type 1 and 2, Gilbert syndrome) that transient hyperbilirubinemia can occur due to physiological condition, as long as there is clear documentation of diagnosis, allowed to be enrolled if direct (conjugated) bilirubin is =\< 1.5 X ULN * Alanine aminotransferase and aspartate aminotransferase =\< 2.5 X ULN except in patients with aspartate aminotransferase (AST)/alanine aminotransferase (ALT) elevation that is declared to be caused due to liver metastasis, they are allowed to be enrolled as long as \< 5 x ULN * Subject of childbearing potential should be willing to use effective methods of birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study and through at least 4 months after the last dose of study drug; subject of childbearing potential is defined as has not been surgically sterilized or free from menses for \> 1 year; effective methods of birth control include: 1) use of hormonal birth control methods: pills, shots/injections, implants (placed under the skin by a health care provider), or patches (placed on the skin); 2) intrauterine devices (IUDs); 3) using 2 barrier methods (each partner must use 1 barrier method) with a spermicide; males must use the male condom (latex or other synthetic material) with spermicide; females must choose either a diaphragm with spermicide, or cervical cap with spermicide, or a sponge (spermicide is already in the contraceptive sponge); female patients of childbearing potential must have a negative urine pregnancy test no more than 7 days prior to starting study drug; 4) for male participant, they must agree and commit to use a barrier method of contraception while on treatment and for 3 months after the last dose of investigational product * Cohort 1: Phase 1b: patient must have HER2 + (regardless of hormonal receptor status) metastatic or locally advanced breast cancer (IBC or Non-IBC); HER2 positive status is defined as strongly positive (3+) staining score by immunohistochemistry (IHC), or gene amplification using fluorescence in situ hybridization (FISH), if performed; if IHC is equivocal (2+), please refer to current American Society of Clinical Oncology (ASCO) guidelines algorithm for evaluation of HER2; HER2 negative status, which is determined by assays using IHC require negative (0 or 1+) staining score; if IHC is equivocal (2+) staining score, please refer to current ASCO guidelines algorithm for evaluation of HER2; IBC is determined by using international consensus criteria: onset: rapid onset of breast erythema, edema and/or peau d'orange, and/or warm breast, with/without an underlying breast mass; duration: history of such findings no more than 6 months; extent erythema occupying at least 1/3 of whole breast; pathology: pathologic confirmation of invasive carcinoma * Cohort 1: Phase II: patient must have HER2+ (regardless of hormonal receptor status) stage III IBC or Stage IV IBC if the metastatic sites are amenable for local therapy (i.e. radiation and/ or surgery) and will have breast surgery * Cohort 2: Patient must have HER2-/HR+ stage III IBC or Stage IV IBC if the metastatic sites are amenable for local therapy (i.e. radiation and/ or surgery) and will have breast surgery Exclusion Criteria: * Excisional biopsy or lumpectomy for the current breast cancer * Any other previous malignancies (except for cervical in situ cancers treated only by local excision, and basal and squamous cell carcinomas of the skin) within 5 years * Any other previous antitumor therapies for the current cancer event; this exclusion does not apply to phase Ib part of cohort 1 * Breast-feeding at screening or planning to become pregnant during the course of therapy * History of active or known autoimmune disease that can cause diarrhea like (but not limited to) Addison's disease, celiac disease/gluten intolerance/irritable bowel syndrome, scleroderma * Active infection or chronic infection requiring chronic suppressive antibiotics * Known hepatitis B or hepatitis C with abnormal liver function tests * Malabsorption syndrome, ulcerative colitis, inflammatory bowel disease, resection of the stomach or small bowel, or other disease or condition significantly affecting gastrointestinal function * Persistent \>= grade 2 diarrhea regardless of etiology * Sensory or motor neuropathy \>= grade 2 * Conditions that would prohibit intermittent administration of corticosteroids for paclitaxel premedication; however, corticosteroid can be dropped after confirming of no asthma like reaction to paclitaxel after 3 doses * Uncontrolled hypertension defined as a systolic blood pressure (BP) \> 150 mmHg or diastolic BP \> 90 mmHg, with or without anti-hypertensive medications * Cardiac disease (history of and/or active disease) that would preclude the use of any of the drugs included in the treatment regimen; this includes but is not confined to: * Active cardiac diseases including: * Symptomatic angina pectoris within the past 180 days that required the initiation of or increase in anti-anginal medication or other intervention * Ventricular arrhythmias except for benign premature ventricular contractions * Supraventricular and nodal arrhythmias requiring a pacemaker or not controlled with medication * Conduction abnormality requiring a pacemaker * Valvular disease with documented compromise in cardiac function * Symptomatic pericarditis * History of cardiac disease: * Myocardial infarction documented by elevated cardiac enzymes or persistent regional wall abnormalities on assessment of left ventricular (LV) function * History of documented congestive heart failure (CHF) * Documented cardiomyopathy * If you are pregnant, you will not be enrolled on this study