Fulvestrant With or Without Lapatinib in Treating Postmenopausal Women With Stage III or Stage IV Breast Cancer That is Hormone Receptor-Positive

Inclusion Criteria: * Histologic, pathologic or cytologic diagnosis of cancer of the female breast in either primary or metastatic setting; histological documentation of metastatic/recurrent disease is not required if there is unequivocal clinical evidence for recurrence * Stage IV breast cancer (using American Joint Committee on Cancer \[AJCC\] criteria, 6th edition), or locally advanced (stage III) breast cancer not considered amenable to curative therapy * Patients with symptomatic brain metastases or other symptomatic central nervous system (CNS) metastases are not eligible for the study; no screening studies are required among asymptomatic patients; patients with previously treated brain metastases, who are free of symptoms referable to CNS disease and who are \> 3 months from treatment for brain metastases are eligible * Tumors (as determined on pathology from either primary or metastatic sites) must be potentially sensitive to endocrine therapy, defined as expressing estrogen receptor (ER) and/or progesterone receptor (PgR) as determined immunohistochemical methods according to the local institution's standard protocol, \>= 1% cells will be considered to be positive * The protocol has been amended to permit tumors with any HER2 status, though a determination of HER2 status must have been made; patients will be considered to be eligible if HER2 expression is documented by one of the following methods: * Immunohistochemistry (IHC) 0 (i.e., negative), 1+, 2+, or 3+ levels of expression, or * Gene amplification (fluorescent in situ hybridization \[FISH\]) positive or negative * Patients must have at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) as \>= 2.0 cm with conventional techniques or as \>= 1.0 cm with spiral computed tomography (CT) scan * Exception: Patients with lytic or blastic bone metastases as their only site of disease will be eligible for the study even though these patients are not considered to have measurable disease by Response Evaluation Criteria in Solid Tumors (RECIST) criteria; these patients will be evaluable for time to progression, but not response * Patients with all other lesions, including small lesions (longest diameter \< 2.0 cm with conventional techniques or \< 1.0 cm with spiral CT scan) and truly non-measurable lesions including those listed below are not eligible * Lesions that are considered non-measurable include the following: * Bone lesions (women with bone lesions will be eligible as described above) * Leptomeningeal disease * Ascites * Pleural/pericardial effusion * Inflammatory breast cancer * Lymphangitis cutis/pulmonitis * Abdominal masses that are not confirmed and followed by imaging techniques * Cystic lesions * Patients must have had one or two prior endocrine treatments for breast cancer in either the adjuvant or metastatic setting, exclusive of treatment-related amenorrhea or ovarian suppression; sequential use of two different third-generation aromatase inhibitors is considered "one" treatment; it is not required that tumors be resistant to such treatments; for example: * A patient with de novo metastatic breast cancer who had never received endocrine therapy is not eligible; * A patient who received adjuvant tamoxifen and subsequent therapy with an aromatase inhibitor (adjuvant or metastatic) is eligible; * A patient who received an aromatase inhibitor in either the adjuvant or metastatic setting, and who discontinued therapy after several months because of side effects, is eligible; * A patient who received an aromatase inhibitor in the adjuvant setting is eligible, regardless of whether they did or did not receive tamoxifen at some point; * A patient who received adjuvant tamoxifen, and subsequently a nonsteroidal aromatase inhibitor and a steroidal aromatase inhibitor for advanced breast cancer in the adjuvant or metastatic setting is eligible; * A patient who received adjuvant tamoxifen, and then a nonsteroidal aromatase inhibitor and subsequently megesterol acetate for advanced breast cancer is not eligible * Tumors potentially sensitive to endocrine therapy, defined as \>= 3 months of prior endocrine therapy without disease progression in the adjuvant or metastatic setting * Patients must have had prior treatment in either the adjuvant or metastatic setting with a commercially available third-generation aromatase inhibitor (i.e. anastrozole, exemestane, or letrozole); it is not required that tumors be resistant to such therapies * Patients may have received up to one prior chemotherapy regimen for stage IV breast cancer; prior chemotherapy in the adjuvant and/or neoadjuvant setting is permitted; patients must have finished chemotherapy at least 1 week prior to starting protocol based treatment * Patients may have received prior trastuzumab therapy for stage IV breast cancer, in combination with up to one chemotherapy and/or endocrine therapy regimen, but that must have concluded at least 3 weeks prior to starting protocol-based therapy; prior trastuzumab therapy in the adjuvant and/or neoadjuvant setting is permitted, but must have concluded at least 3 weeks prior to starting protocol-based therapy * Prior therapy with commercially available inhibitor of EGFR (including but not limited to gefitinib, erlotinib, lapatinib or cetuximab) or experimental inhibitors of EGFR is prohibited * Patients may have initiated bisphosphonate therapy prior to study entry; such patients will have bone lesions considered evaluable for progression but not for response * Prior fulvestrant therapy is prohibited * Patients receiving a gonadotropin-releasing hormone (GnRH) agonist for ovarian suppression must remain on such therapy throughout the course of protocol treatment; patients must discontinue other endocrine treatments, including systemic hormone-replacement therapy and intravaginal estrogens prior to study entry; patients must have concluded radiation therapy prior to study entry; patients must be at least 1 week from prior chemotherapy or 3 weeks from prior trastuzumab therapy, with adequate recovery of bone marrow function and performance status * Patients must be postmenopausal women, defined as a woman fulfilling any of the following criteria: * Age \>= 60 years; or * Age \>= 45 years with an intact uterus and amenorrhea for 12 months or more; or * History of bilateral oophorectomy; or * Follicle stimulating hormone (FSH) levels within postmenopausal range according to the ranges established by the testing facility; or * Treatment with a GnRH agonist for ovarian suppression for at least 3 consecutive months prior to study registration, and remaining on such therapy throughout the course of protocol treatment * Women who are pregnant or nursing are not eligible for the study; clinicians should advise patients that there are no data for the safety of lapatinib or fulvestrant among pregnant patients, nor data on the impact of these agents on fertility or pregnancy * Eastern Cooperative Oncology Group (ECOG) (Zubrod) performance status 0-2 * Absence of pending visceral crisis, in the opinion of the treating physician * Absence of acquired or inherited bleeding disorder * Absence of need for therapeutic systemic anticoagulation (defined as maintaining international normalized ratio \[INR\] \> 1.6); patients may take low-dose warfarin or aspirin (or equivalent) for maintenance of central venous catheter patency * Granulocytes \>= 1,000/μl * Platelet count \>= 100,000/μl * Creatinine =\< 2 mg/dl * Total bilirubin =\< 1.5 x upper limits of normal (ULN) unless due to Gilbert's syndrome * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x ULN without liver metastases; =\< 5 x ULN with liver metastases * INR =\< 1.6 * Left ventricular ejection fraction (LVEF) within institutional limits of normal