Z-Endoxifen Hydrochloride in Treating Patients With Metastatic or Locally Recurrent Estrogen Receptor-Positive Breast Cancer

Inclusion Criteria: * Histologically confirmed diagnosis of metastatic or locally recurrent breast cancer * ER positive defined as \> 1% nuclear staining on the biopsy that was obtained at the confirmation of metastatic or locally recurrent disease * Lesion type of either evaluable or measurable disease * Pre- or post-menopausal female * For the expansion cohorts: tumor that is accessible for biopsy * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 * Life expectancy \> 16 weeks * Capable of understanding investigational nature, potential risks and benefits of the study and able to provide written informed consent * Absolute neutrophil count (ANC) \>= 1,000/uL * Platelet count \>= 75,000/uL * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional ULN (\< 5 x institutional ULN if liver function test \[LFT\] elevations due to liver metastases) * Creatinine =\< 1.5 x institutional ULN * Women with human epidermal growth factor (HER)-2 positive disease must have received and progressed on at least one prior anti-HER-2 directed regimen (trastuzumab, lapatinib) for their metastatic disease * For dose escalation cohort: * Any number of prior systematic therapy regimens is allowed * NOTE: prior systematic therapy in the adjuvant setting is not required * At least one prior hormone containing regimen in the metastatic setting (tamoxifen if pre-menopausal; aromatase inhibitor if post-menopausal) * NOTE: exception: patients that fail (defined as the development of metastatic disease while receiving an adjuvant hormone containing regimen of tamoxifen for premenopausal and aromatase inhibitor for postmenopausal) are eligible and not required to receive additional hormonal containing regimens prior to enrollment * At least one prior chemotherapy containing regimen in adjuvant and/or metastatic setting * For the expansion cohort(s): * At least one prior hormone containing regimen in the metastatic setting (tamoxifen if pre-menopausal; aromatase inhibitor if post-menopausal) * NOTE: exception: patients that fail (defined as the development of metastatic disease while receiving an adjuvant hormone containing regimen of tamoxifen for premenopausal and aromatase inhibitor for postmenopausal) are eligible and not required to receive additional hormonal containing regimens prior to enrollment * NOTE: a prior hormone containing regimen in the adjuvant setting is not required; a hormonal regimen containing everolimus is allowed * Either 1 or 2 prior chemotherapy regimens are allowed but not required such that both are in the metastatic setting or one is in the adjuvant setting and one in the metastatic setting (note, an anthracycline and taxane based regimen delivered in the adjuvant setting would be considered one regimen) * Willingness to return to Mayo Clinic Rochester, Arizona, or Florida during treatment phase of the trial * Dose Escalation cohort only: * Mandatory Translational Research Components * Willingness to provide biologic specimens (blood and urine) * Dose Escalation cohorts beginning at 160 mg/day: Mandatory Translational Research Components * Willingness to provide biologic specimens (tissue) * Dose Expansion cohort(s): * Mandatory Translational Research Components * Willingness to provide biologic specimens (blood, tissue and urine) * Note: The goals of this study include assessment of the biologic effects on surrogate markers of Z-endoxifen and therefore, are contingent upon availability of the biologic specimens * Women of childbearing potential only: negative serum pregnancy test done =\< 48 hours prior to registration * Capable of swallowing 20-mg capsules Exclusion Criteria: * Any of the following therapies prior to registration: * Chemotherapy =\< 3 weeks * Immunotherapy =\< 3 weeks * Biologic therapy =\< 3 weeks * Hormonal therapy =\< 3 weeks * Monoclonal antibodies =\< 3 weeks * Radiation therapy =\< 3 weeks * Anti-Her-2 directed therapy =\< 3 weeks * Prior endoxifen therapy * Prior history of: * Stroke =\< 6 months prior to registration * Seizures =\< 3 months prior to registration * Deep vein thrombosis (DVT) or pulmonary embolism (PE) =\< 12 months prior to registration * Two or more episodes of DVT and/or PE =\< 5 years prior to registration * Crystalline retinopathy * Abnormal uterine bleeding =\< 1 year prior to registration * Personal history of coagulopathy * Active DVT and/or PE requiring anti-coagulant therapy * Patients who are on anti-coagulant therapy for maintenance are eligible as long as the DVT and/or PE was \> 12 months prior to enrollment and there is no evidence for active thrombosis (either DVT or PE) * Clinically symptomatic cataracts requiring imminent surgery * Note: patients that have cataracts that do not require surgery are eligible * Other invasive malignancy that has been diagnosed or has recurred \< 2 years prior to registration; EXCEPTIONS: non-melanotic skin cancer or carcinoma-in-situ of the cervix * Any co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the investigator, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, hypertension, or psychiatric illness/social situations that would limit compliance with study requirements * Failure to fully recover from acute, reversible effects of prior chemotherapy regardless of interval since last treatment; EXCEPTION: neuropathies - if grade 2 neuropathies have been stable for at least 3 months since completion of prior treatment patient is eligible * Tumors involving the spinal cord or heart * Uncontrolled brain metastases * Note: brain metastases are not permitted on study unless the metastases have been treated by surgery or radiotherapy, and the patient has been neurologically stable and off steroids for \>= 12 weeks * Plans to begin bisphosphonates or denosumab after registration or began a bisphosphonate or denosumab regimen \< 90 days before registration * Note: patients on a stable dose of bisphosphonates or denosumab for \> 90 days prior to registration are eligible * Any of the following: * Pregnant women * Nursing women * Women of childbearing potential who are unwilling to employ adequate contraception * Other concurrent chemotherapy or anti HER2 therapy, immunotherapy, radiotherapy, or any ancillary therapy considered investigational (utilized for a non-Food and Drug Administration \[FDA\]-approved indication and in the context of a research investigation)