Temsirolimus in Treating Patients With Locally Advanced or Metastatic Breast Cancer

Inclusion Criteria: * Patients must have histologically or cytologically confirmed metastatic or recurrent breast cancer not amenable to local therapy (surgery and radiation) (histologic/cytologic confirmation of recurrence preferred, but not required) * Either the primary or metastatic tumor must be positive for estrogen receptor (\>= 1% by immunohistochemical staining) and/or progesterone receptor (\>= 1% by immunohistochemical staining) and/or human epidermal growth factor receptor (HER2neu) (3+ immunohistochemical staining or fluorescence in situ hybridization \[FISH\] positive) * Patients must have measurable disease; measurable disease is defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) as \>= 20 mm with conventional techniques or as \>= 10 mm with spiral computed tomography (CT) scan * There are no limitations on the number of prior therapy regimens; however, patients who have had prior exposure to rapamycin or any other mechanistic target of rapamycin (mTOR) inhibitor are excluded from the trial * Eastern Cooperative Oncology Group (ECOG) performance status 0 or 1 * Absolute neutrophil count \>= 1,500/uL * Platelets \>= 100,000/uL * Total bilirubin =\< institutional upper limit of normal * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 times institutional upper limit of normal * Creatinine =\< 2.0 x normal institutional upper limit of normal * Cholesterol =\< 350 mg/dL (fasting) * Triglycerides =\< 400 mg/dL (fasting) * Albumin \>= 3.3 mg/dL * Women of child-bearing potential and men must agree to use adequate contraception (barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately; women of child-bearing potential must have a negative pregnancy test prior to treatment on study; breastfeeding should be discontinued if the mother is treated with temsirolimus * Ability to understand and the willingness to sign a written informed consent document * Tissue for correlative studies must be available and the subject must agree to use of tissue for these studies Exclusion Criteria: * Patients must be off of hormonal agents used for the treatment of breast cancer for one week with the exception that premenopausal women who have been on a gonadotropin-releasing hormone (GnRH) agonist and subsequently progressed may, at the discretion of the treating physician, continue on the GnRH agonist * Patients should have recovered from the adverse effects of prior chemotherapy; in general, this will mean that the patient would have been due or overdue for the next dose of the prior regimen: three weeks should have elapsed for a regimen administered once every three weeks, etc * Radiotherapy should have been completed * Three weeks should have elapsed since prior therapy with monoclonal antibodies * Patients may not be receiving any other investigational agents or herbal preparations; patients may not be taking corticosteroids except in low doses as replacement for adrenal insufficiency or for short -term (less than 5 days) use for other reasons * Patients with known brain metastases are not permitted on study unless the metastases have been controlled by prior surgery or radiotherapy, and the patient has been neurologically stable and off of steroids for at least 4 weeks * Patients cannot be receiving enzyme-inducing antiepileptic drugs (enzyme-inducing antiepileptic drugs \[EIAEDs\]; e.g., phenytoin, carbamazepine, phenobarbital) nor any other CYP3A4 inducer such as rifampin or St. John's wort, as these may decrease temsirolimus levels; use of agents that potently inhibit CYP3A (and hence may raise temsirolimus levels), such as ketoconazole, is discouraged, but not specifically prohibited; CCI-779 can inhibit CYP2D6, and may decrease metabolism (and increase drug levels) of drugs that are substrates for CYP2D6, such as codeine; the appropriateness of use of such agents is left to physician discretion; if there is any doubt about eligibility based on concomitant medication, the study chair, Dr Fleming, should be contacted; all concomitant medications must be recorded * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, uncontrolled symptomatic cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Human immunodeficiency virus (HIV)-positive patients receiving combination anti-retroviral therapy are excluded from the study * Patients with known hypersensitivity reactions to macrolide antibiotics (such as erythromycin, clarithromycin, and azithromycin) are not eligible for this trial