Cixutumumab and Temsirolimus in Treating Patients With Locally Recurrent or Metastatic Breast Cancer

Inclusion Criteria: * Histologically confirmed diagnosis of breast cancer with diagnosis of metastatic or locally recurrent disease (locally recurrent disease should be stage IV e.g. chest wall involvement) * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 (Karnofsky \>= 80%) * Life expectancy of \> 12 weeks * Capable of understanding investigational nature, potential risks and benefits of the study and able to provide written informed consent * Negative serum pregnancy test =\< 7 days of registration for women of childbearing potential: * Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) for the duration of study therapy and for 3 months after the last dose of IMC-A12 and CCI-779 (temsirolimus) * Should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately * Nursing women must be willing to discontinue nursing; NOTE: breastfeeding should be discontinued if the mother is treated with CCI-779 and IMC-A12 * Absolute neutrophil count \>= 1,500/mcL * Hemoglobin \>= 8.5 g/dL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 X institutional upper limit of normal (ULN) * Aspartate aminotransferase (AST)(serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional ULN (=\< 5 X institutional ULN if liver function test \[LFT\] elevations due to liver metastases) * Creatinine =\< 1.5 X institutional ULN OR creatinine clearance \>= 60 mL/min/1.73\^2 for patients with creatinine \> institutional ULN * Fasting serum cholesterol =\< 350 mg/dL (9.0 mmol/L) * Fasting triglycerides =\< 400 mg/dL (4.56 mmol/L) * Albumin \>= 3.4 mg/dL * Fasting or non fasting serum glucose \< 120 mg/dL * Hemoglobin A1c (HbA1c) (for all patients with a history of diabetes mellitus) \< 8% * Phase I only: Any number of prior therapy regimens is allowed * Phase II only: Measurable disease is required for the Phase II portion of the study; measurable disease is defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) as \>= 20 mm with conventional techniques (computed tomography \[CT\], magnetic resonance imaging \[MRI\], x-ray) or as \>= 10 mm with spiral CT scan * Phase II only: =\< two and at least one prior chemotherapy regimens in the setting of metastatic or locally recurrent (stage IV chest wall involvement) disease are required Exclusion Criteria: * Phase I patients only: Patients with base line diabetes requiring oral hypoglycemics or insulin * Phase II patients only: Poorly controlled diabetes mellitus; NOTE: patients with a history of diabetes mellitus on oral hypoglycemics or insulin are allowed to participate, provided that their fasting blood glucose is \< 120 mg/dL and that they are on a stable dietary or therapeutic regimen for this condition * Any of the following: * Pregnant women * Nursing women * Men or women of childbearing potential who are unwilling to employ adequate contraception (hormonal agents are not allowed and oral contraceptives are not acceptable for contraception) * Receiving hormonal agents used for the treatment of breast cancer with the exception that premenopausal women who have been on a gonadotropin-releasing hormone (GnRH) agonist and subsequently progressed may, at the discretion of the treating physician, continue on the GnRH agonist * Any of the following prior therapies: * Systemic anti-cancer therapy =\< 3 weeks prior to registration * Radiation therapy =\< 2 weeks prior to registration * Prior invasive non-breast malignancy, except for adequately treated basal or squamous cell carcinoma of the skin or other cancer from which the patient has been disease free for \>= 5 years * Known hypersensitivity reactions to macrolide antibiotics (such as erythromycin, clarithromycin, and azithromycin); allergic reactions attributed to compounds of similar chemical or biologic composition to IMC-A12, or temsirolimus * Prior treatment with agents targeting the insulin-like growth factor-I receptor (IGF-IR)/insulin-like growth factors (IGFs) or phosphatidylinositol-4,5-bisphosphate 3-kinase, catalytic subunit alpha (PI3K)/v-akt murine thymoma viral oncogene homolog 1 (Akt)/mechanistic target of rapamycin (mTOR) pathway * Receiving any other investigational agents or herbal preparations * Patients may not be taking oral corticosteroids except for replacement for adrenal insufficiency * Uncontrolled brain metastases; Note: brain metastases are not permitted on study unless the metastases have been treated by surgery or radiotherapy, and the patient has been neurologically stable and off of steroids for \>= 12 weeks * Known human immunodeficiency virus (HIV)-positive patients who have cluster of differentiation (CD)4 counts below the normal range or who are on anti-retroviral therapy that may interfere with the metabolism of temsirolimus * Uncontrolled intercurrent illness including, but not limited to: * Ongoing or active infection * Symptomatic congestive heart failure * Unstable angina pectoris * Uncontrolled symptomatic cardiac arrhythmia * Psychiatric illness/social situations that would limit compliance with study requirements * Receiving enzyme-inducing antiepileptic drugs (EIAEDs; e.g., phenytoin, carbamazepine, phenobarbital) or any other cytochrome P450, family 3, subfamily A, polypeptide 4 (CYP3A4) inducer such as rifampin or St. John's wort