Cixutumumab and Temsirolimus in Treating Younger Patients With Solid Tumors That Have Recurred or Not Responded to Treatment

Inclusion Criteria: * Patients must have had histologic verification of malignancy at original diagnosis or relapse except in patients with intrinsic brain stem tumors, optic pathway gliomas, or patients with pineal tumors and elevations of serum or cerebrospinal fluid (CSF) alpha-fetoprotein or beta-human chorionic gonadotropin (HCG); slides or tissue blocks from either initial diagnosis or relapse must be available for central review * Patients must have either measurable or evaluable disease * Patient's current disease state must be one for which there is no known curative therapy or therapy proven to prolong survival with an acceptable quality of life * Karnofsky \>= 50% for patients \> 16 years of age and Lansky \>= 50 for patients =\< 16 years of age; note: neurologic deficits in patients with central nervous system (CNS) tumors must have been clinically stable for a minimum of 1 week prior to study enrollment; patients who are unable to walk because of paralysis, but who are up in a wheelchair, will be considered ambulatory for the purpose of assessing the performance score * Patients must have fully recovered from the acute toxic effects of all prior chemotherapy, immunotherapy, or radiotherapy prior to entering this study * Myelosuppressive chemotherapy: must not have received within 3 weeks of enrollment onto this study (6 weeks if prior nitrosourea) * Hematopoietic growth factors: at least 7 days since the completion of therapy with a growth factor that supports platelet or white cell number or function * Biologic (anti-neoplastic agent): at least 7 days since the completion of therapy with a biologic agent; at least 6 weeks must have elapsed since prior therapy that includes a monoclonal antibody; for agents that have known adverse events occurring beyond 7 days after administration, this period must be extended beyond the time during which adverse events are known to occur; the duration of this interval must be discussed with the study chair * Radiation therapy (XRT): \>= 2 wks for local palliative XRT (small port); \>= 3 months must have elapsed if prior total-body irradiation (TBI), craniospinal XRT or if \>= 50% radiation of pelvis; \>= 6 wks must have elapsed if other substantial bone marrow (BM) radiation * Stem cell transplant or rescue: no evidence of active graft vs. host disease and \>= 2 months must have elapsed since transplant * Peripheral absolute neutrophil count (ANC) \>= 1000/mm\^3 * Platelet count \>= 100,000/mm\^3 (transfusion independent, defined as not receiving platelet transfusions within a 7 day period prior to enrollment) * Hemoglobin \>= 8.0 g/dL (may receive red blood cell \[RBC\] transfusions) * Creatinine clearance or radioisotope glomerular filtration rate (GFR) \>= 70 mL/min/1.73 m\^2 OR a serum creatinine based on age/gender as follows: * 0.6 mg/dL (for patients 1 year of age) * 0.8 mg/dL (for patients 2 to 5 years of age) * 1 mg/dL (for patients 6 to 9 years of age) * 1.2 mg/dL (for patients 10 to 12 years of age) * 1.5 mg/dL (males) or 1.4 mg/dL (females) (for patients 13 to 15 years of age) * 1.7 mg/dL (males) or 1.4 mg/dL (females) (for patients \>= 16 years of age) * Bilirubin (sum of conjugated + unconjugated) =\< 1.5 x upper limit of normal (ULN) for age * Serum albumin \>= 2 g/dL * Patients with seizure disorder may be enrolled if receiving non-enzyme inducing anticonvulsants and well controlled * Prothrombin time (PT) and international normalized ratio (INR) \< 1.2 x ULN * Random or fasting blood glucose within the upper normal limits for age; if the initial blood glucose is a random sample that is outside of the normal limits, then a follow-up fasting blood glucose can be obtained and must be within the upper normal limits for age * Serum cholesterol and serum triglyceride levels must be \< grade 2 * All patients and/or their parents or legal guardians must sign a written informed consent; assent, when appropriate, will be obtained according to institutional guidelines Exclusion Criteria: * Serum glutamate pyruvate transaminase (SGPT) (alanine aminotransferase \[ALT\]) =\< 110 U/L; for the purpose of this study, the ULN for SGPT is 45 U/L * Pregnant or breast-feeding women will not be entered on this study; pregnancy tests must be obtained in girls who are post-menarchal; males or females of reproductive potential may not participate unless they have agreed to use an effective contraceptive method for the duration of the study and for 3 months after the last dose of IMC-A12. * Patients receiving corticosteroids who have not been on a stable or decreasing dose of corticosteroid for the 7 days prior to enrollment are not eligible * Patients who are currently receiving another investigational drug are not eligible * Patients who are currently receiving other anticancer agents are not eligible * Patients receiving insulin or growth hormone therapy are not eligible * Patients must not be receiving enzyme-inducing anticonvulsants * Patients must not be receiving any of the following potent cytochrome P450, family 3, subfamily A, polypeptide 4 (CYP3A4) inducers or inhibitors: erythromycin, clarithromycin, ketoconazole, azithromycin, itraconazole, grapefruit juice or St. John's wort * Patients receiving warfarin for the purpose of systemic anticoagulation are not eligible; use of low-dose warfarin for maintaining patency of central venous catheters is allowed * Patients who have an uncontrolled infection are not eligible * Patients with known type I or type II diabetes mellitus are not eligible * Patients with known bone marrow involvement are not eligible * Patients who in the opinion of the investigator may not be able to comply with the safety monitoring requirements of the study are not eligible * Patients who have received prior monoclonal antibody therapy targeting IGF-1R or temsirolimus are not eligible * Patients with history of allergic reactions attributed to compounds of similar chemical; or biologic composition to IMC-A12 or temsirolimus are not eligible * Patients must not have had major surgery for 6 weeks prior to enrollment on the study; patients with history of recent minor surgical procedures (vascular catheter placement, bone marrow evaluation, laparoscopic surgery) will be eligible