Paclitaxel With or Without Cixutumumab as Second-Line Therapy in Treating Patients With Metastatic Esophageal Cancer or Gastroesophageal Junction Cancer

Inclusion Criteria: * Life expectancy \>= 12 weeks * Women must not be pregnant or breast-feeding due to potential harm to fetus from cixutumumab (IMC-A12) and paclitaxel; all females of childbearing potential must have a blood test or urine study within 48 hours prior to registration to rule out pregnancy * Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method or birth control; abstinence) for the duration of study therapy and for 3 months after the last dose of cixutumumab (IMC-A12); should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately * Patients must have measurable disease * Patients must have metastatic disease of the esophagus or gastroesophageal junction * Histologic, cytologic or radiologic documentation of metastatic squamous cell carcinoma or adenocarcinoma of the esophagus or gastroesophageal junction; radiologic, endoscopic, histologic or cytologic evidence of locally recurrent or locally residual (post-resection) disease is also permitted * For the purposes of this study, undifferentiated adenocarcinomas and adenosquamous tumors will be considered as adenocarcinomas; in addition, tumors involving the gastroesophageal junction will be defined by the Siewert classification * Patients with gastroesophageal junction tumors who are eligible: * Adenocarcinoma of the esophageal junction (AEG) Type I: adenocarcinoma of the distal esophagus which usually arises from an area with specialized intestinal metaplasia of the esophagus, i.e., Barrett's esophagus, and may infiltrate the esophagogastric junction from above * AEG Type II: true carcinoma of the cardia arising from the cardiac epithelium or short segments with intestinal metaplasia at the esophagogastric junction * Patients with gastroesophageal junction tumors who are NOT eligible: * AEG Type III: subcardial gastric carcinoma which infiltrates the esophagogastric junction and distal esophagus from below * Patients must have received and progressed on one and only one line of prior systemic therapy for esophagus or esophagogastric cancer; this could have included one regimen for metastatic disease, or one regimen with radiotherapy for initially locally advanced disease; prior radiation therapy is permitted * If patients progress or recur within 6 months of neoadjuvant/adjuvant therapy, this will be considered one line of therapy; for patients progressing or recurring more than 6 months after neoadjuvant/adjuvant therapy, they will need to receive one line of therapy for recurrent disease to be eligible * If patients receive one regimen in which a chemotherapy agent is dropped for toxicity without progression, this treatment will be considered one line of therapy; however, substitution or addition of a new agent will be considered a second line of therapy * Eastern Cooperative Oncology Group (ECOG) performance status 0-2 * Leukocytes \> 3,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Hemoglobin \>= 9 g/dL * Platelets \>= 100,000/mcL * Total bilirubin =\< institutional upper limit of normal (ULN) * Aspartate transaminase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine transaminase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 X institutional ULN * Creatinine =\< 1.5 X institutional ULN or creatinine clearance \>= 60 mL/min/1.73m\^2 for patients with creatinine levels above institutional normal * Patients must have fasting serum glucose =\< 160 mg/dL (8.8 mmol/L) or =\< ULN, and hemoglobin A1C =\< 7% (0.07 International System of Units \[SI units\]) within 14 days of registration; if baseline nonfasting glucose =\< 160 mg/dL (8.8 mmol/L), fasting glucose measurement is not required * Registration no fewer than 28 days from last chemotherapy * A "currently active" second malignancy other than non-melanoma skin cancers are not to be registered; patients are not considered to have a "currently active" malignancy if they have completed therapy and are considered by their physician to be at less than 30% risk of relapse Exclusion Criteria: * Patients have received prior taxane or anti-insulin growth factor receptor (IGFR) therapy * Patients must not have any of the following conditions: * Poorly controlled diabetes mellitus; patients with a history of diabetes mellitus are allowed to participate, provided that their blood glucose is within normal range (fasting glucose =\< 160 mg/dL \[8.8 mmol/L\] or below the ULN and hemoglobin A1C =\< 7% \[0.07 SI units\]) and that they are on a stable dietary or therapeutic regimen for this condition * Recent major surgery, hormonal therapy (other than replacement) or chemotherapy, within 4 weeks prior to entering the study or those who have not recovered from adverse events * History of allergic reactions attributed to compounds of similar chemical or biologic composition to cixutumumab (IMC-A12) * Psychiatric illness that would prevent the patient from giving informed consent * Medical conditions such as active/uncontrolled infection (including HIV) or cardiac disease that would make this protocol unreasonably hazardous for the patient in the opinion of the treating physician; cardiac disease may include uncontrolled high blood pressure, unstable angina, or serious uncontrolled cardiac arrhythmia