AMG386 Comb w. Either Pegylated Liposomal Doxorubicin or Topotecan Subjects w. Advanced Recurrent Epithelial Ovarian CR

Inclusion Criteria: * Histologically or cytologically documented recurrent invasive epithelial ovarian, fallopian tube, or primary peritoneal cancer * Subjects must have received at least one platinum containing regimen * Radiographically documented progression per RECIST criteria with modifications or progression of CA 125 as adopted by GCIG during or subsequent to the last chemotherapy regimen * Subjects may include those with measurable or non measurable disease * All scans and x-rays used to document measurable or non measurable disease must be done within 28 days prior to enrollment * Female 18 years of age or older at the time the written informed consent is obtained * GOG Performance Status of 0 or 1 * Left Ventricular Ejection Fraction (LVEF) \>= institutional lower limit of normal for subjects assigned to cohort A only * Adequate organ function as assessed by laboratory studies (hematological and chemistries) * Life expectancy \>= 3 months (per investigator opinion) * Subjects of child bearing potential who have not undergone a bilateral salpingo oophorectomy and are sexually active must consent to use an accepted and effective double barrier non hormonal method of contraception from signing the informed consent through 6 months after last dose of study drug Exclusion Criteria: * Subjects believed to be a higher than average risk of bowel perforation. This includes symptoms of partial or complete bowel obstruction, recent (within 6 months) history of fistula or bowel perforation, subjects requiring total parenteral nutrition and continuous hydration * Previous abdominal /or pelvic external beam radiotherapy * Known history of central nervous system metastases * Subjects with a history of prior malignancy, except: * Malignancy treated with curative intent and with no known active disease present for \>= 3 years before study day 1 and felt to be at low risk for recurrence by treating physician * Adequately treated non melanomatous skin cancer or lentigo maligna without evidence of disease * Adequately treated cervical carcinoma in situ without evidence of disease * Prior myeloablative high dose chemotherapy with allogeneic or autologous stem cell (or bone marrow) transplant * History of arterial or deep venous thromboembolism within 12 months prior to enrollment * Clinically significant cardiac disease within 12 months prior to enrollment * Prior treatment with doxorubicin or pegylated liposomal doxorubicin (cohort A subjects) and topotecan (cohort B subjects) * Current or within 30 days prior to enrollment treatment with immune modulators such as systemic cyclosporine and tacrolimus