Study to Test the Combination of Fulvestrant With Lu-DOTATATE for Advanced Pancreatic Neuroendocrine Tumors

Inclusion Criteria: * Patients must have histologically confirmed pancreatic neuroendocrine tumor, WHO grades 1-2, Ki-67 index of \&lt;20%. * Patients must have locally advanced disease that is ineligible for curative-intent resection, or metastatic disease. * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as ≥20 mm with conventional techniques or as ≥10 mm with spiral CT scan, MRI, or calipers by clinical exam. * Patients must also have tumors expressing somatostatin receptor, defined as radiotracer avid lesions as assessed by a previous DOTATATE PET scan. * Patients must have radiographically progressed (as per RECIST v1.1 criteria) on one or more prior-lines of systemic therapy prior to enrollment in the trial. Prior systemic therapies include but are not limited to somatostatin analogs (octreotide LAR, lanreotide), Capecitabine/Temozolomide, tyrosine kinase inhibitors (e.g. everolimus), VEGF pathway inhibitors (e.g. sunitinib, cabozantinib), and/or other systemic chemotherapy. * Patients should not have received any prior systemic therapy with peptide receptor radionuclide therapy (including 177Lu-DOTATATE) and/or Fulvestrant. * Prior treatment with hepatic intra-arterial embolic therapies is allowed if there is recovery from all toxicities, measurable lesions do not include embolized liver unless there has been clear subsequent progression, all measurable lesions are somatostatin receptor avid, and treatment completed at least 2 months prior to registration. * Prior treatment with cryoablation or thermal/radiofrequency ablation of metastases is allowed if there is recovery from all toxicities, measurable lesions do not include treated metastases, and treatment completed at least 2 months prior to registration. * Concomitant Medication: Concurrent SSA use while on protocol therapy is allowed provided that the patient: 1) has a functional tumor (evidence of peptide hormones and/or bioactive substances associated with a clinical hormone syndrome such as carcinoid syndrome or Cushing's syndrome), 2) has been on a stable dose of somatostatin analog therapy for at least three months, and 3) has previously demonstrated radiographic disease progression while on SSA therapy. There should be a minimum of 28 days between long-acting SSA and 177Lu-DOTATATE dosing. Short-acting SSAs should not be administered within 24 hours of 177Lu-DOTATATE dosing. Following lutetium 177Lu-DOTATATE dosing, long-acting SSAs may be administered between 4 and 24 hours after each dose. * Age ≥18 years. As neuroendocrine tumors are very rare in children, patients \<18 years of age will be excluded from this study. Furthermore, no reliable dosing or adverse event data are currently available on the use of Fulvestrant patients \<18 years of age. * ECOG performance status of 0 or 1. * Patients must have adequate organ and marrow function as defined below: 1. Absolute Neutrophil Count ≥1,500/µL 2. Platelet Count ≥100,000/µL 3. Hemoglobin ≥ 9g/dL 4. Creatinine \<1.5 mg/dL * -OR- 1. Creatinine Clearance ≥ 50 mL/min (Cockroft- Gault calculated) 2. Total bilirubin ≤ 1.5 × institutional upper limit of normal 3. AST AND ALT ≤ 3 × institutional upper limit of normal 4. Serum Albumin ≥ 3.0 g/dL * Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial. * For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated. * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load. * Patients should not have any other active malignancies at the time of study enrollment or a history of myelodysplastic syndromes/acute myeloid leukemia accounting for the rare but serious bone-marrow toxicity of 177Lu-DOTATATE. Patients with a remote history of other malignancies may be eligible for enrollment if the malignancy was curable, they have completed all curative-intent treatment (such as surgery, radiation and adjuvant therapy if warranted), and the malignancy has not recurred after 3 years of initial diagnosis. * Patients with treated brain metastases are eligible if follow-up brain imaging after central nervous system (CNS)-directed therapy shows no evidence of progression for at least 3 months. * Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should have a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, patients should be class 2B or better. * Not pregnant and not nursing since the effects of 177Lu-DOTATATE on the developing human fetus are not well-established. For women of childbearing potential only, a negative pregnancy test done ≤ 28 days prior to registration is required. * Patients must have a life expectancy of \>12 weeks. * Ability to understand and the willingness to sign a written informed consent document. Exclusion Criteria: * Patients who are unable to provide or understand written informed consent and comply with scheduled visits, drug administration plan, laboratory tests, or other study procedures and study restrictions. * Patients who are receiving any other investigational agents. * Patients with a "currently active" second malignancy other than non-melanoma skin cancers. Patients are not considered to have a "currently active" malignancy if they have completed therapy and are free of disease for ≥ 3 years. * Pregnant or breastfeeding. * Clinical or laboratory signs of imminent organ failure, as determined by the treating investigator. * Patients with known untreated brain metastases should be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events.