Palbociclib After CDK and Endocrine Therapy (PACE)

Inclusion Criteria: * Participants must have histologically confirmed hormone receptor positive (HR+) HER2 negative metastatic or locally recurrent unresectable invasive breast cancer. ER, PR and HER2 measurements should be performed according to institutional guidelines, in a CLIA-approved setting. Cut-off values for positive/negative staining should be in accordance with current ASCO/CAP (American Society of Clinical Oncology/College of American Pathologists) guidelines. * Men and pre- and postmenopausal women are eligible. Ongoing monthly GNRH agonist is required in pre-menopausal women or male participants for at least 4 weeks prior to study entry. * Participants must have radiological or objective evidence of progression on an endocrine and CDK 4/6 inhibitor regimen in the metastatic setting, and/or relapse/progression during or within 12 months of completion of an endocrine and CDK4/6 inhibitor regimen in the adjuvant setting. * Participants must have previously been exposed to CDK4/6 inhibitor therapy in combination with endocrine therapy. Exposure to any prior CDK4/6 inhibitor, (including palbociclib, abemaciclib, and ribociclib) is allowed. Patients may have a line of endocrine therapy after combination endocrine and CDK4/6 inhibitor exposure. * Participants must have remained on prior endocrine and CDK4/6 therapy in the metastatic setting without progression for at least 6 months prior to study entry. * It is not mandatory to have a CDK 4/6 inhibitor containing regimen as the most recent treatment. * Participants may have 0-1 prior lines of cytotoxic chemotherapy in the metastatic setting. * Prior endocrine therapy in the metastatic setting may include any aromatase inhibitor (AI) or tamoxifen, but may not include prior fulvestrant. In the metastatic setting, 1-2 prior lines of endocrine therapy are allowed. * Participants may have received radiotherapy for palliative purpose, but must not be experiencing \> grade 1 treatment related toxicities, and must have completed treatment \> 14 days prior to registration. * Age ≥18 years. Because no dosing or adverse event data are currently available on the use of study agents in participants \<18 years of age, children are excluded from this study. * ECOG performance status 0-1 (see Appendix A). * Participants must have normal organ and marrow function as defined below: * Absolute neutrophil count \> 1,500/µL * Platelets \> 100,000/µL * Hemoglobin \> 9g/dL * Total bilirubin \< 1.5 x institutional upper limit of normal (ULN) (participants with documented Gilbert's disease are allowed total bilirubin up to 1.5X ULN) * AST (SGOT)/ALT (SGPT) \< 2.5 x institutional ULN, or ≤ 5 x ULN for subjects with documented metastatic disease to the liver. * Creatinine \< institutional ULN or creatinine clearance \> 60 mL/min/1.73 m2 for subjects with creatinine levels above institutional ULN. * Baseline QTc \< 480 ms * The effects of palbociclib and avelumab on the developing human fetus are unknown. If, for any reason, a woman should become pregnant or suspect that she is pregnant while participating in this study, she should inform her treating physician immediately. Women of childbearing age, women who are made postmenopausal through use of GNRH agonists, and men must agree to use adequate contraception for the duration of protocol treatment and for at least 60 days after the last dose of study medication if the risk of contraception exists. * Adequate contraception is defined as one highly effective non-hormonal form of contraception or two effective forms of non-hormonal contraception by the participant and/or partner. * Highly Effective Non-Hormonal Contraception * Methods of birth control which result in a low failure rate (i.e., less than 1% per year) when used consistently and correctly are considered highly-effective forms of contraception. * The following non-hormonal methods of contraception are acceptable: * True abstinence when this is in line with the preferred and usual lifestyle of the participant. \[Periodic abstinence (e.g., calendar, ovulation, symptothermal post-ovulation methods) and withdrawal are not acceptable methods of contraception\]. * Male sterilization (with appropriate post-vasectomy documentation of the absence of sperm in the ejaculate). For female participants, the vasectomized male partner should be the sole partner. OR -Effective Non-Hormonal Contraception Alternatively two of the following effective forms of contraception may be used instead: * Placement of non-hormonal intrauterine device (IUD) or intrauterine system (IUS). Consideration should be given to the type of device being used, as there is higher failure rates quoted for certain types, e.g., steel or copper wire. * Condom with spermicidal foam/gel/film/cream/suppository. * Occlusive cap (diaphragm or cervical/vault caps) with spermicidal foam/gel/film/cream/suppository. * The use of barrier contraceptives should always be supplemented with the use of spermicide. The following should be noted: * Failure rates indicate that, when used alone, the diaphragm and condom are not highly effective forms of contraception. Therefore, the use of additional spermicides does confer additional theoretical contraceptive protection. * However, spermicides alone are ineffective at preventing pregnancy when the whole ejaculate is spilled. Therefore, spermicides are not a barrier method of contraception and should not be used alone. It should be noted that two forms of effective contraception are required. A double barrier method is acceptable, which is defined as condom and occlusive cap (diaphragm or cervical/vault caps) with spermicidal foam/gel/film/cream /suppository. * Premenopausal women must have a negative serum or urine pregnancy test. Pregnancy testing does not need to be pursued in female participants who are: * Age \> 60 years; or * Age \< 60 with intact uterus and amenorrhea for 12 consecutive months or more AND estrogen (estradiol) levels within postmenopausal range; or * Status-post bilateral oophorectomy, total hysterectomy, or bilateral tubal ligation. * Participant must be able to swallow and retain oral medication. * Ability to understand and the willingness to sign a written informed consent document Exclusion Criteria: * Participants who have had endocrine, chemotherapy, and/or biologic therapy within 14 days prior to entering the study or those who have not recovered from any prior treatment-related toxicities (must recover to no more than grade 1. Alopecia, sensory neuropathy Grade ≤ 2, or other Grade ≤ 2 toxicity not constituting a safety risk based on investigator's judgment are acceptable). * Participants who are receiving concurrent therapy with other investigational agents. * Rapidly progressive, symptomatic, visceral spread of disease placing participant at risk of life-threatening complications in the short term. * Participants with active brain metastases. Stable treated brain metastases are allowed (this includes participants who have documented radiologic stability at least 4 weeks after radiotherapy, and do not require systemic steroids for management of symptoms from CNS metastatic lesions). * Participants who have discontinued prior palbociclib for toxicity, or have needed more than one dose or schedule reduction for toxicity from prior palbociclib therapy. If a participant required a single dose reduction during prior palbociclib therapy and tolerated it well, for example prior dosing at 100 mg qd 3 weeks on 1 week off schedule, than that dose may be selected for this trial. * History of allergic reactions to palbociclib or attributed to compounds of similar chemical or biologic composition to palbociclib. * Known prior severe hypersensitivity to investigational product or any component in its formulations, including known severe hypersensitivity reactions to monoclonal antibodies (NCI CTCAE v4.03 Grade ≥ 3) * Participants receiving any medications or substances that are strong inhibitors or inducers of CYP3A isoenzymes are ineligible. Lists including medications and substances known or with the potential to interact with the CYP3A isoenzymes are provided in Appendix B, and can also be found within Section 5.4. Because the lists of these agents are constantly changing, it is important to regularly consult a frequently-updated list such as http://medicine.iupui.edu/clinpharm/ddis/table.aspx; medical reference texts such as the Physicians' Desk Reference may also provide this information. As part of the enrollment/informed consent procedures, the participant will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the participant is considering a new over-the-counter medicine or herbal product. * Current use of drugs listed in Appendix C that are known to prolong the QT interval (See Appendix C) * Prior organ transplantation including allogenic stem-cell transplantation * Current use of immunosuppressive medication, except for the following: a. intranasal, inhaled, topical steroids, or local steroid injection (e.g., intra-articular injection); b. Systemic corticosteroids at physiologic doses ≤ 10 mg/day of prednisone or equivalent; c. Steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication). * Uncontrolled intercurrent illness including, but not limited to: ongoing or active infection requiring systemic therapy, clinically significant cardiovascular disease including: cerebral vascular accident/stroke (\< 6 months prior to enrollment), myocardial infarction (\< 6 months prior to enrollment), unstable angina, congestive heart failure (≥ New York Heart Association Classification Class II), or serious cardiac arrhythmia requiring medication, uncontrolled diabetes mellitus, or psychiatric illness/social situations that would limit compliance with study requirements. Ability to comply with study requirements is to be assessed by each investigator at the time of screening for study participation. * Active autoimmune disease that might deteriorate when receiving an immuno-stimulatory agent. Participants with diabetes type I, vitiligo, psoriasis, or hypo- or hyperthyroid diseases not requiring immunosuppressive treatment are eligible. * Known history of testing positive for HIV or known acquired immunodeficiency syndrome, or need to receive combination antiretroviral therapy for HIV * Known history of immune-mediated conditions including colitis, inflammatory bowel disease requiring steroid or immunosuppressive therapy, pneumonitis, or pulmonary fibrosis. * Hepatitis B virus (HBV) or hepatitis C virus (HCV) infection at screening (positive HBV surface antigen or HCV RNA if anti-HCV antibody screening test positive) * Live vaccination within 4 weeks of the first dose of avelumab * Pregnant women are excluded from this study because effect of palbociclib and avelumab on a developing fetus is unknown. Breastfeeding should be discontinued prior to entry onto the study. * Individuals with a history of a different malignancy are ineligible except for the following circumstances. Individuals with a history of other malignancies are eligible if they have been disease-free for at least 3 years and are deemed by the investigator to be at low risk for recurrence of that malignancy. Individuals with the following cancers are eligible if diagnosed and treated within the past 5 years: ductal carcinoma in situ of the breast, cervical cancer in situ, and basal cell or squamous cell carcinoma of the skin