Neoadjuvant Endocrine Therapy, Palbociclib, Avelumab in Estrogen Receptor Positive Breast Cancer

Inclusion Criteria: * Stage II-III ER-positive breast cancer * Tumor evaluable either by ultrasound or by touch. * Age ≥ 18 years. * Eastern Cooperative Oncology Group performance status of 1 or less. * Adequate organ and bone marrow function within 28 days prior to registration. * Females of child-bearing potential and males must agree to use adequate contraception prior to study entry, for the duration of study participation, and for 30 days following completion of therapy. * Females of child-bearing potential must have a negative pregnancy test within 7 days prior to registration on study. * Patients must have the ability to understand and the willingness to sign a written informed consent prior to registration on study Exclusion Criteria: * Evidence of metastatic disease or inflammatory breast cancer. * Patients not felt to be sensitive to endocrine therapy, such that a neoadjuvant endocrine-based approach would not be appropriate (i.e. PR-negative, high grade/Ki67, high gene expression profile, clinically aggressive presentation) * Previous treatment with endocrine therapy within the last 10 years (i.e. tamoxifen, aromatase inhibitors). * Previous treatment with CDK4/6 inhibitors, or immune checkpoint inhibitors. * Use of SSRIs and/or any concomitant use of medications outlined in section 6.5 within 4 days prior to enrollment. If patients are on stable doses and there are no good alternatives, the treating physician may discuss with Study Chair. * May not be receiving any other investigational agents. * May not be receiving immunosuppressive therapy within 2 weeks of study entry. * History of allergic reactions attributed to compounds of similar chemical or biologic composition to corresponding endocrine therapy (tamoxifen, aromatase inhibitors, GnRH agonists), palbociclib, and avelumab are not eligible. * May not have had a prior diagnosis of cancer if it has been \< 3 years since their last treatment (with the exception of squamous cell carcinoma or basal cell carcinoma of the skin or cervical intraepithelial neoplasia). NOTE: Patients with a history of breast cancer or breast cancer treatment within the last 10 years are also excluded. Any previous radiation to affected breast is excluded. * Autoimmune disease within the last 3 years with the exception of: Vitiligo or alopecia; Hypothyroidism on stable doses of thyroid medication; and Psoriasis not requiring systemic therapy * Uncontrolled intercurrent illness including, but not limited to any of the following, are not eligible: Ongoing or active infection requiring systemic treatment (including HIV, TB, hepatitis viruses), symptomatic congestive heart failure, cerebral vascular accident/stroke (\< 6 months prior to enrollment), myocardial infarction (\< 6 months prior to enrollment), unstable angina pectoris, cardiac arrhythmia, psychiatric illness/social situations that would limit compliance with study requirements, any other illness or condition that the treating investigator feels would interfere with study compliance or would compromise the patient's safety or study endpoints * Other severe acute or chronic medical conditions including immune colitis, inflammatory bowel disease, immune pneumonitis, pulmonary fibrosis or psychiatric conditions including recent (within the past year) or active suicidal ideation or behavior; or laboratory abnormalities that may increase the risk associated with study participation or study treatment administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for entry into this study * Vaccination within 4 weeks of the first dose of avelumab and while on trials is prohibited except for administration of inactivated vaccines * Prior organ transplantation including allogenic stem-cell transplantation * Female patients who are pregnant or nursing are not eligible.