Durvalumab and Tremelimumab Before Surgery in Treating Patients With Hormone Receptor Positive, HER2 Negative Stage II-III Breast Cancer

Inclusion Criteria: * Written informed consent and any locally-required authorization (e.g., Health Insurance Portability and Accountability Act \[HIPAA\]) obtained from the subject prior to performing any protocol-related procedures, including screening evaluations * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 * Hormone receptor positive (defined as estrogen receptor \[ER\] and/or progesterone receptor \[PR\] positive), HER2 negative breast cancer that is clinically staged II-III with no known metastatic disease. ER and/or PR defined as positive if expression \> 10% by immunohistochemistry (IHC). HER2 negative or non-amplified as determined by the current American Society of Clinical Oncology-College of American Pathologists (ASCO-CAP) criteria which are as follows: HER2 testing by immunohistochemistry (IHC) as 0 or 1+. If HER2 is 2+, ISH (in situ hybridization) must be performed. HER2 is positive if: IHC 3+ based on circumferential membrane staining that is complete, intense ISH positive based on: 1) Single-probe average HER2 copy number \>= 6.0 signals/cell, 2) Dual-probe HER2/CEP17 ratio \>= 2.0; c,e with an average HER2 copy number \>= 4.0 signals/cell, 3) Dual-probe HER2/CEP17 ratio \>= 2.0; c,e with an average HER2 copy number \< 4.0 signals/cell, 4) Dual-probe HER2/CEP17 ratio \< 2.0; c,e with an average HER2 copy number \>= 6.0 signals/cell * Chemotherapy is planned for the patient in the neoadjuvant setting * Hemoglobin \>= 9.0 g/dL * Absolute neutrophil count (ANC) \>=1.5 x 10\^9/L (\>= 1500 per mm\^3) * Platelet count \>= 100 x 10\^9/L (\>= 100,000 per mm\^3) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN). This will not apply to subjects with confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only upon treating physician, principal investigators (PI) or co-PIs approval * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal unless liver metastases are present, in which case it must be =\< 5 x ULN * Creatinine clearance (CL) \> 40 mL/min by the Cockcroft-Gault formula (Cockcroft and Gault 1976) or by 24-hour urine collection for determination of creatinine clearance * Female subjects must either be of non-reproductive potential (i.e., post-menopausal by history: \>= 60 years old and no menses for \>= 1 year without an alternative medical cause; OR history of hysterectomy, OR history of bilateral tubal ligation, OR history of bilateral oophorectomy) or must have a negative urine pregnancy test upon study entry * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up Exclusion Criteria: * Involvement in the planning and/or conduct of the study (applies to both AstraZeneca staff and/or staff at the study site) * Participation in another clinical study with an investigational product during the last 1 month prior to initiation of therapy * Any previous treatment with a PD1 or PD-L1 inhibitor, including durvalumab or an anti-CTLA4, including tremelimumab * History of another primary malignancy except for: * Malignancy treated with curative intent and with no known active disease \>= 5 years before the first dose of study drug and of low potential risk for recurrence * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease * Adequately treated carcinoma in situ without evidence of disease e.g., cervical cancer in situ * Has received therapy for this current diagnosis of breast cancer including endocrine therapy or chemotherapy * A single QT interval corrected for heart rate (QTc) \>= 470 ms. If an electrocardiogram (ECG) is interpreted to be a prolonged QT interval, 2 additional ECGs will be obtained and the PI will then evaluate all three ECGs and determine whether the patient should be excluded. Mean QT interval corrected for heart rate (QTc) \>= 470 ms calculated from 3 electrocardiograms (ECGs) using Fridericia's correction * Current or prior use of immunosuppressive medication within 28 days before the first dose of durvalumab or tremelimumab, with the exceptions of intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid * Active or prior documented autoimmune disease within the past 2 years * NOTE: Subjects with vitiligo, Graves disease, or psoriasis not requiring systemic treatment (within the past 2 years) are not excluded * Active or prior documented inflammatory bowel disease (e.g., Crohn's disease, ulcerative colitis) * History of primary immunodeficiency * History of allogeneic organ transplant * History of hypersensitivity to durvalumab or tremelimumab or any excipient * History of hypersensitivity to the combination or comparator agent (if applicable) * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, active peptic ulcer disease or gastritis, active bleeding diatheses including any subject known to have evidence of acute or chronic hepatitis B, hepatitis C or human immunodeficiency virus (HIV), or psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent * Known history of previous clinical diagnosis of tuberculosis * History of leptomeningeal carcinomatosis * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab or tremelimumab * Female subjects who are pregnant, breast-feeding or male or female patients of reproductive potential who are not employing an effective method of birth control * Any condition that, in the opinion of the investigator, would interfere with evaluation of study treatment or interpretation of patient safety or study results * Subjects with uncontrolled seizures