MEDI4736 and Tremelimumab in Treating Patients With Metastatic HER2 Negative Breast Cancer

Inclusion Criteria: * Patients must have a histologically documented (either primary or metastatic site) diagnosis of breast cancer that is HER2 non-overexpressing by immunohistochemistry, namely 0 or 1; if they have an equivocal immunohistochemistry, 2, the tumor must be non-gene amplified by fluorescence in situ hybridization (FISH) performed upon the primary tumor or metastatic lesion (ratio \< 2 and HER2 copy number \< 4); estrogen receptor (ER) positivity is defined as 1% or greater * Patients must have measurable disease by Response Evaluation Criteria in Solid Tumors (RECIST) criteria * Patients who are ER negative must have progressed through at least one prior chemotherapy regimen in the metastatic setting or within 12 months of their last adjuvant systemic treatment; patients who are ER positive must have progressed through standard hormone therapy options and have received at least one line of chemotherapy in the metastatic setting * Completion of prior chemotherapy systemic anticancer therapy at least 2 weeks prior to study entry * Radiation therapy must be completed at least 2 weeks prior to study entry; radiated lesions may not serve as measurable disease unless they have been radiated over 12 months prior to enrollment * Patients may have parenchymal brain metastases if stable (no evidence of progression) for at least 1 month after local therapy (radiation or surgery); leptomeningeal disease is excluded; must have completed any prescribed steroid taper * Patients may have had a prior diagnosis of cancer if it has been \> 5 years since their last treatment * Patients must exhibit an Eastern Cooperative Oncology Group (ECOG) performance status of =\< 2 * Absolute neutrophil count \>= 1,000/mcL * Platelets \>= 50,000/mcl * Total bilirubin =\< 1.5 times the institutional upper limit of normal (ULN) (or =\< 3 times ULN in case of liver metastasis) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SPGT\]) =\< 2.5 X institutional ULN (or =\< 5 times ULN in case of liver metastasis) * Creatinine =\< 2 ng/ml * Females of child-bearing potential (FOCBP) and males must agree to use 2 methods of adequate contraception prior to study entry, for the duration of study participation, and for number (#) days following completion of therapy; should a female patient become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately * NOTE: A FOCBP is any woman (regardless of sexual orientation, having undergone a tubal ligation, or remaining celibate by choice) who meets the following criteria: * Has not undergone a hysterectomy or bilateral oophorectomy * Has had menses at any time in the preceding 12 consecutive months (and therefore has not been naturally postmenopausal for \> 12 months) * FOCBP must have a negative pregnancy test within 7 days prior to registration on study * Willingness to provide a fresh biopsy prior to study enrollment and after 2 cycles of treatment * Patients must have the ability to understand and the willingness to sign a written informed consent prior to registration on study Exclusion Criteria: * Patients who have had chemotherapy or radiotherapy within 2 weeks prior to entering the study or those who have not recovered from adverse events due to agents administered more than 4 weeks earlier are not eligible. * Current or prior use of immunosuppressive therapy within 2 weeks of starting investigational therapy * Patients who are taking any herbal (alternative) medicines are NOT eligible for participation; patients must be off any such medications by the time of registration for at least 2 weeks; NOTE: Vitamin supplements are acceptable * Patients may not have received any other investigational agents within 4 weeks prior to registration * Prior treatment with immune therapy (including but not limited to cluster of differentiation \[CD\]137, OX40, programmed death \[PD\]-1, PD-L1 or cytotoxic T-lymphocyte antigen 4 \[CTLA4\] inhibitors) * Prior severe infusion reaction to a monoclonal antibody * Patients with a history of or active autoimmune disease within the past 3 years with the following exceptions: * Vitiligo or alopecia * Hypothyroidism on stable doses of thyroid replacement therapy * Psoriasis not requiring systemic therapy within the past 3 years * History of primary immunodeficiency disease or tuberculosis * Major medical conditions that might affect study participation (uncontrolled pulmonary, renal, or hepatic dysfunction, uncontrolled infection) are not eligible; other significant comorbid condition which the investigator feels might compromise effective and safe participation in the study * Patients who have an uncontrolled intercurrent illness including, but not limited to any of the following, are not eligible: * Uncontrolled pulmonary, renal, or hepatic dysfunction * Ongoing or active infection requiring systemic treatment * Known active or chronic viral hepatitis or human immunodeficiency virus (HIV) * Psychiatric illness/social situations that would limit compliance with study requirements * Any other illness or condition that the treating investigator feels would interfere with study compliance or would compromise the patient's safety or study endpoints * Female patients who are pregnant or nursing are not eligible