Ipilimumab and Nivolumab in Treating Patients With Recurrent Stage IV HER2 Negative Inflammatory Breast Cancer

Inclusion Criteria: * Patients must have histologically or cytologically confirmed stage IV breast carcinoma with a previous clinical diagnosis of IBC based on the presence of inflammatory changes in the involved breast, such as diffuse erythema and edema (peau d'orange), with or without an underlying palpable mass involving the majority of the skin of the breast; pathological evidence of dermal lymphatic invasion should be noted but is not required for diagnosis * Patients must have local or metastatic recurrence of IBC after prior surgery * Patients must have a metastatic tumor negative for HER2; the lack of HER2 overexpression by immunohistochemistry (IHC), is defined as 0 or 1+ where as hyperexpression is defined as 3+; if equivocal IHC, 2+, the tumor must be non-gene amplified by fluorescence in situ hybridization (FISH) performed upon the primary tumor or metastatic lesion (ratio \< 2 and HER2 copy number \< 4) * Patients may have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension in accordance with RECIST criteria v. 1.1 OR non-measurable tumors; NOTE: Non-measurable tumors are small lesions (longest diameter \< 10mm or pathological lymph nodes with \>= 10 to \< 15 mm short axis); bone lesions, leptomeningeal disease, ascites, pleural/pericardial effusions, lymphangitis cutis/pulmonitis, inflammatory breast disease, and abdominal masses (not followed by computed tomography \[CT\] or magnetic resonance imaging \[MRI\]) are considered as non-measurable * Patients must be in consideration for 1st line systemic therapy for recurrent IBC; NOTE: Patients must not have received chemotherapy in the metastatic setting, but adjuvant treatment after surgery is acceptable * Patients must have confirmed availability of archival or freshly biopsied tumor tissue meeting protocol-defined specifications prior to study enrollment * Patients must exhibit an Eastern Cooperative Oncology Group (ECOG) status of 0-1; ECOG performance status 2 and 3 will be allowed only if decline in performance status is thought to be directly secondary to breast cancer disease burden by treating physician * Patients must have adequate organ and bone marrow function within 14 days prior to registration, as defined below: * Leukocytes \>= 2,000/mcL * Absolute neutrophil count \>= 1,500/mcL, regardless of transfusion or growth factor support * Platelets \>= 100,000/mcl, regardless of transfusion or growth factor support * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (except patients with Gilbert syndrome or liver metastasis who can have total bilirubin \< 3.0 x ULN) * Aspartate aminotransferase (AST) serum glutamic-oxaloacetic transaminase (SGOT)/ alanine aminotransferase (ALT) serum glutamate pyruvate transaminase (SGPT) =\< 2.5 x upper limit of normal (ULN) (or =\< 5 times ULN in case of liver metastasis) * Serum creatinine of \< 3.0 x ULN (upper limit of normal) * Patients with history of central nervous system (CNS) metastases are eligible if CNS disease has been stable for at least 6 weeks prior to study registration in the opinion of the investigator and do not require corticosteroids (of any dose) for symptomatic management; NOTE: Patients are not required to have CNS imaging prior to study entry * Females of childbearing potential (FOCBP) must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 72 hours of registration NOTE: A FOCBP is any woman (regardless of sexual orientation, having undergone a tubal ligation, or remaining celibate by choice) who meets the following criteria: * Has not undergone a hysterectomy or bilateral oophorectomy * Has had menses at any time in the preceding 12 consecutive months (and therefore has not been naturally postmenopausal for \> 12 months) * FOCBP and men who are sexually active with FOCBP must agree to follow instructions for method(s) of contraception for the duration of treatment and the designated post-treatment period * Patients must have the ability to understand and the willingness to sign a written informed consent prior to registration on study Exclusion Criteria: * Patients must not have had chemotherapy or radiotherapy within 4 weeks prior to study registration * Patients who already received chemotherapy for recurrent metastatic IBC are not eligible * Patients who have not recovered to =\< grade 1 from adverse events due to agents administered more than 4 weeks earlier are not eligible * Patients may not be receiving any other investigational agents * Patients who have had prior exposure to immune checkpoint inhibitors are not eligible; please contact principal investigator, Ricardo Costaat 312-472-1234 for specific questions on potential interactions * Programmed cell death protein 1 (PD-1) monoclonal antibody: pembrolizumab, pidilizumab, MEDI-0680, anti-PD-1 fusion protein AMP-224 (AMP-224), anti-PD-1 checkpoint inhibitor PF-06801591 (PF-06801591), anti-PD-1 monoclonal antibody BGB-A317 (BGB-A317), anti-PD-1 monoclonal antibody PDR001 (PDR001), anti-PD-1 monoclonal antibody REGN2810 (REGN2810), anti-PD-1 monoclonal antibody SHR-1210 (SHR-1210) * PD-L1 monoclonal antibody: durvalumab, avelumab, anti-PD-L1 monoclonal antibody MDX-1105 (MDX-1105), atezolizumab, zirconium Zr 89-labeled anti-PD-L1 monoclonal antibody MPDL3280A (MPDL3280A) * Cytotoxic T-lymphocyte protein 4 (CTLA4) monoclonal antibody: tremelimumab, abatacept * Tumor necrosis factor receptor superfamily member 4 (OX40): agonistic anti-OX40 monoclonal antibody MEDI6383 (MEDI6383), agonistic anti-OX40 monoclonal antibody MEDI6469 (MEDI6469), anti-OX40 monoclonal antibody MEDI0562 (MEDI0562), oxelumab, anti-OX40 antibody PF-04518600 (PF-04518600) * Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including chronic prolonged systemic corticosteroids (defined as corticosteroid use of duration one month or greater), should be excluded; these include but are not limited to patients with a history of: * Immune related neurologic disease * Multiple sclerosis * Autoimmune (demyelinating) neuropathy * Guillain-Barre syndrome * Myasthenia gravis * Systemic autoimmune disease such as systemic lupus erythematosus (SLE) * Connective tissue diseases * Scleroderma * Inflammatory bowel disease (IBD) * Crohn's * Ulcerative colitis * Patients with a history of toxic epidermal necrolysis (TEN) * Stevens-Johnson syndrome * Anti-phospholipid syndrome * NOTE: Subjects with vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll * Patients who have an uncontrolled intercurrent illness including, but not limited to any of the following, are not eligible: * Ongoing or active infection (including minor localized infections) requiring oral or IV treatment * Symptomatic congestive heart failure, defined as a clinical syndrome resulting from any structural or functional cardiac disorder that impairs the ability of the ventricle to fill with or eject blood * Unstable angina pectoris * Cardiac arrhythmia * Psychiatric illness/social situations that would limit compliance with study requirements * Any other illness or condition that the treating investigator feels would interfere with study compliance or would compromise the patient's safety or study endpoints * Patients should not have any condition requiring systemic treatment with corticosteroids (\< 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days prior to first dose of study drug; NOTE: Inhaled or topical steroids and adrenal replacement steroid doses \< 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease; a brief (less than 3 weeks) course of corticosteroids for prophylaxis (eg, contrast dye allergy) or for treatment of non-autoimmune conditions (eg, delayed-type hypersensitivity reaction caused by a contact allergen) is permitted * Female patients who are pregnant or nursing are not eligible * No other prior malignancy is allowed except for the following: * Adequately treated basal cell or squamous cell skin cancer * In situ cervical cancer * Or any other cancer from which the patient has been disease free for at least three years * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS) is not permitted * Any known positive test for hepatitis B or hepatitis C virus indicating acute or chronic infection is not permitted * Patients who have received a live attenuated vaccine within 30 days are not eligible