Entinostat, Nivolumab, and Ipilimumab in Treating Patients With Solid Tumors That Are Metastatic or Cannot Be Removed by Surgery or Locally Advanced or Metastatic HER2-Negative Breast Cancer

Inclusion Criteria: * Dose escalation: patients must have histologically or cytologically confirmed solid tumor malignancy that is metastatic or unresectable and for whom either standard curative or palliative measures do not exist or are no longer effective, or for whom anti-PD-L1/cytotoxic T-lymphocyte antigen (CTLA)-4 is appropriate * Dose expansion: patients must have histologically or cytologically confirmed invasive adenocarcinoma of the breast (human epidermal growth factor receptor 2 \[HER2\]-negative) that is locally advanced/metastatic and has progressed despite standard therapy; at least 1 prior chemotherapy regimen in the metastatic setting, and two lines of hormonal therapy (administered in the adjuvant or metastatic setting) for patients with hormone receptor-positive disease; NOTE: HER2-negativity will be defined per American Society of Clinical Oncology (ASCO)-College of American Pathologists (CAP) guidelines; patients whose tumors have HER2 immunohistochemistry (IHC) 3+, in situ hybridization (ISH) \>= 2.0, or average HER2 copy number \>= 6.0 signals per cell are not eligible * Age ≥ 18 years. * NOTE: Because no dosing or adverse event data are currently available on the use of these agents in patients \<18 years of age, children are excluded from this study, but will be eligible for future pediatric trials * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 (Karnofsky \>= 70%) * Life expectancy of greater than 12 weeks * Hemoglobin (HgB) \>= 9.0 g/dL * Leukocytes \>= 3,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN); an exception to this may be allowed for participants with Gilbert's syndrome with documented approval by the protocol chair * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional ULN * Creatinine within normal institutional limits OR creatinine clearance \>= 60 mL/min using the modified Cockcroft-Gault formula * Negative (serum or urine) pregnancy test, for women of childbearing potential only * Patients must have measurable or evaluable/non-measurable disease per Response Evaluation Criteria in Solid Tumors (RECIST) 1.1; patients with bone only disease are not eligible due to difficulties in obtaining serial bone biopsies for correlative analyses; NOTE: for patients with metastatic disease in the liver, tumor burden should not be deemed significant (e.g., to no more than 30% of total liver volume as assessed by local review/investigator) * Adequate pulmonary function as assessed by oxygen saturation \>= 90% when ambulating and not requiring supplemental oxygen * Patient must have an accessible non-bone tumor lesion from which serial core biopsy specimen can be obtained; NOTE: if baseline biopsy is attempted and is unsuccessful (e.g., patient intolerance, inadequate tissue), the patient will still be considered eligible for the study; if core biopsy is not possible, other methods may be approved in advance by the protocol chair/designee * The effects of nivolumab, ipilimumab and entinostat on the developing human fetus are unknown; for this reason and because other therapeutic agents used in this trial are known to be teratogenic, women of child-bearing potential (WOCBP) and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; WOCBP must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) at baseline; women must not be breastfeeding; women who are not of childbearing potential (i.e., who are postmenopausal or surgically sterile as well as azoospermic men) do not require contraception; NOTE: A WOCBP is defined as any female who has experienced menarche and who has not undergone surgical sterilization (hysterectomy or bilateral oophorectomy) or who is not postmenopausal; menopause is defined clinically as 12 months of amenorrhea in a woman over 45 in the absence of other biological or physiological causes * Willingness to provide tissue and blood samples for mandatory translational research * Willingness to return to the enrolling institution for follow-up * Ability to understand and the willingness to sign a written informed consent document Exclusion Criteria: * Patients who have had chemotherapy or other systemic therapy or radiotherapy, or those who have not recovered from adverse events due to prior administered agents as follows: * Chemotherapy \< 3 weeks prior to registration * Hormone therapy \< 2 weeks prior to registration * Targeted therapy (other than below) \< 2 weeks prior to registration (e.g., tyrosine kinase inhibitors) * Trastuzumab \< 6 weeks prior to registration * Bevacizumab \< 6 weeks prior to registration * Nitrosoureas/mitomycin C \< 6 weeks prior to registration * Radiotherapy \< 3 weeks prior to registration (NOTE: a previously irradiated lesion may not be used as a target lesion unless there is evidence of post-radiation progression) * Surgery \< 3 weeks prior to registration * Other approved or investigational agents \< 3 weeks prior to registration unless otherwise noted by the protocol chair * Patients who have received prior epigenetic (e.g., histone deacetylase \[HDAC\] inhibitors such as entinostat, panobinostat, vorinostat, romidepsin or demethylating agents such as 5-azacitidine or decitabine) immunomodulatory or other checkpoint inhibitors should only be considered after discussion with the protocol chair * Those who have not recovered from acute adverse events to grade \< 2 or baseline due to agents administered, with exception of alopecia, unless approved by the protocol chair * Known sensitivity to or history of allergic reactions attributed to compounds of similar chemical or biologic composition entinostat, nivolumab, or ipilimumab; history of severe hypersensitivity reaction to any monoclonal antibody * Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including systemic corticosteroids, should be excluded; these include but are not limited to patients with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome, myasthenia gravis, systemic autoimmune disease such as systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, autoimmune hepatitis, and patients with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or phospholipid syndrome should be excluded; patients with vitiligo, endocrine deficiencies including thyroiditis managed with replacement hormones including physiologic corticosteroids are eligible; patients with rheumatoid arthritis and other arthropathies, Sjogren's syndrome and psoriasis controlled with topical medication and patients with positive serology, such as antinuclear antibodies (ANA), anti-thyroid antibodies should be evaluated for the presence of target organ involvement and potential need for systemic treatment but should otherwise be eligible * NOTE: patients are permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger (precipitating event) * Subjects with interstitial lung disease that is symptomatic or may interfere with the detection or management of suspected drug-related pulmonary toxicity; NOTE: subjects must have baseline oxygen/saturation level requirements as above * Patients with active or untreated brain metastases or leptomeningeal metastases are excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events; NOTE: Patients with previously treated brain metastases must have stable neurologic status and magnetic resonance imaging (MRI) imaging following local therapy (surgery or radiation) for at least 4 weeks, with no requirement for immunosuppressive doses of systemic corticosteroids (\> 10 mg/day prednisone equivalents) for at least 2 weeks prior to study drug administration (stable low dose dexamethasone allowed at discretion of protocol chair) * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that in the judgment of the investigator would limit compliance with study requirements * Pregnant women are excluded from this study because the agents have the potential for teratogenic or abortifacient effects; because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother, breastfeeding should be discontinued * Known human immunodeficiency virus (HIV)-positive patients on combination antiretroviral therapy are ineligible; patients who have a positive test for hepatitis B virus surface antigen (HBV sAg) or hepatitis C antibody (HCV Ab)/ribonucleic acid (RNA) (HCV RNA) indicating acute or chronic infection are also ineligible (baseline testing required); these are because of the potential for pharmacokinetic interactions with the study agents; in addition, these patients are at increased risk of lethal infections when treated with marrow-suppressive therapy; appropriate studies will be undertaken in patients receiving combination antiretroviral therapy when indicated * Patients should be excluded if they have a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration; inhaled or topical steroids and adrenal replacement doses =\< 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease; patients are permitted to use topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption); physiologic replacement doses of systemic corticosteroids are permitted, even if =\< 10 mg/day prednisone equivalents; a brief course of corticosteroids for prophylaxis (e.g., contrast dye allergy) or for treatment of non-autoimmune conditions (e.g., delayed-type hypersensitivity reaction caused by contact allergen) is permitted * Patients requiring concurrent administration of valproic acid are also excluded * Patients who have had evidence of active or acute diverticulitis, intra-abdominal abscess, gastrointestinal (GI) obstruction and abdominal carcinomatosis which are known risk factors for bowel perforation should be evaluated for the potential need for additional treatment before coming on study * Another active malignancy =\< 3 years prior to registration with the exception of non-melanotic skin cancer or carcinoma-in-situ of any type; NOTE: if there is a history of prior malignancy, they must not be receiving other specific treatment for their cancer; any malignancy considered to be indolent and that has never required therapy may allowed at the discretion of the protocol chair