Nivolumab and Ipilimumab in Treating Patients With Metastatic/Recurrent ACC of All Sites and Non-ACC Salivary Gland Cancer

Inclusion Criteria: * Patients must have histologically or cytologically confirmed metastatic/recurrent adenoid cystic carcinoma (ACC) or non-adenoid cystic carcinomas (non-ACC) of major or minor salivary glands * Patients must have evidence of disease progression and cannot be a candidate for surgical treatment * NOTE: Disease progression is defined as one of the following occurring within the 6 months prior to study entry: * At least a 20% increase in radiologically or clinically measurable lesions * Appearance of any new lesions or * Symptomatic and/or deterioration in clinical status * Patients must have received at least one prior line of systemic therapy * NOTE: There is no limit to the number of prior therapies for stage IV disease * NOTE: Patients should not be a candidate for surgical treatment * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension in accordance with RECIST criteria v1.1 * Patients must exhibit an Eastern Cooperative Oncology Group (ECOG) status of 0-2 * NOTE: ECOG performance status 3 will be allowed only if thought to be directly secondary to adenoid cystic carcinoma disease by treating physician * Patients must have adequate organ and bone marrow function within 14 days prior to registration, as defined by: leukocytes \>= 2,000/mcL * Patients must have adequate organ and bone marrow function within 14 days prior to registration, as defined by: absolute neutrophil count \>= 1,500/mcL, regardless of transfusion or growth factor support * Patients must have adequate organ and bone marrow function within 14 days prior to registration, as defined by: platelets \>= 100,000/mcl, regardless of transfusion or growth factor support * Patients must have adequate organ and bone marrow function within 14 days prior to registration, as defined by: total bilirubin total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (except patients with Gilbert syndrome or liver metastasis, who can have total bilirubin \< 3.0 x ULN) * Patients must have adequate organ and bone marrow function within 14 days prior to registration, as defined by: aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 X institutional upper limit of normal (ULN) (or =\< 5 times ULN in case of liver metastasis) * Patients must have adequate organ and bone marrow function within 14 days prior to registration, as defined by: serum creatinine of \< 3.0 X ULN (upper limit of normal) or creatinine clearance \> 30 mL/minute (using Cockcroft/Gault formula) * Patients with history of central nervous system (CNS) metastases are eligible if CNS disease has been stable for at least 6 weeks prior to study registration in the opinion of the investigator and does not require corticosteroids (of any dose) for symptomatic management * NOTE: Only patients with a known history or indication of CNS disease are required to have CNS imaging prior to study entry * Females of childbearing potential (FOCBP) must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of HCG) within 72 hours of registration * NOTE: A FOCBP is any woman (regardless of sexual orientation, having undergone a tubal ligation, or remaining celibate by choice) who meets the following criteria: * Has not undergone a hysterectomy or bilateral oophorectomy * Has had menses at any time in the preceding 12 consecutive months (and therefore has not been naturally postmenopausal for \> 12 months) * FOCBP and men who are sexually active with FOCBP must agree to follow instructions for method(s) of contraception for the duration of treatment and the designated post-treatment period * Patients must have the ability to understand and the willingness to sign a written informed consent prior to registration on study Exclusion Criteria: * Patients must not have had chemotherapy or radiotherapy =\< 28 days prior to study registration * Patients who have not recovered to =\< grade 1 or tolerable grade 2 from adverse events due to agents administered \>= 28 days earlier are not eligible * Patient must not be a candidate for surgical treatment or radiation * Patients may not be receiving any other investigational agents =\< 28 days prior to registration * Patients who have had prior exposure to immune checkpoint inhibitors are not eligible; please contact principal investigator, 312-926-4248 for specific questions on potential interactions * NOTE: Immune checkpoint inhibitors working through OX40 are an exception (for example, MEDI6383, MEDI6469, MEDI0562, oxelumab, and PF-04518600) and are permitted \>= 28 days prior to study registration * Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including chronic prolonged systemic corticosteroids (defined as corticosteroid use of duration one month or greater), should be excluded; these include but are not limited to patients with a history of: * Immune related neurologic disease * Multiple sclerosis * Autoimmune (demyelinating) neuropathy * Guillain-Barre syndrome * Myasthenia gravis * Systemic autoimmune disease such as SLE * Connective tissue diseases * Scleroderma * Inflammatory bowel disease (IBD) * Crohn's * Ulcerative colitis * Patients with a history of toxic epidermal necrolysis (TEN) * Stevens-Johnson syndrome * Anti-phospholipid syndrome * NOTE: Subjects with vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll * Patients who have an uncontrolled intercurrent illness including, but not limited to any of the following, are not eligible: * Ongoing or active infection (including minor localized infections) requiring oral or IV treatment * Symptomatic class 3 or 4 congestive heart failure, defined as a clinical syndrome resulting from any structural or functional cardiac disorder that impairs the ability of the ventricle to fill with or eject blood * Unstable angina pectoris * Cardiac arrhythmia * Psychiatric illness/social situations that would limit compliance with study requirements * Any other illness or condition that the treating investigator feels would interfere with study compliance or would compromise the patient's safety or study endpoints * Patients should not have any condition requiring systemic treatment with corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days prior to first dose of study drug * NOTE: Inhaled or topical steroids and adrenal replacement steroids at any dose are permitted in the absence of active autoimmune disease; a brief (less than 3 weeks) course of corticosteroids for prophylaxis (e.g., contrast dye allergy) or for treatment of non-autoimmune conditions (e.g., delayed-type hypersensitivity reaction caused by a contact allergen) is permitted * Female patients who are pregnant or nursing are not eligible * No other prior malignancy is allowed except for the following: * Adequately treated basal cell or squamous cell skin cancer, * In situ cervical cancer, * Or any other cancer from which the patient has been disease free for at least three years * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS) is not permitted * Any known positive test for hepatitis B or hepatitis C virus indicating acute or chronic infection is not permitted * Patients who received a live, attenuated vaccine =\< 30 days before study registration or are anticipated to require such a live attenuated vaccine are not eligible * NOTE: Influenza vaccination should be given during influenza season only (approximately October to March); patients must not receive live, attenuated influenza vaccine (e.g., FluMist) =\< 30 days prior to study registration or at any time during the study