Axitinib and Avelumab in Treating Patients With Recurrent or Metastatic Adenoid Cystic Carcinoma

Inclusion Criteria: * Eastern Cooperative Oncology Group ECOG performance status 0 or 1 * Histologically confirmed recurrent or metastatic adenoid cystic carcinoma not amenable to curative intent surgery or radiotherapy * Measurable disease per RECIST 1.1 * Evidence of disease progression within 6 months of study enrollment or worsening disease-related symptoms * Previously untreated subjects and subject treated with any number of prior lines of therapy are eligible * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L * Platelet count \>= 100 x 10\^9/L * Hemoglobin \>= 9 g/dL (may have been transfused) * Total bilirubin level =\< 1.5 x the upper limit of normal (ULN) range * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) levels =\< 2.5 x ULN or AST and ALT levels =\< 5 x ULN (for subjects with documented metastatic disease to the liver) * Estimated creatinine clearance \>= 30 mL/min according to the Cockcroft-Gault formula (or local institutional standard method) * Must have archival tissue (formalin-fixed, paraffin-embedded \[FFPE\] tissue available-minimum of 15 unstained slides) or be willing to undergo a biopsy * For patients receiving anti-therapeutic coagulation, patients must be on stable anticoagulant regimen and international normalized ratio (INR) or activated partial thromboplastin time (aPTT) must be =\< 1.5 upper limit of normal * Females of childbearing potential must not be breast feeding and must have a negative serum or urine pregnancy test and must agree to use highly effective contraception for a minimum of two weeks prior to receiving study medication until 30 days after discontinuation of the study medication. Acceptable methods of contraception include total and true sexual abstinence, hormonal contraceptives that are not prone to drug-drug interactions (intra uterine system \[IUS\] levonorgestrel intra uterine system \[Mirena\], medroxyprogesterone injections \[Depo-Provera\]), copper-banded intra-uterine devices, and vasectomized partner. All hormonal methods of contraception should be used in combination with the use of a condom by their sexual male partner. Females of childbearing potential are defined as those who are not surgically sterile (i.e., bilateral tubal ligation, bilateral oophorectomy, or complete hysterectomy) or postmenopausal (defined as 12 months with no menses without an alternative medical cause) * Women will be considered post-menopausal if they have been amenorrheic for the past 12 months without an alternative medical cause. The following age-specific requirements must also apply: Women \< 50 years old: they would be considered post-menopausal if they have been amenorrheic for the past 12 months or more following cessation of exogenous hormonal treatments. The levels of luteinizing hormone (LH) and follicle-stimulating Hormone (FSH) must also be in the post menopausal range (as per the institution). Women \>= 50 years old: they would be considered post-menopausal if they have been amenorrheic for the past 12 months or more following cessation of all exogenous hormonal treatments, or have had radiation-induced oophorectomy with the last menses \> 1 year ago, or have had chemotherapy-induced menopause with \> 1 year interval since last menses, or have had surgical sterilization by either bilateral oophorectomy or hysterectomy * Non-sterilized males who are sexually active with a female partner of childbearing potential must use adequate contraception for the duration of the study and 30 days after the last dose of study medication. Adequate contraception methods include: birth control pills (e.g. combined oral contraceptive pill), barrier protection (e.g. condom plus spermicide, cervical/vault cap or intrauterine device), and abstinence. Patients should not father a child for 6 months after completion of the study medication. Patients should refrain from donating sperm from the start of dosing until 6 months after discontinuing the study medication. If male patients wish to father children they should be advised to arrange for freezing of sperm samples prior to the start of the study medication * For patients with hypertension, upon entry into study must have blood pressure of \< 140/90 * Corrected QT interval (QTc) \< 470 msec Exclusion Criteria: * Current use of immunosuppressive medication, EXCEPT for the following: * Intranasal, inhaled, topical steroids, or local steroid injection (e.g., intra-articular injection) * Systemic corticosteroids at physiologic doses =\< 10 mg/day of prednisone or equivalent * Steroids as premedication for hypersensitivity reactions (e.g., computed tomography \[CT\] scan premedication) * Active autoimmune disease that might deteriorate when receiving an immuno-stimulatory agent. Patients with diabetes type I, vitiligo, psoriasis, or hypo- or hyperthyroid diseases not requiring immunosuppressive treatment are eligible * Prior organ transplantation including allogenic stem-cell transplantation * Active infection requiring systemic therapy * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome * Hepatitis B virus (HBV) or hepatitis C virus (HCV) infection at screening (positive HBV surface antigen or HCV ribonucleic acid \[RNA\] if anti-HCV antibody screening test positive) * Vaccination within 4 weeks of the first dose of avelumab and while on trials is prohibited except for administration of inactivated vaccines * Known prior severe hypersensitivity to investigational product or any component in its formulations, including known severe hypersensitivity reactions to monoclonal antibodies (National Cancer Institute \[NCI\] Common Terminology Criteria for Adverse Events \[CTCAE\] version \[v\]4.03 grade \>= 3) * Clinically significant (i.e., active) cardiovascular disease: cerebral vascular accident/stroke (\< 6 months prior to enrollment), myocardial infarction (\< 6 months prior to enrollment), unstable angina, congestive heart failure (\>= New York Heart Association Classification class II), or serious cardiac arrhythmia requiring medication * Persisting toxicity related to prior therapy (NCI CTCAE v. 4.03 grade \> 1); however, alopecia, sensory neuropathy grade =\< 2, or other grade =\< 2 not constituting a safety risk based on investigator's judgment are acceptable * Inadequately controlled hypertension (defined as systolic blood pressure \> 140 mmHg and/or diastolic blood pressure \> 90 mmHg). Anti-hypertensive therapy to maintain a systolic blood pressure \< 140 mmHg and/or diastolic blood pressure \< 90 mmHg is permitted * Prior history of hypertensive crisis or hypertensive encephalopathy * Patients with a baseline electrocardiography (EKG) demonstrating a QTc \> 470 ms * Serious non-healing or dehiscing wound, active ulcer or untreated bone fracture * Proteinuria as demonstrated by urine dipstick or \> 1 g of protein in a 24 hour urine collection. All patients with \>= 2+ protein on dipstick urinalysis at baseline must undergo a 24 hour urine collection for protein * Evidence of bleeding diathesis or clinically significant coagulopathy (in the absence of therapeutic anticoagulation) * Other severe acute or chronic medical conditions including immune colitis, inflammatory bowel disease, immune pneumonitis, pulmonary fibrosis or psychiatric conditions including recent (within the past year) or active suicidal ideation or behavior; or laboratory abnormalities that may increase the risk associated with study participation or study treatment administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for entry into this study * Subject with an uncontrolled seizure disorder, active neurologic disease, or active central nervous system (CNS) involvement except for individuals who have previously-treated CNS metastases, are asymptomatic, and have no requirement for doses of corticosteroids (indicated to reduce brain edema) higher than the equivalent of 10 mg of oral prednisone a day or anti-seizure medication for at least 2 weeks prior to first dose of study drug * History of ongoing malignancies or malignancies in remission \< 2 years. Adequately curative intent treated initial stage non-melanoma skin cancers; in situ carcinoma of the cervix; breast carcinoma in situ; low-grade local bladder cancer; and low-risk prostate cancer undergoing active surveillance will be allowed * Pregnant women are excluded from this study. Based on its mechanism of action. Avelumab can cause fetal harm when administered to a pregnant woman. In animal models, the PD-1/PD-L1 signaling pathway is important in the maintenance of pregnancy through induction of maternal immune tolerance to fetal tissue. Human IgG1 immunoglobulins are known to cross the placenta. Therefore, avelumab has the potential to be transmitted from the mother to the developing fetus. Blockade of PD-L1 signaling has been shown in murine models of pregnancy to disrupt tolerance to the fetus and to result in an increase in fetal loss. Therefore, potential risks of administering avelumab during pregnancy include increased rates of abortion or stillbirth. Advise females of reproductive potential to use effective contraception during treatment with avelumab and for at least one month after the last dose of avelumab * Lactating females: There is no information regarding the presence of avelumab in human milk, the effects on the breastfed infant, or the effects on milk production. Since many drugs including antibodies are excreted in human milk, advise a lactating woman not to breastfeed during treatment and for at least one month after the last dose of avelumab due to the potential for serious adverse reactions in breastfed infants * Prior treatment with immune checkpoint inhibitor (e.g. anti-PD-1/PD-L1) * Prior treatment with VEGF or VEGFR inhibitors (e.g. lenvatinib, bevacizumab)