Avelumab, Utomilumab, Rituximab, Ibrutinib, and Combination Chemotherapy in Treating Patients With Relapsed or Refractory Diffuse Large B-Cell Lymphoma or Mantle Cell Lymphoma

Inclusion Criteria: * All patients and/or their parents or legal guardians must have the ability to understand and the willingness to sign a written informed consent * Voluntary written informed consent must be obtained before performance of any study-related procedure not part of normal medical care, with the understanding that consent may be withdrawn by the subject at any time without prejudice to future medical care * Weight over 40 kilograms (kg) * Life expectancy of greater than 3 months * Cohort #1: histologically confirmed CD20-positive, relapsed or refractory DLBCL, including de novo and transformed DLBCL (from follicular or marginal zone lymphoma); this includes patients with DLBCL who are found to have small cell infiltration of the bone marrow or other diagnostic material (representing a discordant lymphoma) * Cohort #1: patients must be either refractory to or relapsed after up to 2 lines of prior therapy * Cohort #2: histologic confirmation of relapsed or relapsed/refractory MCL confirmed by presence of cyclin D1 by immunohistochemistry (IHC) or fluorescence in situ hybridization (FISH) * Patients must have measurable disease \> 1.5 cm evidenced by computed tomography (CT) scan of the neck/chest/abdomen (abd)/pelvis or CT/positron emission tomography (PET) scans * Cohort #2: patients must be either refractory to or relapsed after at least 1 line of prior therapy * Eastern Cooperative Oncology Group (ECOG) performance score of 0-2 * Cohort #1: received only frontline CD20-directed immunotherapy with anthracycline- or anthracenedione-based multi-agent chemotherapy for patients with DLBCL; monotherapy rituximab or other CD20-directed immunotherapy prior to frontline chemotherapy, as maintenance therapy, and radiotherapy in a limited field or as a part of the frontline treatment plan are permitted; last treatment dose should be 3 weeks before start of study treatment * Cohort #1: considered eligible for high-dose chemotherapy followed by autologous stem cell transplantation (ASCT) * Cohort #2: patients with MCL with prior allogeneic hematopoietic stem cell transplant, minimum 6 months after transplant, not on immunosuppression, and without prior or active graft versus host disease (GVHD), are allowed * Cohort #2: prior treatment with ibrutinib is allowed; patients should not have received any anti-lymphoma therapy within 3 weeks from start of study treatment, with the exception of ibrutinib * Absolute neutrophil count (ANC) \>= 1,000/mm\^3 (performed within 14 days prior to day 1 of protocol therapy) * Filgrastim can be given prior to enrollment to achieve target ANC \>= 1000/uL * Platelets \>= 50,000/mm\^3 for MCL cohort and platelets \>= 75,000/mm\^3 for DLBCL cohort (performed within 14 days prior to day 1 of protocol therapy) * NOTE: platelet transfusions and packed red blood cell transfusion can be given prior to enrollment to achieve a target platelet (Plt) \>= 50,000/uL for MCL and \>= 75,000/uL for DLBCL and hemoglobin of \>= 8.5 g/dL * Hemoglobin \>= 8.5 g/dL (performed within 14 days prior to day 1 of protocol therapy) * Total bilirubin within 1.5 x the upper limit of normal institutional limits; patients with elevation of unconjugated bilirubin alone, as in Gilbert's disease, are eligible (performed within 14 days prior to day 1 of protocol therapy) * Aspartate aminotransferase (AST) =\< 3 x upper limit of normal (ULN) unless demonstrated lymphoma involvement of the liver (performed within 14 days prior to day 1 of protocol therapy) * Alanine aminotransferase (ALT) =\< 2.5 x ULN unless demonstrated lymphoma involvement of the liver (performed within 14 days prior to day 1 of protocol therapy) * Creatinine clearance \>= 50 mL/min for Cohort #1 (DLBCL) and \>= 30 mL/min for Cohort #2 (MCL) per the Cockcroft-Gault formula (performed within 14 days prior to day 1 of protocol therapy) * If not receiving anticoagulants: international normalized ratio (INR) OR prothrombin (PT) =\< 1.5 x ULN (performed within 14 days prior to day 1 of protocol therapy) * If on anticoagulant therapy: PT must be within therapeutic range of intended use of anticoagulants * If not receiving anticoagulants: activated partial thromboplastin time (aPTT) =\< 1.5 x ULN (performed within 14 days prior to day 1 of protocol therapy) * If on anticoagulant therapy: aPTT must be within therapeutic range of intended use of anticoagulants * Female of childbearing potential: negative urine or serum pregnancy test, performed within 14 days prior to day 1 of protocol therapy * If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Cardiac function (12 lead- electrocardiogram \[ECG\] versus \[vs\] non-12 lead ECG), performed within 14 days prior to day 1 of protocol therapy * Female subjects must be either post-menopausal, surgically sterilized, or willing to use an acceptable method of birth control (i.e. a hormonal contraceptive, intra-uterine device, diaphragm with spermicide, condom with spermicide, or abstinence) beginning prior to study entry, for the duration of the study, and for six months following last dose of avelumab/utomilumab; should a woman become pregnant or suspect that she is pregnant while participating on the trial, she should inform her treating physician immediately * Male subjects must agree to use an acceptable method of contraception beginning prior to study entry, for the duration of the study, and for six months following last dose of avelumab/utomilumab Exclusion Criteria: * Patients who are not hematopoietic stem cell transplant candidates are excluded for the DLBCL cohort (cohort #1) * Vaccinated with live, attenuated vaccines within 4 weeks of first dose of avelumab * Patients may be on steroids prior to initiation of treatment, provided that, by cycle 1 day 1, steroid use is tapered down to less than or equal to 10 mg/day of prednisone * For cohort 1 (DLBCL) only: prior organ transplantation including allogeneic stem-cell transplantation * For cohort 1 (DLBCL) only: prior RICE chemotherapy * Patients with prior treatment with PD-1 or PD-L1 inhibitor * Patients may not be receiving any other investigational agents, or concurrent biological therapy, chemotherapy, or radiation therapy * Current use of immunosuppressive medication, except for the following: a. intranasal, inhaled, topical steroids, or local steroid injection (e.g., intra-articular injection); b. systemic corticosteroids at physiologic doses =\< 10 mg/day of prednisone or equivalent; c. steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) * For cohort 2 (MCL) only: strong CYP3A4 inducers/inhibitors within 14 days prior to day 1 of protocol therapy and/or requires treatment with a strong cytochrome P450 (CYP) 3A4/5 inhibitor * CRITERIA SPECIFIC FOR COHORT #2 (MCL): Significant screening electrocardiogram (ECG) abnormalities including, but not limited to, left bundle branch block, 2nd degree atrioventricular (AV) block type II, 3rd degree block, or corrected QT interval (QTc) \>= 470 msec; subjects with a cardiac pacemaker who have a QTc interval of \>= 470 msec may be eligible if these findings are considered not clinically significant as documented via a cardiology evaluation * CRITERIA SPECIFIC FOR COHORT #2 (MCL): Currently active, clinically significant hepatic impairment Child-Pugh class B or C according to the Child Pugh classification * CRITERIA SPECIFIC FOR COHORT #2 (MCL): Unable to swallow capsules or malabsorption syndrome, disease significantly affecting gastrointestinal function, or resection of the stomach or small bowel, symptomatic inflammatory bowel disease or ulcerative colitis, or partial or complete bowel obstruction * CRITERIA SPECIFIC FOR COHORT #2 (MCL): Known bleeding disorders (e.g., von Willebrand's disease) or hemophilia * CRITERIA SPECIFIC FOR COHORT #2 (MCL): Major surgery within 4 weeks of first dose of study drug * Active GVHD or on immunosuppressive medication for GVHD (applies to cohort #2) * Recent infection requiring intravenous anti-infective treatment that was completed =\< 14 days before enrollment; active infection requiring systemic therapy * Persisting toxicity related to prior therapy (National Cancer Institute \[NCI\] Common Terminology Criteria for Adverse Events \[CTCAE\] version \[v\]. 4.03 grade \> 1); however, alopecia, sensory neuropathy grade =\< 2, or other grade =\< 2 not constituting a safety risk based on investigator's judgement are acceptable * Hypersensitivity to chemotherapy or history of allergic reactions attributed to compounds of similar chemical or biologic composition to rituximab, ibrutinib, ICE, Ave, or Uto * Patients should not have any uncontrolled illness including ongoing or active infection * Any life-threatening illness, medical condition, or organ system dysfunction that, in the investigator's opinion, could compromise the subject's safety or put the study outcomes at undue risk * Myocardial infarction within 6 months prior to enrollment or New York Heart Association (NYHA) class III or IV heart failure, uncontrolled angina, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia or active conduction system abnormalities; prior to study entry, any ECG abnormality at screening has to be documented by the Investigator as not medically relevant * Clinically significant (i.e., active) cardiovascular disease: cerebral vascular accident/stroke (\< 6 months prior to enrollment), myocardial infarction (\< 6 months prior to enrollment), unstable angina, congestive heart failure (\>= New York Heart Association classification class II), or serious cardiac arrhythmia requiring medication * Diagnosed or treated for another malignancy within 3 years of enrollment, with the exception of complete resection of basal cell carcinoma or squamous cell carcinoma of the skin, an in situ malignancy, or low-risk prostate cancer after curative therapy * Patients with active central nervous system (CNS) disease or history of brain metastases are excluded from study * Active human immunodeficiency virus (HIV) or hepatitis C virus (HCV) or hepatitis B virus (HBV); subjects who have an undetectable HIV viral load with CD4 \>= 200 and are on highly active antiretroviral therapy (HAART) medication are allowed; subjects who are positive for hepatitis B core antibody or hepatitis B surface antigen must have a negative polymerase chain reaction (PCR) result before enrollment; those who are PCR positive will be excluded; patients who have had hepatitis C but have finished treatment and are PCR negative will be allowed; (testing to be done only in patients suspected of having infections or exposures) * Active autoimmune disease that might deteriorate when receiving an immuno-stimulatory agent; patients with diabetes type I, vitiligo, psoriasis, or hypo- or hyperthyroid diseases not requiring immunosuppressive treatment are eligible * Other severe acute or chronic medical conditions including immune colitis, inflammatory bowel disease, immune pneumonitis, pulmonary fibrosis or psychiatric conditions including recent (within the past year) or active suicidal ideation or behavior; or laboratory abnormalities that may increase the risk associated with study participation or study treatment administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for entry into this study * Pregnant women are excluded from this study; breastfeeding should be discontinued * Other active malignancy * Any other condition that would, in the investigator's judgement, contraindicate the patient's participation in the clinical study due to safety concerns with clinical study procedures * Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics) * Unable to understand the purpose and risks of the study and to provide a signed and dated informed consent form (ICF) and authorization to use protected health information (in accordance with national and local subject privacy regulations)