Nivolumab With or Without Varlilumab in Treating Patients With Relapsed or Refractory Aggressive B-cell Lymphomas

Inclusion Criteria: * Patients must have a histopathologically confirmed diagnosis of an aggressive B-cell non-Hodgkin lymphoma that is recurrent or refractory to standard therapy * For the purpose of this study, aggressive B-cell NHL will be deemed any lymphoma belonging to one of the following groups according to the 2016 revision of the World Health Organization (WHO) classification of lymphoid neoplasms * For the purposes of stratification, diagnoses are grouped into 2 categories: * Category A * Burkitt lymphoma * Burkitt-like lymphoma with 11q aberration * High-grade B-cell lymphoma, with MYC and BCL2 and/or BCL6 rearrangements * High-grade B-cell lymphoma, not otherwise specified (NOS) * Category B * Diffuse large B-cell lymphoma (DLBCL), NOS * Diffuse large B-cell lymphoma (DLBCL), NOS; germinal center B-cell type * Diffuse large B-cell lymphoma (DLBCL), NOS; activated B-cell type * Large B-cell lymphoma with IRF4 rearrangement * T-cell/histiocyte-rich large B-cell lymphoma * Primary DLBCL of the central nervous system (CNS) * Primary cutaneous DLBCL, leg type * Epstein-Barr virus (EBV)+ DLBCL, NOS * EBV+ mucocutaneous ulcer * DLBCL associated with chronic inflammation * Lymphomatoid granulomatosis * Primary mediastinal (thymic) large B-cell lymphoma * Intravascular large B-cell lymphoma * ALK+ large B-cell lymphoma * Plasmablastic lymphoma * Primary effusion lymphoma * Human herpesvirus (HHV)-8+ DLBCL, NOS * B-cell lymphoma, unclassifiable, with features intermediate between DLBCL and classical Hodgkin lymphoma * Patients must have measurable disease, defined as at least one lesion that is \> 15 mm (1.5 cm) in the longest axis on cross-sectional imaging and measurable in two perpendicular dimensions per computed tomography (spiral computed tomography \[CT\]), positron emission tomography (PET)-CT or magnetic resonance imaging (MRI) * Patients must have disease that has relapsed after or is refractory to at least 2 lines of standard therapy; the remaining standard treatment options are unlikely to be effective in the opinion of the treating physician, or patient is felt to be ineligible for such therapies or the patient refuses such therapies; patients who have undergone autologous stem cell transplant are eligible as long as they meet all other criteria * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 * Life expectancy of greater than 12 weeks * White blood cell (WBC) \>= 2000/mm\^3 (within 14 days of registration) * Absolute neutrophil count (ANC) \>= 1500/mm\^3 (within 14 days of registration) * Platelet count \>= 100,000/mm\^3 (within 14 days of registration) * Hemoglobin \> 9.0 g/dL (within 14 days of registration) * Total bilirubin =\< 1.5 x upper limit of normal (ULN) (except patients with Gilbert Syndrome, who can have total bilirubin \< 3.0 mg/dL) (within 14 days of registration) * Aspartate transaminase (aspartate aminotransferase \[AST\]) =\< 2.5 x ULN (within 14 days of registration) * Calculated creatinine clearance (CrCl) \>= 50 mL/min (if using the Cockcroft-Gault formula) (within 14 days of registration) * Females of child bearing potential must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) * Ability to understand and the willingness to sign a written informed consent document Exclusion Criteria: * Patient has received chemotherapy, targeted agent, or radiotherapy within 4 weeks or at least 5 half-lives, whichever is longer, prior to registration * Palliative (limited-field) radiation therapy is permitted, if all of the following criteria are met: * Repeat imaging demonstrates no new sites of bone metastases * The lesion being considered for palliative radiation is not a target lesion * Patient has received immunotherapy (including monoclonal antibodies) within 4 weeks prior to registration * Patients who have not recovered to grade 1 or less from any adverse events due to agents administered more than 4 weeks earlier (excluding alopecia) * Patients who are receiving any other investigational agents * Patients should be excluded if they have had prior treatment with an anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CTLA-4 antibody, or any other antibody or drug specifically targeting T-cell co-stimulation or immune checkpoint pathways * Patients who have received autologous stem cell transplant (ASCT) =\< 12 weeks prior to the first dose of study drug * Patients with a prior history of allogeneic stem cell or solid organ transplantation * Patients with evidence of active disease in the central nervous system (CNS) defined as either the presence of active lesions on MRI obtained within 4 weeks of registration or progressive neurological decline * Patients with primary CNS lymphoma who develop systemic recurrence following standard therapy may be included as long as no active CNS disease is present at the time or enrollment; similarly, patients with secondary involvement of the CNS from a systemic lymphoma may be included as long as the CNS disease has been optimally treated and they demonstrate no evidence of active CNS disease * History of allergic reactions attributed to compounds of similar chemical or biologic composition to CDX-1127 (varlilumab) and/or nivolumab * History of severe hypersensitivity reaction to any monoclonal antibody * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Pregnant women are excluded from this study because CDX-1127 (varlilumab) and nivolumab are agents with the potential for teratogenic or abortifacient effects; because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with CDX-1127 (varlilumab) or nivolumab, breastfeeding should be discontinued if the mother is treated with CDX-1127 (varlilumab) or nivolumab * Patients with human immunodeficiency virus (HIV) are eligible for the study provided they meet the other protocol criteria in addition to the following: * Undetectable HIV load by standard polymerase chain reaction (PCR) clinical assay within 60 days prior to registration * Absolute CD4 count of \>= 200 mm\^3 within 60 days prior to registration * Willing to maintain adherence to combination antiretroviral therapy * No history of acquired immunodeficiency syndrome (AIDS) defining condition (other than lymphoma or CD4 cell count \< 200 mm\^3) * Likely to have near normal lifespan if not for the presence of relapsed/refractory lymphoma * Patients with evidence of hepatitis B virus (HBV) are eligible provided there is minimal hepatic injury and the patient has undetectable HBV on suppressive HBV therapy; patient must be willing to maintain adherence to HBV therapy * Patients with previously treated and eradicated hepatitis C virus (HCV) who have minimal hepatic injury are eligible * Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including systemic corticosteroids, should be excluded; these include but are not limited to patients with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome, myasthenia gravis; systemic autoimmune disease such as systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, hepatitis; and patients with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or phospholipid syndrome should be excluded because of the risk of recurrence or exacerbation of disease; patients with vitiligo, endocrine deficiencies including thyroiditis managed with replacement hormones including physiologic corticosteroids are eligible; patients with rheumatoid arthritis and other arthropathies, Sjogren's syndrome and psoriasis controlled with topical medication and patients with positive serology, such as antinuclear antibodies (ANA), anti-thyroid antibodies should be evaluated for the presence of target organ involvement and potential need for systemic treatment but should otherwise be eligible * Patients are permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger (precipitating event) * Patients should be excluded if they have a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration; inhaled or topical steroids and adrenal replacement doses =\< 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease; patients are permitted to use topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption); physiologic replacement doses of systemic corticosteroids are permitted, even if =\< 10 mg/day prednisone equivalents; a brief course of corticosteroids for prophylaxis (e.g., contrast dye allergy) or for treatment of non-autoimmune conditions (e.g., delayed-type hypersensitivity reaction caused by contact allergen) is permitted * Patients who have had evidence of active or acute diverticulitis, intra-abdominal abscess, gastrointestinal (GI) obstruction and abdominal carcinomatosis which are known risk factors for bowel perforation should be evaluated for the potential need for additional treatment before coming on study * Patients with other active malignancy =\< 3 years prior to registration for which active treatment is required must be excluded; patients with composite lymphomas that have a non-B-cell component must be excluded EXCEPTIONS: Non-melanotic skin cancer or carcinoma-in-situ of the cervix