Brentuximab Vedotin and Nivolumab With or Without Ipilimumab in Treating Patients With Relapsed or Refractory Hodgkin Lymphoma

Inclusion Criteria: * PHASE I (ARMS A, B, C, D, E, F, G, H, I, X, Y, Z) * Age \>= 18 years * Patients must have pathologically confirmed relapsed or refractory classical Hodgkin lymphoma (cHL); a biopsy at any relapse is acceptable; other histologies including lymphocyte predominant (LP) HL are not permitted * Patients must have relapsed after first line chemotherapy; may have relapsed after autologous or allogeneic stem cell transplant, or have primary refractory disease; no upper limit for number of prior therapies; if status post allogeneic stem cell transplant, no active graft versus host disease * Patients may have received prior brentuximab vedotin, but must not have received brentuximab vedotin within 6 months prior to registration, and must not have relapsed within 6 months of receiving previous brentuximab vedotin; patients may not have received prior nivolumab or PD1/PDL1 axis agents; patients in the nivolumab/brentuximab cohorts ONLY (D, E, F, Y) may have received prior ipilimumab * Patients may have received other prior activating immunotherapies (i.e. checkpoint inhibitors), but must not have received them within 6 months prior to registration, and there must be no serious unresolved complication of therapy at the time of registration; for the purposes of this study monoclonal antibodies and antibody drug conjugates are not considered to be activating immunotherapies and there are no additional time restrictions on prior exposure to these agents (except prior brentuximab vedotin) * Eastern Cooperative Oncology Group (ECOG)-American College of Radiology Imaging Network (ACRIN) performance status between 0-2 * Patients must have measurable disease; baseline measurements and evaluations must be obtained within 4 weeks of registration to the study; abnormal PET scans will not constitute evaluable disease unless verified by a diagnostic quality CT scan; patients must use the same imaging modality (CT or PET/CT) throughout the study * Patient must not be pregnant or breast-feeding due to risk of fetal harm by the chemotherapeutic agents prescribed in this protocol; all patients of childbearing potential must have a blood test or urine study within 2 weeks prior to registration to rule out pregnancy; a patient of childbearing potential is anyone, regardless of whether they have undergone tubal ligation, who meets the following criteria: 1) has not undergone a hysterectomy or bilateral oophorectomy; or 2) has not been naturally postmenopausal for at least 24 consecutive months (i.e., has had menses at any time in the preceding 24 consecutive months) * Patient of childbearing potential and/or sexually active patients must either abstain from sexual intercourse for the duration of their participation in the study or agree to use both single barrier contraception and birth control pills or implants for at least one week prior to the start of the study drug and continuing for 5 months after the last dose of study drug (for patients of childbearing potential) and for 7 months after the last dose of study drug (for patients who are sexually active with anyone of childbearing potential); should a patient become pregnant or suspect pregnancy while the patient or their partner is participating in this study, the patient (or the participating partner) should inform the treating physician immediately * Patients must have no evidence of dyspnea at rest and a pulse oximetry \> 92% while breathing room air * Patients must have forced expiratory volume in 1 second (FEV1)/forced vital capacity (FVC) \> 60% by pulmonary function test (PFT), unless due to large mediastinal mass from HL; carbon monoxide diffusion capacity (DLCO), FEV1, and FVC all \> 50% predicted value; all pulmonary function tests must be obtained within one month prior to registration * Absolute neutrophil count (ANC) \>= 1500/mcL (1.5 x 10\^9/L) (obtained within 2 weeks prior to registration) * Platelets \>= 75,000/mcL (75 x 10\^9/L) (obtained within 2 weeks prior to registration) * Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 2.5 x upper limit of normal (ULN) (obtained within 2 weeks prior to registration) * Bilirubin =\< 2 x upper limit of normal (ULN) (unless documented Gilbert's syndrome, for which bilirubin =\< 3 x upper limit of normal \[ULN\] is permitted) (obtained within 2 weeks prior to registration) * Calculated creatinine clearance by Cockcroft-Gault formula \>= 30 ml/min (obtained within 2 weeks prior to registration) * No evidence of prior malignancy except adequately treated non-melanoma skin cancer, in situ cervical carcinoma or any surgically- or radiation-cured malignancy continuously disease free for \>= 5 years so as not to interfere with interpretation of radiographic response * Patient must have no current or prior history of central nervous system (CNS) involvement * All prior therapy must have been completed at least 21 days prior to enrollment; no concomitant anti lymphoma therapy, including systemic corticosteroids for the purpose of treatment of lymphoma are allowed; topical steroids are allowed * No history of Steven's Johnson's syndrome, toxic epidermal necrolysis (TEN)s syndrome, or motor neuropathy * Human immunodeficiency virus (HIV) positive patients are allowed on this study if they have a CD4 count \> 400, and are on a stable antiviral regimen; patients with poorly controlled HIV or other chronic active viral infections will be excluded * Patients must not have autoimmune disorders or conditions of immunosuppression that require current ongoing treatment with systemic corticosteroids (or other systemic immunosuppressants), including oral steroids (i.e., prednisone, dexamethasone) or continuous use of topical steroid creams or ointments or ophthalmologic steroids; a history of occasional (but not continuous) use of steroid inhalers is allowed * Replacement doses of steroids for patients with adrenal insufficiency are allowed; patients who discontinue use of these classes of medication for at least 2 weeks prior to initiation of study treatment are eligible if, in the judgment of the treating physician investigator, the patient is not likely to require resumption of treatment with these classes of drugs during the study * Exclusion from this study also includes patients with a history of symptomatic autoimmune disease (e.g., rheumatoid arthritis, systemic progressive sclerosis \[scleroderma\], systemic lupus erythematosus, Sjogren's syndrome, autoimmune vasculitis \[e.g., Wegener's Granulomatosis\]); motor neuropathy considered of autoimmune origin (e.g., Guillain-Barre syndrome and Myasthenia Gravis); other CNS autoimmune disease (e.g., Multiple sclerosis); patients with autoimmune hypothyroid disease or type I diabetes on replacement treatment are eligible * Patients must not have grade 2 or greater peripheral sensory neuropathy * Patients must not have New York Heart Association (NYHA) class III or IV heart failure, uncontrolled angina, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia * Patients must not have previously existing hypersensitivity to brentuximab vedotin or ipilimumab * Patients must not have a serious medical or psychiatric illness likely to interfere with study participation * Patients must not be participating in any other clinical trial or taking any other experimental medications within 21 days prior to registration * Routine vaccinations, including seasonal influenza, should be given at least 2 weeks prior to study treatment; vaccines are not prohibited on study, but must be given at least 6 weeks after cycle 1 and not within 7 days of treatment * Patients registering to Arms D, E, F, G, H, I, X, Y must not currently be smoking tobacco or other substances and must not have smoked within the past 6 months * RANDOMIZED PHASE II (ARMS K AND L): Age \>= 12 years * Pediatric patients will include any patients \< 18 years of age * RANDOMIZED PHASE II (ARMS K AND L): Patients must have pathologically confirmed relapsed or refractory classical Hodgkin lymphoma (cHL); a biopsy at any relapse is acceptable; other histologies including lymphocyte predominant (LP) HL are not permitted * RANDOMIZED PHASE II (ARMS K AND L): Patients must have relapsed after first line chemotherapy; may have relapsed after autologous stem cell transplant, or have primary refractory disease; no upper limit for number of prior therapies; patient must not have received a prior allogeneic stem cell transplant (out of risk of reactivation of pulmonary graft versus host disease \[GVHD\]) * RANDOMIZED PHASE II (ARMS K AND L): Patients may have received prior brentuximab vedotin, but must not have received brentuximab vedotin within 6 months prior to registration, and must not have relapsed within 6 months of receiving previous brentuximab vedotin; patients may not have received prior nivolumab or PD1/PDL1 axis agents; patients may not have received prior ipilimumab * RANDOMIZED PHASE II (ARMS K AND L): Patients may not have received other prior activating immunotherapies (i.e. checkpoint inhibitor therapies); for the purposes of this study monoclonal antibodies and antibody drug conjugates are not considered to be activating immunotherapies and there are no additional time restrictions on prior exposure to these agents (except prior brentuximab vedotin) * RANDOMIZED PHASE II (ARMS K AND L): Adult patient (\>= 18 years of age) ECOG-ACRIN performance status between 0-2 * Pediatric patients (16-17 years of age) must have a Karnofsky performance level \>= 50% * Pediatric patients (12-15 years of age) must have a Lansky performance level \>= 50 * RANDOMIZED PHASE II (ARMS K AND L): Patients must have measurable disease; baseline measurements and evaluations must be obtained within 4 weeks of registration to the study; abnormal PET scans will not constitute evaluable disease unless verified by a diagnostic quality CT scan; patients must use the same imaging modality (CT or PET/CT) throughout the study * RANDOMIZED PHASE II (ARMS K AND L): Patient must not be pregnant or breast-feeding due to risk of fetal harm by the chemotherapeutic agents prescribed in this protocol; all patients of childbearing potential must have a blood test or urine study within 2 weeks prior to registration to rule out pregnancy; a patient of childbearing potential is defined as anyone, regardless of whether they have undergone tubal ligation, who meets the following criteria: 1) has achieved menarche at some point, 2) has not undergone a hysterectomy or bilateral oophorectomy; or 2) has not been naturally postmenopausal (amenorrhea following cancer therapy does not rule out childbearing potential) for at least 24 consecutive months (i.e., has had menses at any time in the preceding 24 consecutive months) * RANDOMIZED PHASE II (ARMS K AND L): Patient of childbearing potential and/or sexually active patient must either abstain from sexual intercourse for the duration of their participation in the study or agree to use both double barrier contraception and birth control pills or implants for at least one week prior to the start of the study drug and continuing for 5 months after the last dose of study drug (for patients of childbearing potential) and for 7 months after the last dose of study drug (for patients who are sexually active with anyone of childbearing potential); should a patient become pregnant or suspect pregnancy while the patient or their partner is participating in this study, the patient (or the participating partner) should inform the treating physician immediately * RANDOMIZED PHASE II (ARMS K AND L): Patients with impaired decision-making capacity are eligible with legally authorized representative * RANDOMIZED PHASE II (ARMS K AND L): Patients must have no evidence of dyspnea at rest and a pulse oximetry \> 92% while breathing room air * RANDOMIZED PHASE II (ARMS K AND L): Patients must have FEV1/FVC \> 60% by pulmonary function test (PFT), unless due to large mediastinal mass from HL; carbon monoxide diffusion capacity (DLCO), FEV1, and FVC all \> 50% predicted value; all pulmonary function tests must be obtained within one month prior to registration * RANDOMIZED PHASE II (ARMS K AND L): ANC \>= 1500/mcL (1.5 x 0\^9/L) (obtained within 2 weeks prior to registration) * RANDOMIZED PHASE II (ARMS K AND L): Platelets \>= 75,000/mcL (75 x 10\^9/L) (obtained within 2 weeks prior to registration) * RANDOMIZED PHASE II (ARMS K AND L): AST/ALT =\< 2.5 x upper limit of normal (ULN) for age (obtained within 2 weeks prior to registration) * RANDOMIZED PHASE II (ARMS K AND L): Bilirubin =\< 2 x upper limit of normal (ULN) (unless documented Gilbert's syndrome, for which bilirubin =\< 3 x upper limit of normal \[ULN\] is permitted) (obtained within 2 weeks prior to registration) * RANDOMIZED PHASE II (ARMS K AND L): Adult patients (\>= 18 years old) must have a calculated creatinine clearance by Cockcroft-Gault formula \>= 30 ml/min (obtained within 2 weeks prior to registration) * RANDOMIZED PHASE II (ARMS K AND L): Pediatric patients (\< 18 years old) must have a creatinine clearance or radioisotope glomerular filtration rate (GFR) \>= 70 mL/min/1.73 m\^2 or serum creatinine based on age/sex as follows: * Age: maximum serum creatinine (mg/dL) * \< 13 years: male (1.2), female (1.2) * 13 to \< 16 years: male (1.5), female (1.4) * \>= 16 years: male (1.7), female (1.4) * RANDOMIZED PHASE II (ARMS K AND L): No evidence of prior malignancy except adequately treated non-melanoma skin cancer, in situ cervical carcinoma or any surgically- or radiation-cured malignancy continuously disease free for \>= 5 years so as not to interfere with interpretation of radiographic response * RANDOMIZED PHASE II (ARMS K AND L): Patient must have no current or prior history of CNS involvement * RANDOMIZED PHASE II (ARMS K AND L): All prior therapy must have been completed at least 21 days prior to enrollment (6 weeks for nitrosoureas or mitomycin C); no concomitant anti lymphoma therapy, including systemic corticosteroids for the purpose of treatment of lymphoma are allowed; topical steroids are allowed * RANDOMIZED PHASE II (ARMS K AND L): No history of Steven's Johnson's syndrome, TENs syndrome, or motor neuropathy * RANDOMIZED PHASE II (ARMS K AND L): HIV positive patients are eligible provided they meet the other protocol criteria including the following: * Long term survival expected were it not for the cHL * HIV viral loads undetectable by standard clinical HIV testing * Willing to adhere to effective combination antiretroviral therapy * RANDOMIZED PHASE II (ARMS K AND L): Patients must not have autoimmune disorders, prior solid organ transplant, or conditions of immunosuppression that require current ongoing treatment with systemic corticosteroids (or other systemic immunosuppressants), including oral steroids (i.e., prednisone, dexamethasone) or continuous use of topical steroid creams or ointments or ophthalmologic steroids; a history of occasional (but not continuous) use of steroid inhalers is allowed; replacement doses of steroids for patients with adrenal insufficiency are allowed; patients who discontinue use of steroid medication for at least 2 weeks prior to initiation of therapy are eligible if, in the judgment of the treating physician investigator, the patient is not likely to require resumption of treatment with these classes of drugs during the study; exclusion from this study also includes patients with a history of symptomatic autoimmune disease (e.g., rheumatoid arthritis, systemic progressive sclerosis \[scleroderma\], systemic lupus erythematosus, Sjogren's syndrome, autoimmune vasculitis \[e.g., Wegener's Granulomatosis\]); motor neuropathy considered of autoimmune origin (e.g., Guillain-Barre syndrome and Myasthenia Gravis); other CNS autoimmune disease (e.g., Multiple sclerosis); patients with autoimmune hypothyroid disease or type I diabetes on replacement treatment are eligible * RANDOMIZED PHASE II (ARMS K AND L): Patients must not have grade 2 or greater peripheral sensory neuropathy * RANDOMIZED PHASE II (ARMS K AND L): Patients must not have NYHA class III or IV heart failure, uncontrolled angina, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia * RANDOMIZED PHASE II (ARMS K AND L): Patients must not have previously existing hypersensitivity to brentuximab vedotin or ipilimumab * RANDOMIZED PHASE II (ARMS K AND L): Patients must not have a serious medical or psychiatric illness likely to interfere with study participation * RANDOMIZED PHASE II (ARMS K AND L): Patients must not be participating in any other clinical trial or taking any other experimental medications within 21 days prior to registration * RANDOMIZED PHASE II (ARMS K AND L): Routine vaccinations, including seasonal influenza, should be given at least 2 weeks prior to study treatment; vaccines are not prohibited on study, but must be given at least 6 weeks after cycle 1 and not within 7 days of treatment * RANDOMIZED PHASE II (ARMS K AND L): Patients must not currently be smoking tobacco or other agents; vaping is not allowed * RANDOMIZED PHASE II (ARMS K AND L): Patients must not have a history of or evidence of cardiovascular risks including any of the following: * QT interval corrected for heart rate using the Bazett's formula QTcB \>= 480 msec at baseline * History of acute coronary syndromes (including myocardial infarction or unstable angina), coronary angioplasty, or stenting within the past 24 weeks prior to registration * History prior to registration or evidence of current \>= class II congestive heart failure as defined by the New York Heart Association (NYHA) functional classification system * Left ventricular ejection fraction (LVEF) =\< lower limit of normal on cardiac echocardiogram (echo) or multigated acquisition scan (MUGA) * Intra-cardiac defibrillator * History of abnormal cardiac valve morphology (\>= grade 2) documented by ECHO; (subjects with grade 1 abnormalities \[i.e., mild regurgitation/stenosis\] can be entered on study); subjects with moderate valvular thickening should not be entered on study * History or evidence of current clinically significant uncontrolled cardiac arrhythmias; clarification: subjects with atrial fibrillation controlled for \> 30 days prior to dosing are eligible * Treatment refractory hypertension defined as a blood pressure of systolic \> 140 mmHg and/or diastolic \> 90 mm Hg which cannot be controlled by anti-hypertensive therapy