Brentuximab Vedotin and Nivolumab for the Treatment of Patients With Relapsed/Refractory Classical Hodgkin Lymphoma

Inclusion Criteria: * Documented informed consent of the participant and/or legally authorized representative * Assent, when appropriate, will be obtained per institutional guidelines * Be willing to provide tissue (either from a fresh core or excisional biopsy performed as standard of care, or from archival tissue) of a biopsy that was performed after frontline systemic therapy, and prior to starting protocol therapy * If unavailable, exceptions may be granted with study principal investigator (PI) approval * Eastern Cooperative Oncology Group (ECOG) =\< 2 * Histologically confirmed diagnosis of classical Hodgkin lymphoma (excluding nodular lymphocyte predominant Hodgkin lymphoma) according to the World Health Organization (WHO) classification, with hematopathology review at the participating institution * Relapsed or refractory disease after no more than 1 line of prior therapy (not counting radiotherapy). However, a maximum of 5 patients with primary refractory disease may be enrolled in this study. Note: Patients who received BV or an anti-PD-1/PD-L1 agent as part of frontline therapy are eligible if they achieved a CMR with frontline therapy and have not relapsed within 6 months from the end of frontline therapy Relapse must have been confirmed histologically (with hematopathology review at the participating institution) * Not a candidate for ASCT, based on age, co-morbidities, or patient preference. The reason for ASCT non-candidacy must be documented in the Case Report Form and verified by the site PI * Measurable disease (at least one non-bony fludeoxyglucose F-18 \[FDG\]-avid lesion \>= 1.5 cm in long axis) * Absolute neutrophil count (ANC) \>= 1,000/mm\^3 * NOTE: Growth factor is not permitted within 7 days of ANC assessment unless cytopenia is secondary to disease involvement * Platelets \>= 50,000/mm\^3 * NOTE: Platelet transfusions are not permitted within 7 days of platelet assessment unless cytopenia is secondary to disease involvement * Hemoglobin \>= 8 g/dL (no transfusion allowed within 3 days prior to screening) * Total bilirubin =\< 1.5 x upper limit of normal (ULN) or direct bilirubin =\< 1.5 x ULN for patients with Gilbert's disease * Aspartate aminotransferase (AST) =\< 2.5 x ULN * Alanine aminotransferase (ALT) =\< 2.5 x ULN * Creatinine clearance of \>= 40 mL/min per 24 hour urine test or the Cockcroft-Gault formula * Women of childbearing potential (WOCBP): negative urine or serum pregnancy test. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Agreement by women and men of childbearing potential\* to use an effective method of birth control or abstain from heterosexual activity for the course of the study through at least 5 months (women) or 7 months (men) after the last dose of protocol therapy * Childbearing potential defined as not being surgically sterilized (men and women) or have not been free from menses for \> 1 year (women only) Exclusion Criteria: * Concomitant investigational therapy * Live vaccine within 30 days prior to day 1 of protocol therapy (e.g. measles, mumps, rubella, varicella, yellow fever, rabies, bacillus Calmette-Guerin \[BCG\], oral polio vaccine, and oral typhoid) * Grade \>= 2 peripheral neuropathy * History of prior \>= grade 3 hypersensitivity to either brentuximab vedotin or nivolumab * Known active central nervous system (CNS) involvement by lymphoma, including parenchymal and/or lymphomatous meningitis * History of another primary malignancy that has not been in remission for at least 3 years, with the following exceptions: * Non-melanoma skin cancer treated with curative intent * In situ cervical cancer * If the malignancy is expected to not require any treatment for at least 2 years (this exception should be discussed with the study PI) * Condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone or equivalent) or other immunosuppressive medications within 14 days of study drug administration. Exceptions are: * Inhaled or topical steroids and * Adrenal replacement doses \> 10 mg daily prednisone equivalents in the absence of active autoimmune disease * History of progressive multifocal leukoencephalopathy (PML) * Prior diagnosis of inherited or acquired immunodeficiency * Active pneumonitis or interstitial lung disease * Active, known or suspected autoimmune disease. The following are exceptions: * Vitiligo * Psoriasis not requiring systemic treatment * Hemolytic anemia associated with the lymphoma * Type I diabetes mellitus, if adequately controlled with therapy * Thyroid disease, if adequately controlled with therapy * Conditions not expected to recur in the absence of an external trigger (such exceptions should be discussed with the study PI) * Active history of: * Hepatitis B (hepatitis B virus \[HBV\]) or C (hepatitis C virus \[HCV\]) infection. Patients with past HBV infection (defined as negative hepatitis B surface antigen \[HBsAg\] and positive hepatitis B core antibody \[HBcAb\]) are eligible if HBV DNA is undetectable. Patients who are positive for HCV antibody are eligible if polymerase chain reaction (PCR) is negative for HCV ribonucleic acid (RNA) * Human immunodeficiency virus (HIV) infection. Subjects who have an undetectable or unquantifiable human immunodeficiency virus (HIV) viral load with CD4 \>= 200 and are on highly active antiretroviral therapy (HAART) medication are allowed. Testing to be done only in patients suspected of having infections or exposures * History of a cerebral vascular event (stroke or transient ischemic attack), unstable angina, myocardial infarction, or cardiac symptoms consistent with New York Heart Association class III-IV within 6 months prior to day 1 of protocol therapy * Pregnant or breastfeeding * Any other condition that would, in the Investigator's judgment, contraindicate the patient's participation in the clinical study due to safety concerns with clinical study procedures * Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics)