Temsirolimus and Brentuximab Vedotin in Treating Patients With Relapsed or Refractory Hodgkin Lymphoma

Inclusion Criteria: * Patients must have histologically confirmed cluster of differentiation (CD)30 positive relapsed or refractory Hodgkin lymphoma * Measurable disease, defined as at least one lesion \> 1.5 cm, in the greatest transverse diameter * Patients must have failed autologous stem cell transplant or at least 2 prior cytotoxic regimens for Hodgkin lymphoma * Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%) * Life expectancy of greater than 3 months * Leukocytes \>= 3,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin within normal institutional limits or \< 3 x the upper limit of normal in patients with Gilbert's disease * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 × institutional upper limit of normal * Creatinine =\< 1.5 x institutional upper limit of normal OR creatinine clearance \>= 40 mL/min/1.73 m\^2 for patients with creatinine levels above institutional normal * Patients must have no evidence of dyspnea at rest, no exercise intolerance, and a pulse oximetry \> 92% while breathing room air * Patients must have forced expiratory volume in 1 second (FEV1)/forced vital capacity (FVC) \> 60% by pulmonary function test (PFT), unless due to large mediastinal mass from Hodgkin's lymphoma (HL); carbon monoxide diffusion capacity (DLCO), FEV1, and FVC all \> 50% predicted value * Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately; men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 4 months after completion of temsirolimus and brentuximab vedotin administration * Ability to understand and the willingness to sign a written informed consent document * Fasting serum cholesterol =\< 300 mg/dL OR =\< 7.75 mmol/L AND fasting triglycerides =\< 2.5 x upper limit of normal (ULN); NOTE: In case one or both of these thresholds are exceeded, the patient can only be included after initiation of appropriate lipid lowering medication * Patients with known human immunodeficiency virus (HIV) infection must have CD4 count greater than 200; anti-retroviral agents that induce or inhibit cytochrome P450 (CYP)3A4 activity are not allowed Exclusion Criteria: * Patients who are eligible for autologous stem cell transplant unless they refuse to receive autologous stem cell transplant * Patients who have had chemotherapy or radiotherapy within 3 weeks of registration or those who have not recovered from adverse events due to agents administered more than 3 weeks earlier; Note: patients are considered enrolled on the study after protocol registration and not after signing consent * Patients who have received brentuximab vedotin within 6 months of enrolling on the study * Patients who are receiving any other investigational agents * Patients with known cerebral or meningeal involvement by lymphoma should be excluded from this clinical trial * History of allergic reactions attributed to compounds of similar chemical or biologic composition to temsirolimus or brentuximab vedotin * Patients receiving any medications or substances that are inhibitors or inducers of CYP3A4 are ineligible * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, uncontrolled diabetes, clinically significant pneumonitis, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with temsirolimus and brentuximab vedotin * Previous primary progression or grade 3 toxicity on treatment with brentuximab vedotin * Patients receiving chronic, systemic treatment with corticosteroids or another immunosuppressive agent, except corticosteroids with a daily dosage equivalent to prednisone =\< 20 mg; topical or inhaled corticosteroids are allowed