Umbralisib and Pembrolizumab in Treating Patients With Relapsed or Refractory Classical Hodgkin Lymphoma

Inclusion Criteria: * Relapsed or refractory CHL that has received at least 1 prior lines of therapy * Measurable fludeoxyglucose F-18 (FDG)-avid disease defined by standard criteria (Lugano 2014) and a minimum of 1.0 cm in diameter * Prior treatment with anti-PD1 or anti-PDL1 therapy is allowed. \* Patients who are currently on anti-PD1 or anti-PDL1 therapy who have failed to achieve a CR after at least 18 weeks of treatment may enroll on study. For these patients, anti-PD1 or anti-PDL1 therapy may be delayed for screening and to align pembrolizumab dosing with the expected cycle 1 day 1. Patients with progressive disease after prior anti-PD1 or anti-PDL1 therapy do not have to be treated for 18 weeks * Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 * Ability to swallow and retain oral medication * Willingness and ability to comply with study and follow-up procedures, and give written informed consent * Female subjects of childbearing potential must be surgically sterile, be post-menopausal (for at least 1 year prior to screening visit), or must have a negative pregnancy test within 3 days prior to cycle 1 day 1 and agree to use medically acceptable contraception throughout the study period and for 4 months after the last dose of either study drug. Men of reproductive potential may not participate unless they agree to use medically acceptable contraception throughout the study period and for 4 months after the last dose of either study drug * Patients must be expected to receive at least 2 cycles of therapy * Patients should have a life expectancy if untreated of \>= 90 days in the opinion of the investigator * Patients must have a FDG-positron emission tomography (PET)-computed tomography (CT) of chest, abdomen, and pelvis within 42 days of enrollment * Absolute neutrophil count (ANC) \> 750 * Platelet count \> 40,000 * Total bilirubin =\< 1.5 times the upper limit of normal (ULN) * Alanine aminotransferase (ALT) and aspartate aminotransferase (AST) =\< 2.5 x ULN (if known liver involvement then =\< 5 x ULN is allowed) * Calculated creatinine clearance \> 30 mL/min (as calculated by the Cockcroft-Gault formula) Exclusion Criteria: * Patients receiving cancer therapy (i.e., chemotherapy, immunotherapy, biologic therapy, hormonal therapy, surgery and/or tumor embolization, prednisone \> 10 mg or equivalent) or any investigational drug within 21 days of cycle 1 day 1. Patients receiving radiation therapy within 14 days from cycle 1 day 1. Anti-PD1 or anti-PDL1 therapy in patients with less than a CR after 18 weeks of such therapy is permitted to continue on schedule * Discontinuation from prior anti-PD1 or anti-PDL1 therapy due to immune-related adverse event or any other treatment-related adverse event * Autologous transplantation within 100 days * Prior allogeneic transplant within 12 months of initiation on study * Active graft versus host disease (GVHD) within 90 days prior to cycle 1 day 1 * Evidence of active central nervous system lymphoma * Pregnant or nursing women * Evidence of chronic active hepatitis B or chronic active hepatitis C infection (hepatitis C virus \[HCV\]), active cytomegalovirus (CMV), or known history of human immunodeficiency virus (HIV). If hepatitis B core (HBc) antibody, HCV antibody or CMV immunoglobulin (Ig)M is positive, the patient should be correspondingly evaluated for the presence of HBV, HCV or CMV by deoxyribonucleic acid (DNA) (polymerase chain reaction \[PCR\]) to determine presence or absence of active infection. If HBc antibody is positive, the subject must be evaluated for the presence of hepatitis B virus (HBV) DNA by polymerase chain reaction (PCR). If HCV antibody is positive, the subject must be evaluated for the presence of HCV ribonucleic acid (RNA) by PCR. Subjects with positive HBc antibody and negative HBV DNA by PCR are eligible. Subjects with positive HCV antibody and negative HCV RNA by PCR are eligible. Subjects who are cytomegalovirus (CMV) IgG or CMV IgM positive but who are CMV DNA negative by PCR are eligible * Prior exposure to idelalisib (CAL-101), duvelisib (IPI-145), or any drug that specifically inhibits phosphoinositide-3-kinase (PI3K) * Evidence of ongoing active systemic bacterial, fungal or viral infection * Any severe and/or uncontrolled medical conditions or other conditions that could affect their participation in the study such as: * Symptomatic, or history of documented congestive heart failure (New York \[NY\] Heart Association functional classification III-IV) * Significant cardiovascular disease such as uncontrolled or symptomatic arrhythmias, congestive heart failure, or myocardial infarction within 6 months of randomization * Concomitant use of medication known to cause QT prolongation or torsades de pointes. Poorly controlled or clinically significant atherosclerotic vascular disease including cerebrovascular accident (CVA), transient ischemic attack (TIA), angioplasty, cardiac/vascular stenting within 3 months of randomization * Patients with other prior malignancies except for adequately treated basal cell carcinoma, squamous cell carcinoma of the skin, breast or cervical cancer in situ, or other cancer from which the patient has been disease-free for 5 years or greater, unless approved by the protocol principal investigator * Patients with an active autoimmune disorder (with the exception of autoimmune hemolytic anemia, idiopathic thrombocytopenic purpura \[ITP\] or vitiligo) * History of non-infectious pneumonitis related to prior line of therapy