Nivolumab and Lenalidomide in Treating Patients With Relapsed or Refractory Non-Hodgkin or Hodgkin Lymphoma

Inclusion Criteria: * PHASE I: Histologically confirmed B-cell NHL with any of the following subtypes: DLBCL, mantle cell lymphoma (MCL), FL, marginal zone lymphoma (MZL) and lymphoplasmacytic lymphoma/Waldenstrom's macroglobulinemia (LL/WM), Burkitt's lymphoma (BL); patients with histological transformation to DLBCL from indolent lymphoma, primary mediastinal lymphoma and grey zone lymphoma are eligible * PHASE I: Histologically confirmed classical or lymphocyte predominant Hodgkin's disease that is relapsed or refractory after at least one prior chemotherapy * PHASE I: Patients must have received at least one prior therapy; prior autologous stem cell transplant is permitted; patients with DLBCL who have not had prior high-dose therapy (HDT)/autologous stem cell transplant (ASCT) must be ineligible for transplant; prior lenalidomide is not permitted if patients have progressed on therapy * PHASE IB DOSE EXPANSION: Histologically confirmed classical or lymphocyte predominant Hodgkin's disease * PHASE IB DOSE EXPANSION: Patients must have received at least one prior therapy; prior autologous stem cell transplant is permitted; patients who have not had prior HDT/ASCT must be ineligible for transplant; prior lenalidomide is not permitted if patients have progressed on therapy * PHASE II: Histologically confirmed B-cell NHL: * Cohort 1: with only de novo DLBCL or transformed indolent Non-Hodgkin's lymphoma (excludes Richter's syndrome). * Cohort 2: with only FL of grade 1, 2 or 3a * PHASE II: Patients must have received at least one prior therapy (in patients with transformed DLBCL must have received at least one prior therapy; prior autologous stem cell transplant is permitted; patients with DLBCL who have not had prior HDT/ASCT must be ineligible for transplant; prior lenalidomide is not permitted if patients have progressed on therapy * Be willing and able to provide written informed consent/assent for the trial * Have evaluable disease * Eastern Cooperative Oncology Group (ECOG) performance status of 0-2 * Absolute neutrophil count (ANC) \>= 1,500 /mcL * Platelets \>= 100,000 /mcL in the absence of transfusion support within 7 days of determining eligibility * Hemoglobin \>= 8 g/dL * Serum creatinine =\< 1.5 X upper limit of normal (ULN) OR measured or calculated creatinine clearance (glomerular filtration rate \[GFR\] can also be used in place of creatinine or creatinine clearance \[CrCl\]) \>= 40 mL/min creatinine clearance * Serum total bilirubin =\< 1.5 X ULN OR except subjects with Gilbert syndrome, who can have total bilirubin \< 3.0 mg/dL * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3.0 X ULN * Female subject of childbearing potential should have a negative urine or serum pregnancy at screening and within 24 hours prior to receiving the first dose of study medication; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Female subjects of childbearing potential should be willing to use 2 methods of birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study through 120 days after the last dose of study medication; subjects should agree to ongoing pregnancy testing during the course of the study and after the end of study therapy; female subjects of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year * Male subjects should agree to use an adequate method of contraception starting with the first dose of study therapy through 120 days after the last dose of study therapy; males must refrain from donating sperm during study participation and for 120 days after the last dose of study medication * Willing and able to receive adequate prophylaxis and/or therapy for thromboembolic events * Be willing and able to understand and give written informed consent and comply with all study related procedures Exclusion Criteria: * Is currently participating and receiving study therapy or has participated in a study of an investigational agent and received study therapy or used an investigational device within 3 weeks of the first dose of treatment * Has a diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment; subjects may use topical or inhaled corticosteroids or low-dose steroids (=\< 20 mg of prednisone or equivalent per day) as therapy for comorbid conditions; during study participation, subjects may receive systemic or enteric corticosteroids as needed for treatment-emergent comorbid conditions * Has a known history of active TB (bacillus tuberculosis) * Hypersensitivity to nivolumab or lenalidomide or any of their excipients * Has had a prior anti-cancer monoclonal antibody (mAb) within 2 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier * Has had prior chemotherapy or radiation therapy within 2 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events due to a previously administered agent * Note: subjects with =\< grade 2 neuropathy are an exception to this criterion and may qualify for the study * Note: if subject received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy; patients must be 4 weeks out from major procedures * Has a known additional malignancy that is progressing or requires active treatment; exceptions include basal cell carcinoma of the skin or squamous cell carcinoma of the skin that has undergone potentially curative therapy or in situ cervical cancer * Has known active central nervous system (CNS) lymphoma * Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs); replacement therapy (thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment; conditions expected to not recur in the absence of an external trigger * Has an active infection requiring intravenous systemic therapy * Is unable to swallow capsules or malabsorption syndrome, disease or condition significantly affecting gastrointestinal function * Clinically significant cardiovascular disease with uncontrolled arrhythmia, New York Association class 3 or 4 congestive heart failure, history of myocardial infarction within 6 months, or prolonged corrected QT (QTc) \> 500 msec * Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment * Has progressed on prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent * Has progressed on prior therapy with lenalidomide * Has a known history of human immunodeficiency virus (HIV) (HIV 1/2 antibodies) * Has known active hepatitis B (e.g., hepatitis B surface antigen \[HBsAg\] reactive) or hepatitis C (e.g., hepatitis C virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] is detected) * Has a history of deep venous thrombosis/embolism, threatening thromboembolism or known thrombophilia or are at a high risk for a thromboembolic event in the opinion of the investigator and who are not willing/able to take venous thromboembolic event prophylaxis during the entire treatment period * Has a history of prior allogeneic hematopoietic cell transplant with a history of prior Graft Versus Host Disease