Rituximab, Lenalidomide, and Nivolumab in Treating Participants With Relapsed or Refractory Non-Germinal Center Type Diffuse Large B Cell Lymphoma or Primary Central Nervous System Lymphoma

Inclusion Criteria: * Understand and voluntarily sign an informed consent form. * Able to adhere to the study visit schedule and other protocol requirements in the opinion of the investigator * Patients with histological confirmation of relapsed/refractory non-GCB type (using Hans algorithm) diffuse large B cell lymphoma (DLBCL) or relapsed/refractory primary CNS lymphoma (PCNSL) with at least one of the following characteristics: * Definition of refractory disease: progression of disease based on Cheson criteria for DLBCL or international primary CNS lymphoma cooperative group for PCNSL either with nonresponse or progression within 3 months of prior therapy * Definition of relapsed disease: progression of disease based on Cheson criteria for DLBCL or International primary CNS lymphoma cooperative group for PCNSL at least 3 months after prior therapy * Definition of non-GCB subtype (Hans algorithm): cases will be subclassified based on immunohistochemical staining with CD10, BCL-6 and MUM-1 as previously described. * Patients should have exhausted (or be ineligible for) approved therapies known to provide clinical benefit for DLBCL or PCNSL (e.g. high dose chemotherapy with autologous stem cell transplant, chimeric antigen receptor-transduced \[CAR-T\] therapy, etc.). * Eastern Cooperative Oncology Group (ECOG) performance status of =\< 2 at study entry. * Absolute neutrophil count \>= 1000/mm\^3 within 45 days prior to initiation of therapy. (Patients with documented marrow involvement (with lymphoma) or hypersplenism secondary to involvement of the spleen by lymphoma at the time of randomization are not required to meet the parameter). * Platelet count \>= 75K /mm\^3 within 45 days prior to initiation of therapy. (Patients with documented marrow involvement (with lymphoma) or hypersplenism secondary to involvement of the spleen by lymphoma at the time of randomization are not required to meet the parameter). * Serum creatinine =\< 2.0 mg/dL or creatinine clearance of \> 40 ml/min within 45 days prior to initiation of therapy. (Patients with documented marrow involvement (with lymphoma) or hypersplenism secondary to involvement of the spleen by lymphoma at the time of randomization are not required to meet the parameter). * Total bilirubin =\< 1.5 mg/dL within 45 days prior to initiation of therapy. (Patients with documented marrow involvement (with lymphoma) or hypersplenism secondary to involvement of the spleen by lymphoma at the time of randomization are not required to meet the parameter). * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2 x upper limit of normal (ULN) within 45 days prior to initiation of therapy. (Patients with documented marrow involvement (with lymphoma) or hypersplenism secondary to involvement of the spleen by lymphoma at the time of randomization are not required to meet the parameter). * Disease free of prior malignancies for \>= 3 years with exception of currently treated basal cell or squamous cell carcinoma of the skin, or carcinoma "in situ" of the cervix or breast. * All study participants must be registered into the mandatory Revlimid Risk Evaluation and Mitigation Strategies (REMS) program, and be willing and able to comply with the requirements of the Revlimid REMS program. program, and be willing and able to comply with the requirements of the Revlimid REMS program. \* Females of childbearing potential (FCBP) must have a negative serum or urine pregnancy test with a sensitivity of at least 50 mIU/mL within 10-14 days and again within 24 hours prior to prescribing lenalidomide for cycle 1 (prescriptions must be filled within 7 days as required by the Revlimid REMS program) and must either commit to continued abstinence from heterosexual intercourse or begin TWO acceptable methods of birth control, one highly effective method and one additional effective method AT THE SAME TIME, at least 28 days before she starts taking lenalidomide. FCBP must also agree to ongoing pregnancy testing. Men must agree to use a latex condom during sexual contact with a FCBP even if they have had a successful vasectomy. \*\* A female of childbearing potential is a sexually mature female who: 1) has not undergone a hysterectomy or bilateral oophorectomy; or 2) has not been naturally postmenopausal for at least 24 consecutive months (i.e., has had menses at any time in the preceding 24 consecutive months). * Able to take aspirin (81 or 325 mg) daily or for thromboprophylaxis with lenalidomide. Exclusion Criteria: * Any serious medical condition, laboratory abnormality, or psychiatric illness that would prevent the subject from signing the informed consent. * Pregnant or breast feeding females (lactating females must agree not to breast feed while taking lenalidomide). * Any condition, including the presence of laboratory abnormalities, which places the subject at unacceptable risk if he/she were to participate in the study or confounds the ability to interpret data from the study. * Use of any other experimental drug or therapy within 28 days of baseline. * Known hypersensitivity to thalidomide. * Known prior development of erythema nodosum if characterized by a desquamating rash while taking thalidomide or similar drugs. * Concurrent use of other anti-cancer agents or treatments. * Patients with a prior history of pulmonary toxicity due to medications (Ex: history of carmustine \[BCNU\] toxicity). * Active human immunodeficiency virus (HIV) infection or infectious hepatitis, type B or C. * Patients with a past or resolved hepatitis B virus (HBV) infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of hepatitis B surface antigen \[HBsAg\]) may be included if HBV deoxyribonucleic acid (DNA) is undetectable. If enrolled, patients must be willing to undergo monthly HBV DNA testing. * HIV positive patients may enroll if they meet all of the below criteria: * HIV is sensitive to antiretroviral therapy. * Must be willing to take effective antiretroviral therapy if indicated. * No history of CD4 prior to or at the time of lymphoma diagnosis \< 300 cells/mm\^3. * No history of acquired immunodeficiency syndrome (AIDS)-defining conditions. * If on antiretroviral therapy, must not be taking zidovudine or stavudine. * Must be willing to take prophylaxis for pneumocystis jiroveci pneumonia (PCP) during therapy and until at least 2 months following the completion of therapy or until the CD4 cells recover to over 250 cells/mm\^3, whichever occurs later. * A diagnosis of deep vein thromboses in the preceding four weeks of study enrollment. * Subjects with active interstitial pneumonitis. * Patient with active, uncontrolled infections (patients must be afebrile for \> 48 hours off antibiotics). * Patient with clinically significant history of liver disease, including viral or other hepatitis, current alcohol abuse, or cirrhosis. * Subjects with active, known or suspected autoimmune disease. Subjects with vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll. * Evidence of significant, uncontrolled concomitant diseases that could affect compliance with the protocol or interpretation of results or that could increase risk to the patient, including renal disease that would preclude chemotherapy administration or pulmonary disease (including obstructive pulmonary disease and history of bronchospasm). * Major surgery within 4 weeks prior to cycle 1, other than for diagnosis. * Malabsorption syndrome or other condition that precludes oral route of administration of lenalidomide. * Vaccinations with a live vaccine within 28 days prior to start of treatment or need for live vaccine at any time during study treatment. * Patient must not have serious medical or psychiatric illness likely to interfere with participation in this clinical study in the opinion of the investigator. * Inability to swallow oral medications. * Subjects with New York Heart Association (NYHA) class III or IV heart failure.