Nivolumab and Pomalidomide in Treating Patients With Relapsed or Refractory Central Nervous System Diffuse Large B Cell Lymphoma or Primary Vitreoretinal Diffuse Large B Cell Lymphoma

Inclusion Criteria: * Patient must have one of the following: * Relapsed or refractory primary central nervous system (CNS) diffuse large B cell lymphoma (PCNSDLBCL) with a brain lesion \>= 1 cm, or with cerebrospinal fluid (CSF) relapse with positive CSF cytology, or with ocular relapse with positive ocular tissue biopsy * NOTE: Tissue biopsy is not absolutely necessary for brain tumor unless clinical and radiologic findings strongly suggest other etiologies as per treating physician. Initial diagnosis must be made by tissue biopsy OR * Relapsed/refractory primary vitreoretinal diffuse large B cell lymphoma (DLBCL) with a CNS lesion \>= 1 cm, or with CSF relapse with positive CSF cytology, or with ocular relapse with positive ocular tissue biopsy. Relapsed PVRL must have progressed or failed at least one systemic regimen * NOTE: Intraocular treatments are not regarded as systemic therapy * NOTE: If recurrence in ocular or leptomeningeal space, the patient will need a positive ocular tissue biopsy and CSF biopsy. Tissue biopsy requirement of the CNS lesion is as outlined in bullet above * Patient progressed after or did not respond to at least 1 line of systemic therapy (e.g., high-dose methotrexate, high-dose methotrexate based regimen, high dose cytarabine, etc) * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0, 1, 2 or 3 * Absolute neutrophil count (ANC) \>= 1500/mm\^3 (within =\< 14 days prior to registration) * Platelet count \>= 100,000/mm\^3 (within =\< 14 days prior to registration) * Hemoglobin \>= 9.0 g/dL (within =\< 14 days prior to registration) * Direct bilirubin \< 1.5 x upper limit of normal (ULN) (or total bilirubin =\< 3.0 x ULN with direct bilirubin =\< 1.5 x ULN in patients with well-documented Gilbert?s Syndrome) (within =\< 14 days prior to registration) * Aspartate transaminase (AST) =\< 3 x ULN (within =\< 14 days prior to registration) * Creatinine =\< 1.5 x ULN OR calculated creatinine clearance must be \>= 45 ml/min using the Cockcroft-Gault formula (within =\< 14 days prior to registration) * Prothrombin time (PT)/international normalized ratio (INR)/partial thromboplastin time (PTT) =\< 1.5 x ULN OR if patient is receiving anticoagulant therapy and PT or PTT is within therapeutic range of intended use of anticoagulants (within =\< 14 days prior to registration) * Women of childbearing potential (WOCBP) must have a negative serum or urine pregnancy test with a sensitivity of at least 50 mIU/ml, =\< 14 days prior to registration and within 24 hours of starting pomalidomide. In addition, must either commit to continue abstinence from heterosexual intercourse or begin TWO acceptable methods of birth control, one highly effective method and one additional effective method AT THE SAME TIME, at least 28 days before she starts taking pomalidomide. WOCBP must also agree to ongoing pregnancy testing. Men must agree to use a latex condom during sexual contact with a WOCBP even if they have had a vasectomy. In addition, women of childbearing potential (WOCBP) receiving nivolumab will be instructed to adhere to contraception for a period of 5 months after the last dose of nivolumab. Men receiving nivolumab and who are sexually active with WOCBP will be instructed to adhere to contraception for a period of 7 months after the last dose of nivolumab. All patients must be counseled at a minimum of every 28 days about pregnancy precautions and risks of fetal exposure * A woman of childbearing potential is a sexually mature woman who: 1) has not undergone a hysterectomy or bilateral oophorectomy; or 2) has not been naturally postmenopausal for at least 24 consecutive months (i.e., has had menses at any time in the preceding 24 consecutive months) * Able to take aspirin (81 or 325 mg) daily or an anticoagulant (as determined by treating physician) as prophylactic anticoagulation * Provide written informed consent * Willingness to return to enrolling institution for follow-up (during the active monitoring phase of the study) * Willingness to undergo biopsy procedures, if deemed necessary by treating physician * Willing to be registered into a mandatory POMALYST REMS program, and willing and able to comply with the requirements of the POMALYST REMS program Exclusion Criteria: * Any of the following because this study involves an investigational agent whose genotoxic, mutagenic and teratogenic effects on the developing fetus and newborn are unknown: * Pregnant women * Nursing women * Men or women of childbearing potential who are unwilling to employ adequate contraception * Co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the investigator, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens * Immunocompromised patients and patients known to be human immunodeficiency virus (HIV) positive * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Other concurrent chemotherapy, immunotherapy, radiotherapy, or any ancillary therapy considered investigational (utilized for a non-Food and Drug Administration \[FDA\]-approved indication and in the context of a research investigation) * Other active malignancy =\< 3 years prior to registration * EXCEPTIONS: Non-melanotic skin cancer or carcinoma-in-situ of the cervix * NOTE: If there is a history of prior malignancy, they must not have received immune checkpoint inhibitors or immunomodulatory therapy (IMiD) for their cancer * History of myocardial infarction =\< 6 months prior to registration, or congestive heart failure requiring use of ongoing maintenance therapy for life-threatening ventricular arrhythmias * The development of erythema nodosum if characterized by a desquamating rash while taking thalidomide or similar drugs * Use of corticosteroid in the absence of cerebral edema * NOTE: If a corticosteroid is used, it should be used at the lowest dose possible for the shortest possible duration. Subjects with a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalent) or other immunosuppressive medications within 14 days of treatment assignment are excluded * EXCEPTIONS: Inhaled or topical steroids, and adrenal replacement steroid doses \> 10 mg daily prednisone equivalent, are permitted in the absence of active autoimmune disease * Therapy with myelosuppressive chemotherapy or biologic therapy =\< 21 days prior to registration with incomplete recovery of blood counts * Persistent toxicities \>= grade 3 from prior chemotherapy or biological therapy regardless of interval since last treatment * History of thromboembolic episodes =\< 3 months prior to registration * Active hepatitis B or C with uncontrolled disease * NOTE: A detailed assessment of hepatitis B/C medical history and risk factors must be done at screening for all patients. Hepatitis B core IgM antibody (HBcIgM Ab), Hepatitis B surface antigen (HBsAg) and Hepatitis C virus (HCV) Ab screen (Scrn) w/Reflex testing are required at screening for all patients with a positive medical history based on risk factors and/or confirmation of prior HBV infection * Inability to swallow or impairment of gastrointestinal function or gastrointestinal disease that may significantly alter the absorption of the drugs (e.g., ulcerative disease, uncontrolled nausea, vomiting, diarrhea, malabsorption syndrome or small bowel resection) that would preclude use of oral medications * Major surgery =\< 4 weeks prior to registration or have not recovered from side effects of such therapy * New York Heart Association classification III or IV * Patient received radiation alone previously * NOTE: Radiation therapy would not be regarded as a systemic therapy * EXCEPTION: Patients who have received systemic therapy followed by radiation would be eligible * PCNSL with systemic disease * Inability to undergo or have a magnetic resonance imaging (MRI) performed