Talimogene Laherparepvec and Nivolumab in Treating Patients With Refractory Lymphomas or Advanced or Refractory Non-melanoma Skin Cancers

Inclusion Criteria: * Patients must have histologic or cytologic diagnosis of non-melanoma skin cancer (NMSC) or lymphomas other than B-cell lymphomas; as both of those terms are categories rather than specific diagnoses, specific guidance on eligible tumor types is provided below * PART I (before February 2020 amendment): Included tumor types * T cell and NK cell lymphomas, including, but not limited to: cutaneous T-cell lymphomas (CTCL), mycosis fungoides (MF), Sezary syndrome (SS), peripheral T-cell lymphoma (PTCL), ALK-positive and ALK-negative anaplastic large cell lymphoma (ALCL), and NK-cell lymphomas * Merkel cell carcinoma * Squamous cell carcinoma of the skin, including keratoacanthomas, vulvar squamous carcinoma, and mixed histology tumors, such as basosquamous carcinoma, and squamous cell carcinoma of unknown primary consistent with skin origin * Other non-melanoma skin cancers * Basal cell carcinoma * Malignant sweat gland tumors, including porocarcinoma, hidradenocarcinoma, spiradenocarcinoma, cylindrocarcinoma, microcystic adnexal carcinoma and related entities, squamoid eccrine ductal carcinoma, cutaneous adenoid cystic carcinoma, digital papillary adenocarcinoma, primary cutaneous mucinous carcinoma, endocrine mucin-producing sweat gland carcinoma, primary cutaneous signet ring cell carcinoma, cutaneous apocrine gland carcinoma, and extraocular sebaceous carcinoma * Adnexal carcinoma * Trichilemmal carcinoma * Extramammary Paget's disease * Any other rare tumor of the skin with approval of principle investigator (PI) * PART II (after February 2020 amendment): * The Merkel cell carcinoma (MCC)-2 cohort will include patients with MCC * The squamous cell carcinoma (SCC)-2 cohort will include patients with SCC * PART I (before February 2020 amendment): Patients with T cell and natural killer (NK) cell lymphomas must be refractory to, be intolerant of, have relapsed following, or have refused all standard life-prolonging therapies * PART I (before February 2020 amendment): Patients with non-melanoma skin cancers (NMSC) must have advanced or refractory tumors * Advanced/unresectable is defined by at least 1 of the following criteria: tumors 2 cm or more, tumors considered unresectable, tumors invading deep tissues such as muscle, cartilage or bone, tumors showing perineural invasion, and/or tumors metastatic to loco-regional lymph nodes and/or distant sites * Refractory is defined by persistent or recurrent tumor despite prior therapy consisting of at least 1 of the following: surgery, radiation therapy, intralesional therapy, topical therapy, or systemic therapy * PART I (before February 2020 amendment): Subjects must have at least 1 cutaneous, subcutaneous, or nodal lesion that is suitable for intralesional injection, with or without the use of ultrasound; lesions in mucosal surfaces (periocular, nasal, etc) are not eligible for injection because the area cannot be properly contained with an occlusive dressing * PART I (before February 2020 amendment): Subjects must have radiographically or clinically measurable disease, defined as at least one lesion that is \>= 10 mm in diameter in at least 1 dimension, or an aggregate of lesions that measures \>= 10 mm in diameter in at least 1 dimension * PART I (before February 2020 amendment): Subjects must be able and willing to undergo serial biopsies of injected lesion(s) and, when applicable and clinically feasible, non-injected lesions * PART I (before February 2020 amendment): Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%) * PART I (before February 2020 amendment): Absolute neutrophil count (ANC) \>= 1.2 x 10\^9/L * PART I (before February 2020 amendment): Hemoglobin \>= 9 g/dL without transfusion in the preceding 7 days * PART I (before February 2020 amendment): Platelets \>= 75 x 10\^9/L * PART I (before February 2020 amendment): Serum total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (patients with Gilbert's syndrome with a total bilirubin \< 3.0 mg/dL) * PART I (before February 2020 amendment): Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x institutional ULN * PART I (before February 2020 amendment): Serum creatinine =\< 1.5 mg/dL OR calculated creatinine clearance (Cockcroft-Gault formula) \>= 50 mL/min OR 24-hour urine creatinine clearance \>= 50 mL/min * PART I (before February 2020 amendment): Prothrombin time (PT)/international normalized ratio (INR) and partial thromboplastin time (PTT) =\< 1.5 x institutional ULN, unless the subject is on anticoagulant therapy; (if the subject is receiving anticoagulant therapy, PT, and activated PTT \[aPTT\] must be within therapeutic range of intended use of anticoagulants) * PART I (before February 2020 amendment): Talimogene laherparepvec, nivolumab and other therapeutic agents used in this trial may cause fetal harm when administered to a pregnant woman; women of child-bearing potential (WOCBP) and must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence from heterosexual intercourse) prior to study entry, during the study participation, and for 7 months after the last dose of the drug; WOCBP must have a negative serum pregnancy test within 14 days prior to randomization; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately; men must agree to use adequate contraception prior to study entry, during the study participation and for 7 months after the last dose of the drug * PART I (before February 2020 amendment): Ability to understand and the willingness to sign a written informed consent document * PART II (after February 2020 amendment): Subjects in expansion cohorts MCC-2 and SCC-2 must have a diagnosis of MCC or SCC, respectively * PART II (after February 2020 amendment): Subjects must have refractory disease, defined as evidence of progressive disease despite prior therapy with a PD-1 or PD-L1 blocking antibody (avelumab, pembrolizumab, nivolumab, cemiplimab, etc.); progression must have occurred during PD-1 or PD-L1 directed therapy or within 6 months of the last dose of PD-1 or PD-L1 directed therapy * PART II (after February 2020 amendment): Subjects must have at least 1 cutaneous, subcutaneous, or nodal lesion that is suitable for intralesional injection, with or without the use of ultrasound; lesions in mucosal surfaces (periocular, nasal, etc) are not eligible for injection because the area cannot be properly contained with an occlusive dressing * PART II (after February 2020 amendment): Subjects must have radiographically or clinically measurable disease, defined as at least one lesion that is \>= 10 mm in diameter in at least 1 dimension, or an aggregate of lesions that measures \>= 10 mm in diameter in at least 1 dimension * PART II (after February 2020 amendment): Subjects must be able and willing to undergo serial biopsies of injected lesion(s) and, when applicable and clinically feasible, non-injected lesions * PART II (after February 2020 amendment): ECOG performance status =\< 2 (Karnofsky \>= 60%) * PART II (after February 2020 amendment): Absolute neutrophil count (ANC) \>= 1.2 x 10\^9/L * PART II (after February 2020 amendment): Hemoglobin \>= 9 g/dL without transfusion in the preceding 7 days * PART II: P (after February 2020 amendment): Platelets \>= 75 x 10\^9/L * PART II (after February 2020 amendment): Serum total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (Patients with Gilbert's Syndrome with a total bilirubin \< 3.0 mg/dL.) * PART II (after February 2020 amendment): Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x institutional ULN * PART II (after February 2020 amendment): Serum creatinine =\< 1.5 mg/dL OR calculated creatinine clearance (Cockcroft-Gault formula) \>= 50 mL/min OR 24-hour urine creatinine clearance \>= 50 mL/min * PART II (after February 2020 amendment): Prothrombin time (PT)/international normalized ratio (INR) and partial thromboplastin time (PTT) =\< 1.5 x institutional ULN, unless the subject is on anticoagulant therapy; (if the subject is receiving anticoagulant therapy, PT, and aPTT must be within therapeutic range of intended use of anticoagulants) * PART II (after February 2020 amendment): Talimogene laherparepvec, nivolumab and other therapeutic agents used in this trial may cause fetal harm when administered to a pregnant woman; women of child-bearing potential (WOCBP) and must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence from heterosexual intercourse) prior to study entry, during the study participation, and for 7 months after the last dose of the drug; WOCBP must have a negative serum pregnancy test within 14 days prior to randomization; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately; men must agree to use adequate contraception prior to study entry, during the study participation and for 7 months after the last dose of the drug * PART II (after February 2020 amendment): Ability to understand and the willingness to sign a written informed consent document Exclusion Criteria: * Excluded tumor types * Melanoma * Bone sarcomas * Soft tissue sarcomas, including angiosarcoma, primary cutaneous leiomyosarcoma, dermatofibrosarcoma protuberans * Leukemias * Myeloid sarcoma, leukemia cutis, and chloroma * Hodgkin's lymphoma * B cell lymphoma * Patients who have had systemic therapy or radiotherapy within 3 weeks prior to the first dose of study therapy * Untreated central nervous system (CNS) involvement; patients with known brain metastases are eligible if they have been treated and are stable in the view of the treating investigator * Previous treatment with talimogene laherparepvec or other herpes virus based therapy; (prior therapy with checkpoint inhibitors and/or other immunotherapy is allowed) * Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1 excepting alopecia, peripheral sensory neuropathy, and stable endocrine insufficiencies such as thyroid and adrenal insufficiency) * Second primary malignancy, only if it would affect the safety of the treatment or the subject's ability to complete study-related procedures * History or evidence of active autoimmune disease (e.g., pneumonitis, glomerulonephritis, vasculitis, or other); or history of active autoimmune disease that has required systemic treatment (i.e., use of corticosteroids, immunosuppressive drugs or biological agents used for treatment of autoimmune diseases) within 2 months of enrollment; (replacement therapy \[e.g., thyroxine for hypothyroidism, insulin for diabetes or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency\] is not considered a form of systemic treatment for autoimmune disease) * Evidence of clinically significant immunosuppression such as the following: * Primary immunodeficiency state such as severe combined immunodeficiency disease * Receiving systemic immunosuppressive therapy including prednisone \> 10 mg per day (or equivalent), tacrolimus, everolimus, sirolimus, mycophenolate mofetil, etanercept, infliximab, etc. * Recipients of solid organ, bone marrow, or stem cell transplants; auto transplant recipients are allowed * Notes: Oral steroid doses =\< 10 mg/day of prednisone (or equivalent) are not considered immunosuppressive and are permitted; inhaled and intraarticular corticosteroids are permitted * Active herpetic skin lesions or prior complications of herpetic infection (e.g., herpetic keratitis or encephalitis) * Viral infections requiring intermittent or chronic systemic (intravenous or oral) treatment with an antiherpetic drug (e.g., acyclovir), other than intermittent topical use * Other viral infections: * Known to have acute or chronic active hepatitis B or hepatitis C infection * Known to have human immunodeficiency virus (HIV) infection * Prior therapy with viral-based tumor vaccine * Received live vaccine within 28 days prior to enrollment * Subject who is unwilling to minimize exposure with his/her blood or other body fluids to individuals who are at higher risks for human herpesvirus 1 (HSV-1) induced complications such as immunosuppressed individuals, individuals known to have HIV infection, pregnant women, or children under the age of 1 year, during talimogene laherparepvec treatment and through 30 days after the last dose of talimogene laherparepvec * Has a history of (non-infectious) pneumonitis that required steroids or current pneumonitis * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Female subject is pregnant or breast-feeding, or planning to become pregnant during study treatment and through 7 months after the last dose of treatment; female subject of childbearing potential who is unwilling to use acceptable method(s) of effective contraception during study treatment and through 7 months after the last dose of treatment; sexually active subjects and their partners unwilling to use male or female latex condom to avoid potential viral transmission during sexual contact while on treatment and within 30 days after treatment with talimogene laherparepvec * Have a known immediate or delayed hypersensitivity reaction or idiosyncrasy to talimogene laherparepvec or any of its components or nivolumab, or history of severe hypersensitivity reaction to any monoclonal antibody