Brentuximab Vedotin and Pembrolizumab in Treating Patients With Recurrent Peripheral T-Cell Lymphoma

Inclusion Criteria: * Patients must have a histologically-confirmed diagnosis of CD30- positive/expressing peripheral T-cell lymphoma (PTCL). NOTE: All PTCL subtypes are eligible, except for adult T-cell leukemia/Lymphoma (ATLL) and cutaneous T-cell lymphoma (CTCL); ATLL and CTCL are excluded per exclusion criterion below. Examples of eligible subtypes include but are not limited to the following: * AITL: Angioimmunoblastic T-cell lymphoma * EATL: Enteropathy-associated T-cell lymphoma * ENKTL: Extranodal Natural Killer/T-cell Lymphoma * FTCL: Follicular T-cell lymphoma * HSTCL: Hepatosplenic T-cell lymphoma * PTCL-NOS: Peripheral T-cell lymphoma, not otherwise specified * PTCL-TFH: Nodal peripheral T-cell lymphoma with T-follicular helper phenotype * SPTCL: Subcutaneous Panniculitis-like T-cell Lymphoma * ALCL: Anaplastic Large Cell Lymphoma NOTE: CD30-positivity is defined as \>= 1% of cells expressing CD30 as detected by immunohistochemistry (IHC) and determined by local review. * Patients must have received at least one prior line of systemic therapy and must have relapsed disease or secondary refractory disease meeting one of the below criteria: * Disease that relapsed within \> 6 months after completion of frontline therapy; or * Disease that relapsed within any time after completion of secondary/subsequent lines of therapy; or * Disease that was refractory to secondary/subsequent lines of therapy NOTE: Patients who have primary relapsed/refractory disease with relapse within 6 months of frontline treatment are not eligible. See exclusion criterion below NOTE: Exclusions on receipt of prior brentuximab vedotin are described in exclusion criterion below. Exclusions on receipt of prior immunotherapy are described in exclusion criterion below. * Patients must be \>= 18 years of age * Patients must have an Eastern Cooperative Oncology Group (ECOG) performance status 0-1 * Absolute neutrophil count (ANC) \>= 1,000/mcL * Platelets \>= 50,000/mcL * Hemoglobin \>= 9 g/dL or \>= 5.6 mmol/L (without transfusion or erythropoietin-dependency within =\< 7 days prior to assessment) * Measured or calculated creatinine clearance \>= 60 mL/min * Creatinine clearance should be calculated per institutional standard * Serum total bilirubin =\< 1.5 X upper limit of normal (ULN) OR direct bilirubin =\< ULN for patients with total bilirubin levels \> 1.5 ULN * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) =\< 2.5 X ULN OR =\< 5 X ULN for patients with liver metastases * Female patients of reproductive potential must have a negative urine or serum pregnancy test within 72 hours prior to registration. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required. NOTE: In the event that 72 hours have elapsed between the screening pregnancy test and the first dose of study treatment, another pregnancy test (urine or serum) must be performed and must be negative in order for the subject to start receiving study medication * Female patients of reproductive potential must be willing to use an adequate method of contraception starting \>= 7 days prior to the first dose of study therapy and through 23 weeks after the last dose. Male patients of reproductive potential must agree to use an adequate method of contraception starting \>= 7 days prior to the first dose of study therapy and through 31 weeks after the last dose. NOTE: Abstinence is acceptable if this is the usual lifestyle and preferred contraception for the patient * Patients must have an fludeoxyglucose F 18-positron emission tomography-computed tomography (18FDG-PET-CT) scan (preferred) or CT scan of chest, abdomen, and pelvis (and neck if clinically indicated) at baseline and must have measurable disease per 2014 Lugano Criteria. The same imaging modality should be used throughout the course of study treatment to assess tumor response. NOTE: Imaging with contrast is preferred, but imaging without contrast will be accepted if the use of contrast is not clinically indicated * Patients must have the ability to understand and the willingness to sign a written informed consent form prior to registration on study Exclusion Criteria: * Patients who have received prior systemic anti-cancer therapy (including investigational agents) within 4 weeks prior to registration are not eligible. NOTE: Patients must have recovered from all adverse events due to previous therapies to =\< grade 1 or baseline to be eligible. (Exception: =\< grade 2 alopecia is permitted). NOTE: If a patient underwent a major surgery, he/she must have recovered adequately from the toxicity and/or complications from the surgical intervention prior to starting study treatment * Patients who have received prior radiotherapy within =\< 2 weeks prior to registration are not eligible. EXCEPTION: A \>= 1-week washout is permitted for palliative radiation (=\< 2 weeks of radiotherapy) to non-central nervous system \[CNS\] disease. NOTE: Patients must have recovered from all radiation-related toxicities, not require corticosteroids, and not have had radiation pneumonitis * Patients with adult T-cell leukemia/lymphoma (ATLL) or cutaneous T-cell lymphoma are not eligible * Patients with a history of (non-infectious) pneumonitis/interstitial lung disease that required steroids or with current pneumonitis/interstitial lung disease are not eligible * Patients with a history of allogeneic stem cell transplant or graft-versus host-disease (GvHD) within =\< 5 years prior to registration are not eligible * Patients who have received prior therapy with an anti-PD-1, anti-PD-L1, or anti PD-L2 agent, or with an agent directed to another stimulatory or co-inhibitory T-cell receptor (e.g., CTLA-4, OX40, CD137) are not eligible * Patients with known, active central nervous system (CNS) metastases and/or carcinomatous meningitis are not eligible. NOTE: Patients with previously treated brain metastases may participate, provided they are stable (without evidence of progression by imaging for at least four weeks prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 7 days prior to trial treatment. This exception does not include carcinomatous meningitis, which is excluded regardless of clinical stability * Patients with a diagnosis of immunodeficiency or who are receiving systemic steroid therapy or any other form of immunosuppressive therapy within =\< 7 days prior to the first dose of trial treatment are not eligible. EXCEPTIONS: Short term steroid preparation prior to tumor imaging is permitted for prophylaxis (e.g., contrast dye allergy) * Patients with active autoimmune disease that has required systemic treatment (i.e., with the use of disease modifying agents, corticosteroids or immunosuppressive drugs) within =\< 2 years prior to registration are not eligible. Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment * Patients with a history of progressive multifocal leukoencephalopathy (PML) are not eligible * Patients with a history of pancreatitis are not eligible * Patients with pre-existing \>= grade 2 peripheral neuropathy are not eligible * Patients who have a known, additional, active malignancy that is progressing or that requires active treatment are not eligible. Exceptions include basal cell carcinoma of the skin or squamous cell carcinoma of the skin that has undergone potentially curative therapy and in situ cervical cancer * Patients with a known human immunodeficiency (HIV) infection or active Bacillus Tuberculosis (TB) are not eligible. NOTE: No testing for HIV or TB is required, unless mandated by a local health authority * Patients with a known history of Hepatitis B (defined as Hepatitis B surface antigen \[HBsAg\] reactive) or a known, active Hepatitis C virus infection (defined as HCV ribonucleic acid \[RNA\] \[qualitative\] is detected) are not eligible. NOTE: No testing for Hepatitis B or Hepatitis C is required, unless mandated by a local health authority * Patients with a known hypersensitivity to pembrolizumab, brentuximab vedotin, or any of their excipients are not eligible * Patients who have received a live vaccine or live-attenuated vaccine within ≤ 30 days prior to registration are not eligible. Administration of killed vaccines is allowed. * Patients who are pregnant, breastfeeding or expecting to conceive or father children within the projected duration of the trial, starting \>= 7 days prior to the first dose of study therapy and through 23 weeks after the last dose (for females) or through 31 weeks after the last dose (for males) are not eligible * Patients who are unwilling or unable to comply with the protocol or have a known psychiatric illness or substance abuse disorder that would interfere with cooperation with the requirements of the trial are not eligible * Patients who have an uncontrolled intercurrent illness, as determined by treating investigator judgement, including but not limited to any of the following, are not eligible: * Hypertension that is not controlled on medication, * Ongoing or active infection requiring systemic treatment within ≤ 3 days prior to registration (other than uncomplicated urinary tract infection), * Severe cardiac disease, such as symptomatic congestive heart failure, unstable/uncontrolled angina pectoris, and unstable/uncontrolled cardiac arrhythmia, * Severe renal impairment, * Moderate or severe hepatic impairment (e.g., Child-Pugh B or C) * History or current evidence of any other illness, condition, therapy, or laboratory abnormality that the treating investigator feels would interfere with study compliance, would compromise the patient's safety or study endpoints, would not be in the best interest of the patient, would confound the results of the trial, or would interfere with the patient's participation for the full duration of the trial * Patients who have primary relapsed/refractory disease with relapse within 6 months of receipt of frontline treatment are not eligible * Patients who were refractory to a prior brentuximab vedotin-containing regimen are not eligible. (Note: Patients who previously received a brentuximab vedotin-containing regimen and who experienced stable disease for \>= 3 months or better as best response are eligible)