Mogamulizumab and Pembrolizumab in Treating Patients With Relapsed or Refractory Diffuse Large B Cell Lymphoma

Inclusion Criteria: * Patients must have histologically confirmed diffuse large B-cell lymphoma; all subtypes of diffuse large B-cell lymphoma are eligible, including high-grade B-cell lymphoma and diffuse large B-cell lymphoma (DLBCL) that has transformed from a prior indolent B-cell non-Hodgkin lymphoma * Patients must have measurable disease per 2014 Lugano Classification Criteria which is defined as at least one nodal lesion measuring \> 1.5 cm in greatest diameter or at least one extranodal lesion measuring \> 1.0 cm in greatest diameter * For phase 2: patients and received at least 2 prior lines of therapy and must have previously received, refused, or been deemed ineligible for autologous stem cell transplantation * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 80%) * Absolute neutrophil count \>= 1,500/mcL (if neutropenia is related to bone marrow involvement with lymphoma, the absolute neutrophil count must be \>= 1,000/mcL) * Platelets \>= 75,000/mcL (if thrombocytopenia is related to bone marrow involvement with lymphoma, the platelet count must be \>= 50,000/mcL) * Hemoglobin \>= 9 g/dL (if anemia is related to bone marrow involvement with lymphoma, the hemoglobin must be \>= 8 g/dL) * Total bilirubin =\< 1.5 x the institutional upper limit of normal (ULN) or \< 3 x the ULN for indirect bilirubin in patients with Gilbert's disease * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x institutional upper limit of normal * Creatinine =\< 1.5 x institutional upper limit of normal OR measured or calculated creatinine clearance if creatinine \> 1.5 x ULN then creatinine clearance \>= 40 mL/min/1.73 m\^2 as calculated by Cockcroft and Gault equation * Life expectancy of greater than 3 months * The effects of MK-3475 (pembrolizumab) in combination with KW-0761 (mogamulizumab) on the developing human fetus are unknown; for this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation, and 6 months after completion of MK-3475 (pembrolizumab) in combination with KW-0761 (mogamulizumab) administration; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately; men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 6 months after completion of MK-3475 (pembrolizumab) in combination with KW-0761 (mogamulizumab) administration * Submit adequate archival tissue specimen (25+ unstained slides or 2 tissue blocks) from a biopsy performed after progression of disease on most recent therapy OR subject is willing to undergo a new core or excisional biopsy to obtain evaluable tumor tissue sample for immunohistochemical assessment and sequencing for B2M loss; repeat samples may be required if adequate tissue is not provided, however, patients may still be considered for enrollment on a case by case basis following consultation with the principal investigator (PI) * Ability to understand and the willingness to sign a written informed consent document * Subjects with prior history of chemotherapy-induced or radiation-induced pulmonary toxicity require confirmation of diffuse capacity of the lung for carbon monoxide (DLCO) over 60% (adjusted for hemoglobin) by a pulmonary function test prior to study enrollment Exclusion Criteria: * Patients who have had previous systemic anti-cancer therapy within 3 weeks of registration or those who have not recovered from adverse events due to agents administered previously * Note: Patients are considered enrolled on the study after protocol registration and not after signing consent * Patients who are receiving any other concurrent investigational agents * Patient is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment; the use of physiologic doses of corticosteroids (e.g. prednisone =\< 20 mg/d) may be approved after consultation with the study PI; topical or inhaled corticosteroids are allowed * Has a known additional malignancy that is progressing or requires active treatment; exceptions include basal cell carcinoma of the skin, squamous cell carcinoma of the skin that has undergone potentially curative therapy, or in situ cervical cancer * Patients with active cerebral or meningeal involvement by lymphoma should be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound evaluation of neurologic and other adverse events * History of allergic reactions attributed to compounds of similar chemical or biologic composition to MK-3475 (pembrolizumab) or KW-0761 (mogamulizumab) * Subject with active autoimmune disease; subjects with vitiligo, eczema, alopecia, type I diabetes mellitus, psoriasis not requiring systemic treatment, or endocrine deficiencies (such as hypothyroidism) managed with replacement hormones, including physiologic corticosteroid replacement therapy are eligible * Has a history or currently active (non-infectious) pneumonitis that required steroids unless prior history of chemotherapy or radiotherapy induced pneumonitis meeting the eligibility criteria * Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the patient's participation for the full duration of the trial, or is not in the best interest of the patient to participate, in the opinion of the treating investigator * Has received prior therapy with an anti-PD-1, anti-PD-L1, anti-PD-L2, or anti-CTLA-4 antibody (including ipilimumab or any other antibody or drug specifically targeting T-cell co-stimulation or checkpoint pathways) * Prior allogeneic stem cell transplant (SCT) * Patients who are planning to receive allogeneic SCT in the near future as preliminary reports suggest added toxicity in patients undergoing allogeneic stem cell transplantation after having received mogamulizumab * Autologous SCT =\< 90 days prior to first dose of study drug * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, interstitial lung disease or active, non-infectious pneumonitis, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Pregnant women are excluded from this study because MK-3475 (pembrolizumab) is an agent with the potential for teratogenic or abortifacient effects; because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with MK-3475 (pembrolizumab), breastfeeding should be discontinued if the mother is treated with MK-3475 (pembrolizumab); these potential risks may also apply to KW-0761 (mogamulizumab) * MK-3475 (pembrolizumab) and KW-0761 (mogamulizumab) may have adverse effects on a fetus in utero; furthermore, it is not known if MK-3475 (pembrolizumab) or KW-0761 (mogamulizumab) has transient adverse effects on the composition of sperm; patients are excluded from this study if pregnant or breastfeeding or expecting to conceive or father children within the projected duration of the trial, starting with the screening visit through 180 days after the last dose of trial treatment * Patients with human immunodeficiency virus (HIV) are excluded if they have a detectable viral load, are not on a stable antiretroviral regimen, have a decreased CD4+ T-cell count (\< 500), or require prophylactic antibiotics for the prevention of opportunistic infections * Has a known history of hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] reactive) or known active hepatitis C virus (defined as HCV ribonucleic acid \[RNA\] \[qualitative\] is detected) infection * Note: No testing for hepatitis B and hepatitis C is required unless mandated by local health authority * Has a known history of active tuberculosis (TB) * Patients with significant cardiac disease (e.g., New York Heart Association \[NYHA\] class III-IV congestive heart failure, unstable angina, recent myocardial infarction within the last 6 months, etc.)