Pembrolizumab and External Beam Radiation Therapy in Treating Patients With Relapsed or Refractory Non-Hodgkin Lymphoma

Inclusion Criteria: * Have at least one site of lymphomatous disease amenable to external beam radiation therapy (EBRT) * Have pathologic confirmation of aggressive non-Hodgkin lymphoma (including diffuse large B cell lymphoma, transformed follicular lymphoma, transformed marginal zone lymphoma, primary mediastinal B-cell lymphoma, T cell lymphoma and NK T-cell lymphoma). Patients with indolent B cell lymphoma are excluded * Be willing and able to provide written informed consent/assent for the trial * Have measurable disease (\>= 1.5 cm in the longest diameter for nodal or extranodal disease) * Have provided archival tumor tissue sample or newly obtained core or excisional biopsy of a tumor lesion not previously irradiated. Formalin-fixed, paraffin embedded (FFPE) tissue blocks are preferred to slides. Newly obtained biopsies are preferred to archived tissue. Newly-obtained is defined as a specimen obtained up to 6 weeks (42 days) prior to initiation of treatment on day 1. Subjects for whom newly-obtained samples cannot be provided (e.g. inaccessible or subject safety concern) may submit an archived specimen. Note: If submitting unstained cut slides, newly cut slides should be submitted to the testing laboratory within 14 days from the date slides are cut * Have an Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 1. Evaluation of ECOG is to be performed within 7 days prior to the date of allocation/randomization * Absolute neutrophil count (ANC) \>= 1,000 /mcL (performed within 10 days of treatment initiation) * Platelets \>= 50,000 / mcL (performed within 10 days of treatment initiation) * Hemoglobin \>= 8 g/dL or \>= 5.6 mmol/L without transfusion or erythropoietin (EPO) dependency (within 7 days of assessment) (performed within 10 days of treatment initiation) * Serum creatinine OR measured or calculated creatinine clearance (creatinine clearance should be calculated per institutional standard) (glomerular filtration rate \[GFR\] can also be used in place of creatinine or creatinine clearance \[CrCl\]) =\< 1.5 x upper limit of normal (ULN) OR \>= 60 mL/min for subject with creatinine levels \> 1.5 x institutional ULN (performed within 10 days of treatment initiation) * Serum total bilirubin =\< 1.5 x ULN OR direct bilirubin =\< ULN for subjects with total bilirubin levels \> 1.5 ULN (performed within 10 days of treatment initiation) * Aspartate aminotransferase (AST) serum glutamic-oxaloacetic transaminase (SGOT) and alanine aminotransferase (ALT) serum glutamic pyruvic transaminase (SGPT) =\< 2.5 x ULN OR =\< 5 x ULN for subjects with liver metastases (performed within 10 days of treatment initiation) * Albumin \>= 2.5 mg/dL (performed within 10 days of treatment initiation) * International normalized ratio (INR) or prothrombin time (PT) =\< 1.5 x ULN unless subject is receiving anticoagulant therapy as long as PT or partial thromboplastin time (PTT) is within therapeutic range of intended use of anticoagulants (performed within 10 days of treatment initiation) * Activated partial thromboplastin time (aPTT) =\< 1.5 x ULN unless subject is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants (performed within 10 days of treatment initiation) * Female subject of childbearing potential should have a negative urine or serum pregnancy within 72 hours prior to receiving the first dose of study medication. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Female subjects of childbearing potential must be willing to use an adequate method of contraception as outlined - contraception, for the course of the study through 120, corresponding to time needed to eliminate any Merck study treatment(s) and/or any active comparator/combination, plus 30 days (a menstruation cycle) for study treatments with risk of genotoxicity days after the last dose of study medication. Note: Abstinence is acceptable if this is the usual lifestyle and preferred contraception for the subject Exclusion Criteria: * Has had prior radiation therapy to the potential radiation target such that additional radiation therapy is considered unsafe by the treating radiation oncologist * Has a history of allogeneic stem cell transplantation * Has a diagnosis of active scleroderma or lupus or any other autoimmune disease that by the opinion of the treating radiation oncologist would put the patient at unacceptable risk of toxicity * Is currently participating and receiving study therapy or has participated in a study of an investigational agent and received study therapy or used an investigational device within 4 weeks of the first dose of treatment * Has a diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment * Has a known history of active Bacillus tuberculosis (TB) * Hypersensitivity to pembrolizumab or any of its excipients * Has had a prior anti-cancer monoclonal antibody (mAb) within 4 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier * Has had prior chemotherapy, targeted small molecule therapy, or radiation therapy within 2 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events due to a previously administered agent * Note: subjects with =\< grade 2 neuropathy are an exception to this criterion and may qualify for the study * Note: if subject received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy * Has a known additional malignancy that is progressing or requires active treatment within the past 3 years. Exceptions include basal cell carcinoma of the skin or squamous cell carcinoma of the skin that has undergone potentially curative therapy or in situ cervical cancer * Has known active central nervous system (CNS) lymphoma or lymphomatous meningitis. Subjects with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least four weeks prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 7 days prior to trial treatment * Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment * Has history of (non-infectious) pneumonitis that required steroids, evidence of interstitial lung disease or active, non-infectious pneumonitis * Has an active infection requiring systemic therapy * Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the subject's participation for the full duration of the trial, or is not in the best interest of the subject to participate, in the opinion of the treating investigator * Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial * Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment * A woman of child bearing potential (WOCBP) who has a positive urine pregnancy test within 72 hours prior to receiving the first dose of study medication. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required. Note: in the event that 72 hours have elapsed between the screening pregnancy test and the first dose of study treatment, another pregnancy test (urine or serum) must be performed and must be negative in order for subject to start receiving study medication * Has received prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent directed to another stimulatory or co-inhibitory T-cell receptor (e.g., CTLA-4, OX-40, CD137) * Has a known history of human immunodeficiency virus (HIV) (HIV 1/2 antibodies) * Has known active hepatitis B (e.g., hepatitis B surface antigen \[HBsAg\] reactive) or hepatitis C (e.g., hepatitis C virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] is detected) * Has received a live vaccine within 30 days of planned start of study therapy. Note: seasonal influenza vaccines for injection are generally inactivated flu vaccines and are allowed; however intranasal influenza vaccines (e.g., Flu-Mist) are live attenuated vaccines, and are not allowed