Pembrolizumab With Locally Delivered Radiation Therapy for the Treatment of Metastatic Esophageal Cancers

Inclusion Criteria: * Any patient with metastatic esophageal cancer that is deemed a candidate for brachytherapy for local control or treatment of dysphagia as determined by treating physician * Presence of an evaluable metastatic lesion (locoregional lymph nodes are acceptable) * At least 18 years of age. * ECOG performance status 0-2 * Adequate bone marrow and organ function as defined below: * Absolute neutrophil count ≥ 1,500/mcL * Platelets ≥ 100,000/mcL * Hemoglobin ≥ 9 g/dL * Total bilirubin ≤ 1.5 x IULN OR Direct bilirubin ≤ IULN for patients with total bilirubin \> 1.5 x IULN * AST(SGOT)/ALT(SGPT) ≤ 2.5 x IULN (or ≤ 5 x IULN for patients with liver metastases) * Serum creatinine ≤ 1.5 x IULN OR Creatinine clearance by Cockcroft-Gault ≥ 60 mL/min/1.73 m2 for patients with creatinine levels \> 1.5 x IULN * INR or PT ≤ 1.5 x IULN unless patient is receiving anticoagulant therapy as long as INR or PTT is within therapeutic range of intended use of anticoagulants * aPTT ≤ 1.5 x IULN unless patient is receiving anticoagulant therapy as long as INR or PTT is within therapeutic range of intended use of a anticoagulants * Sexually active women of childbearing potential and men must agree to use contraceptive methods prior to study entry, for the duration of study participation, and for 120 days after the last dose of pembrolizumab. Should a woman become pregnant or suspect she is pregnant while participating in this study, she must inform her treating physician immediately. * Either enrolled in HRPO# 201107221 ("Tissue and Blood Acquisition for Genomic Analysis and Collection of Health Information for Patients with Gastrointestinal Cancers"), which facilitates the collection of specimens for correlative studies, or consenting to collection of blood and tissue as part of this protocol for research testing. * Ability to understand and willingness to sign an IRB approved written informed consent document (or that of legally authorized representative, if applicable). Exclusion Criteria: * Prior treatment with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent * Received a live vaccine within 30 days prior to the first dose of pembrolizumab. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster (chicken pox), yellow fever, rabies, Bacillus Calmette-Guérin (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g. FluMist) are live attenuated vaccines and are not allowed. * Has a diagnosis of immunodeficiency or is receiving chronic systemic steroid therapy (in dosing exceeding 10 mg daily of prednisone equivalent) or any other form of immunosuppressive therapy within 7 days prior to the first dose of pembrolizumab. * Currently receiving any other investigational agents, has participated in a study of an investigational agent, or use of an investigational device within 4 weeks of the first dose of pembrolizumab. * Has received systemic therapy within 4 weeks of the first dose of pembrolizumab. * Known active central nervous system metastases and/or carcinomatous meningitis. Subjects with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least 4 weeks prior to the first dose of MK-3475 and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 7 days prior to trial treatment. This exception does not include carcinomatous meningitis which is excluded regardless of clinical stability. * A history of allergic reactions attributed to compounds of similar chemical or biologic composition to pembrolizumab or other agents used in the study. * Uncontrolled intercurrent illness that would limit compliance with study requirements. This would include, but is not limited to: ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, immunosuppression, autoimmune conditions, underlying pulmonary disease, or psychiatric illness/social situations. * Has an active autoimmune disease requiring systemic treatment within the past 2 years (i.e. with use of disease modifying agents, corticosteroids, or immunosuppressive drugs). Replacement therapy (e.g. thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. * History of (non-infectious) pneumonitis that required steroids or current pneumonitis * Pregnant and/or breastfeeding. Women of childbearing potential must have a negative pregnancy test within 72 hours of study entry. * Known history of hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] reactive) or known active hepatitis C virus (defined as HCV RNA \[qualitative\] is detected). * Known history of active TB. * Known history of HIV (HIV 1/2 antibodies).