Nivolumab +/- Relatlimab Prior to Chemoradiation With II/III Gastro/Esophageal Cancer

Inclusion Criteria: * Men and women aged ≥ 18 years old * Histologically proven (squamous cell or adenocarcinoma) esophageal or gastro- esophageal junction cancer (core biopsy required). * Stage II/III disease as per AJCC staging 7.0 * Baseline imaging with FDG-PET scan and endoscopic ultrasound within 28 days prior to registration * ECOG performance status 0-1 (see Appendix B). * Adequate oral intake/nutritional status without the need for enteral or parenteral feeding during chemoradiation or preoperative period * Adequate organ function as follows: * Leukocytes ≥ 2,000/mm3 * Absolute neutrophil count (ANC) ≥ 1000/mm3 * Platelet count ≥ 100,000/mm3 * Hemoglobin ≥ 9 g/dL * Creatinine ≤ 2.0 mg/dL * Bilirubin (total) within normal institutional limits (except subjects with Gilbert Syndrome who must have total bilirubin \< 3.0 mg/dL) * AST(SGOT), ALT(SGPT), and alkaline phosphatase ≤ 2.5 times the upper limit of normal * PT such that international normalized ratio (INR) is ≤ 1.5 (or an in-range INR, usually between 2 and 3, if a patient is on a stable dose of therapeutic warfarin and a PTT ≤ upper limit of normal * Adequate cardiac function as defined by: no evidence of PR prolongation or AV block on baseline electrocardiogram (ECG). * Radiation oncology consultation within 28 days to confirm that disease can be encompassed in the radiotherapy field and that normal tissue constraints can be met. * Subjects must have adequate lung function to permit surgical resection determined by pre-enrollment pulmonary function tests to include DLCO as follows: * DLCO≥70% predicted OR DLCO\<70% but ≥55% with a VO2 max ≥10L/min/kg (assessed by cardiopulmonary exercise testing) or 6 minute walk test ≥500 meters * Subjects with a DLCO\<55% are excluded from this study. * Subjects must have a baseline O2 saturation by pulse oximetry that is ≥ 92% both at rest and while walking, off supplemental oxygen * Esophagogastrectomies will be performed via a laparotomy and a right thoracotomy with en-bloc removal of perigastric, celiac, periesophageal and subcarinal lymph nodes. Esophagogastric reconstruction will be performed above the level of the azygo-caval junction using an EEA stapling device. * Either a formalin fixed paraffin block or a minimum of ten 5-micron tissue section's (slides) of tumor biopsy sample must be available for biomarker evaluation from baseline and repeat EGD. * The effects of nivolumab, on the developing human fetus are unknown. For this reason women of child-bearing potential (WOCBP) and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation and for 5 months after the last dose of nivolumab. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Sexually active fertile men must use effective barrier birth control if their partners are WOCBP for 7 months after the last dose of nivolumab. WOCBP must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of HCG) within two weeks of registration. * Patient understands the study regimen, its requirements, risks and discomforts and is able and willing to sign the informed consent form. Voluntary signed and dated IRB/IEC approved written informed consent form in accordance with regulatory and institutional guidelines must be obtained before the performance of any protocol related procedures that are not part of normal patient care. Subjects must be competent to report AEs, understand the drug dosing schedule and use of medications to control AEs. * (Relatlimab arm only) LVEF (Left Ventricular Ejection Fraction) assessment with documented LVEF ≥ 50% by either TTE or MUGA (TTE preferred test) within 6 months from first study drug administration Exclusion Criteria: * Patient has active, known or suspected autoimmune disease. Subjects with vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune thyroiditis only requiring hormone replacement, or conditions not expected to recur in the absence of an external trigger are permitted to enroll. * Esophageal tumors that are located in the mid esophagus or higher i.e. not involving distal esophagus or GE junction. * Tumors whose proximal end are higher that the level of the carina * Biopsy proven involvement of supraclavicular lymph nodes * Tumors must not extend 5cm or more into the stomach * Patient has a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalent) or other immunosuppressive medications within 14 days of first dose. Inhaled or topical steroids and adrenal replacement steroid doses are permitted in the absence of active autoimmune disease. * Subjects with previous malignancies (except non-melanoma skin cancers, in situ bladder, gastric, breast, colon or cervical cancers/dysplasia) are excluded unless a complete remission was achieved at least 1 years prior to study entry and no additional therapy (other than adjuvant hormonal therapy for breast cancer) is required or anticipated to be required during the study period. * Subjects with brain metastasis are excluded from this study and all patients should have brain imaging (either MRI brain or CT brain with contrast) prior to enrollment. * Subjects with a history of interstitial lung disease. * Active systemic infection requiring therapy, positive tests for Hepatitis B surface antigen or Hepatitis C ribonucleic acid (RNA). * Known positive history or positive test for Human Immunodeficiency Virus or Acquired ImmunoDeficiency Syndrome (AIDS). * History of allergy to study drug components. * Women who are pregnant or nursing. * WOBP and Men with female partners (WOCBP) that are not willing to use contraception. * Prior therapy with an anti-PD-1, anti-PD-L1, anti-PDL-2, or anti-CTLA-4 antibody (or any other antibody targeting T cell co-regulatory pathways). * Underlying medical conditions that, in the Investigator's opinion, will make the administration of study drug hazardous or obscure the interpretation of toxicity or adverse events. * Prisoners or subjects who are involuntarily incarcerated or compulsorily detained for treatment of either a psychiatric or physical (e.g. infectious disease) illness. * (Relatlimab arm only) Troponin T (TnT) or I (TnI) \> 2 × institutional ULN. Subjects with TnT or TnI levels between \> 1 to 2 × ULN will be permitted if repeat levels within 24 hours are ≤1 x ULN. If TnT or TnI levels are \> 1 to 2 × ULN within 24 hours, the subject may undergo a cardiac evaluation and be considered for treatment, following a discussion with the BMS Medical Monitor or designee. When repeat levels within 24 hours are not available, a repeat test should be conducted as soon as possible. If TnT or TnI repeat levels beyond 24 hours are \< 2 x ULN, the subject may undergo a cardiac evaluation and be considered for treatment, following a discussion with the BMS Medical Monitor or designee. * (Relatlimab arm only) Participants must not have a history of myocarditis * (Relatlimab arm only) Uncontrolled or significant cardiovascular disease including, but not limited to, any of the following: * Myocardial infarction (MI) or stroke/transient ischemic attack (TIA) within the 6 months prior to consent * Uncontrolled angina within the 3 months prior to consent * Any history of clinically significant arrhythmias (such as ventricular tachycardia, poorly controlled atrial fibrillation, ventricular fibrillation, or torsades de pointes) * QTc prolongation \> 480 msec * History of other clinically significant cardiovascular disease (i.e., cardiomyopathy, congestive heart failure with New York Heart Association \[NYHA\] functional classification III-IV, pericarditis, significant pericardial effusion, significant coronary stent occlusion, , poorly controlled venous thrombosis etc.) * Cardiovascular disease-related requirement for daily supplemental oxygen * History of two or more MIs OR two or more coronary revascularization procedures