REACTION (Radiation Enhanced Assessment of Combination Therapies in Immuno-ONcology) - Nivolumab or Nivolumab in Combination With Other Immuno-oncology (IO) Agents After Targeted Systemic Radiation in Patients With Advanced Esophagogastric Cancer

Inclusion Criteria: * Men and women aged ≥ 18 years old. * Histologically proven (squamous cell or adenocarcinoma) esophageal or gastro-esophageal junction cancer or gastric cancer (core biopsy required) * Either a formalin fixed paraffin block or a minimum of ten 5-micron tissue section's (slides) of tumor biopsy sample must be available for biomarker evaluation. * Recurrent disease or Stage IV disease as per American Joint Committee on Cancer (AJCC) staging 8.0 - patients who decline systemic chemotherapy in the first line metastatic setting are eligible. * (Relatlimab arm only) LVEF assessment with documented left ventricular ejection fraction ( LVEF) \>/=50% by either echocardiogram TTE or multigated acquisition scan (MUGA) (TTE preferred test) within 6 months from first study drug administration,whichever is most recent. * Eastern Cooperative Oncology Group (ECOG) performance status 0-1 * Adequate organ function as follows: * Leukocytes ≥ 2,000/mm3 * Absolute neutrophil count (ANC) ≥ 1000/mm3 * Platelet count ≥ 100,000/mm3 * Hemoglobin ≥ 9 g/dL * Creatinine ≤ 2.0 mg/dL * Bilirubin (total) within normal institutional limits (except subjects with Gilbert Syndrome who must have total bilirubin \< 3.0 mg/dL) * Aspartate aminotransferase (AST) (SGOT), Alanine Aminotransferase (ALT) (SGPT), and alkaline phosphatase ≤ 2.5 times the institutional upper limit of normal * prothrombin time (PT) such that international normalized ratio (INR) is ≤ 1.5 (or an in-range INR, usually between 2 and 3, if a patient is on a stable dose of therapeutic warfarin) and a partial thromboplastin time (PTT) ≤ institutional upper limit of normal * Adequate cardiac function as defined by: no evidence of (PR) prolongation or Atrioventricular block (AV block) on baseline electrocardiogram (ECG). * The effects of nivolumab, relatlimab or BMS-986178, on the developing human fetus are unknown. For this reason women of child-bearing potential (WOCBP) and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation and for 5 months after the last dose of nivolumab +/- IO therapy. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Sexually active fertile men must use effective barrier birth control if their partners are WOCBP for 7 months after the last dose of nivolumab +/- IO therapy. WOCBP must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of HCG) within two weeks of registration. * Males who are sexually active with WOCBP must agree to follow instructions for method(s) of contraception (see Appendix 4) for the duration of treatment with study treatment(s) plus 33 weeks after the last dose of the study treatment (ie, 90 days \[duration of sperm turnover\] plus the time required for nivolumab and relatlimab to undergo approximately 5 half-lives). In addition, male participants must be willing to refrain from sperm donation during this time. * Patient understands the study regimen, its requirements, risks and discomforts and is able and willing to sign the informed consent form. Voluntary signed and dated Institutional Review Board (IRB) approved written informed consent form in accordance with regulatory and institutional guidelines must be obtained before the performance of any protocol related procedures that are not part of normal patient care. Subjects must be competent to report Adverse Events (AEs), understand the drug dosing schedule and use of medications to control AEs. Exclusion Criteria: * Any active or history of autoimmune disease (including any history of inflammatory bowel disease), or history of syndrome that required systemic steroids or immune-suppressive medications, except for subjects with vitiligo or resolved childhood asthma/atopy. * (Relatlimab arm only) Troponin T (TnT) or I (TnI) \> 2 × institutional upper limit of normal (ULN). Subjects with TnT or TnI levels between \> 1 to 2 × ULN will be permitted if repeat levels within 24 hours are ≤1 x ULN. If TnT or TnI levels are \> 1 to 2 × ULN within 24 hours, the subject may undergo a cardiac evaluation and be considered for treatment, following a discussion with the BMS Medical Monitor or designee. When repeat levels within 24 hours are not available, a repeat test should be conducted as soon as possible. If TnT or TnI repeat levels beyond 24 hours are \< 2 x ULN, the subject may undergo a cardiac evaluation and be considered for treatment, following a discussion with the Bristol Myers Squibb (BMS) Medical Monitor or designee. * (Relatlimab arm only) Participants must not have a history of myocarditis * (Relatlimab arm only) Uncontrolled or significant cardiovascular disease including, but not limited to, any of the following: * Myocardial infarction (MI) or stroke/transient ischemic attack (TIA) within the 6 months prior to consent * Uncontrolled angina within the 3 months prior to consent * Any history of clinically significant arrhythmias (such as ventricular tachycardia, poorly controlled atrial fibrillation, ventricular fibrillation, or torsades de pointes) * Corrected QT interval (QTc) prolongation \> 480 msec * History of other clinically significant cardiovascular disease (i.e., cardiomyopathy, congestive heart failure with New York Heart Association (NYHA) functional classification III-IV, pericarditis, significant pericardial effusion, significant coronary stent occlusion, , poorly controlled venous thrombosis etc.) * Cardiovascular disease-related requirement for daily supplemental oxygen * History of two or more MIs OR two or more coronary revascularization procedures * (Relatlimab arm only) LVEF (Left Ventricular Ejection Fraction) assessment with documented LVEF ≥ 50% by either TTE or MUGA (TTE preferred test) within 6 months from first study drug administration. * Ongoing requirement for systemic corticosteroids. However, inhalational steroids are allowed. * Subjects with previous malignancies (except non-melanoma skin cancers, in situ bladder, gastric, breast, colon or cervical cancers/dysplasia) are excluded unless a complete remission was achieved at least 2 years prior to study entry and no additional therapy is required or anticipated to be required during the study period. * Subjects with known brain metastasis are excluded from this study. Patients with suspected brain metastasis must have brain imaging (either MRI brain or CT brain with contrast) prior to enrollment. * Subjects with a history of interstitial lung disease. Patients requiring continuous supplemental oxygen are excluded. * Use of any vaccines against infectious diseases (e.g., influenza, varicella. etc.) within 4 weeks (28 days) of initiation of study therapy. * Active systemic infection requiring therapy, positive tests for Hepatitis B surface antigen or Hepatitis C ribonucleic acid (RNA). * Known positive history or positive test for Human Immunodeficiency Virus or Acquired ImmunoDeficiency Syndrome (AIDS). * History of allergy to study drug components. * Women who are pregnant or nursing. * Men with female partners (WOCBP) that are not willing to use contraception * Underlying medical conditions that, in the Investigator's opinion, will make the administration of study drug or radiation hazardous or obscure the interpretation of toxicity or adverse events. * Prisoners or subjects who are involuntarily incarcerated or compulsorily detained for treatment of either a psychiatric or physical (e.g. infectious disease) illness.