Gene-Modified T Cells, Vaccine Therapy, and Nivolumab in Treating Patients With Stage IV or Locally Advanced Solid Tumors Expressing NY-ESO-1

Inclusion Criteria: * Stage IV or locally advanced histologically confirmed solid tumors for which no alternative therapies with proven survival advantage are available * At least 1 lesion amenable for outpatient biopsies; this should be a cutaneous or palpable metastatic site or a deeper site accessible by image-guided biopsy that is deemed safe to access by the treating physicians and interventional radiologists; patients without accessible lesions for biopsy but with prior tissue available from metastatic disease would be eligible at the investigator's discretion * NY-ESO-1 positive malignancy by immunohistochemistry (IHC) utilizing commonly available NY-ESO-1 antibodies * Human leukocyte antigen (HLA)-A\*0201 (HLA-A2.1) positivity by molecular subtyping * Age greater than or equal to 16 years old * A minimum of one measurable lesion defined as: * Meeting the criteria for measurable disease according to Response Evaluation Criteria In Solid Tumors (RECIST) * Skin lesion(s) selected as non-completely biopsied target lesion(s) that can be accurately measured and recorded by color photography with a ruler to document the size of the target lesion(s) * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 * Absolute neutrophil count \>= 1.5 x 10\^9 cells/L * Platelets \>= 100 x 10\^9/L * Hemoglobin \>= 9 g/dL * Aspartate and alanine aminotransferases (AST, ALT) =\< 2.5 x upper limit of normal (ULN) (=\< 5 x ULN, if documented liver metastases are present) * Total bilirubin =\< 2 x ULN (except patients with documented Gilbert's syndrome) * Creatinine \< 2 mg/dl (or a glomerular filtration rate \> 60) * Must be willing and able to accept two leukapheresis procedures * Must be willing and able to provide written informed consent Exclusion Criteria: * Previously known hypersensitivity to any of the agents used in this study * Received systemic treatment for cancer, including immunotherapy, within one month prior to initiation of dosing within this protocol * History of, or significant evidence of risk for, chronic inflammatory or autoimmune disease (eg, Addison's disease, multiple sclerosis, Graves disease, Hashimoto's thyroiditis, inflammatory bowel disease, psoriasis, rheumatoid arthritis, systemic lupus erythematosus, hypophysitis, pituitary disorders, etc.); patients will be eligible if prior autoimmune disease is not deemed to be active (e.x. fibrotic damage of the thyroid after thyroiditis or its treatment, with stable thyroid hormone replacement therapy); vitiligo will not be a basis for exclusion * History of inflammatory bowel disease, celiac disease, or other chronic gastrointestinal conditions associated with diarrhea or bleeding, or current acute colitis of any origin * Potential requirement for systemic corticosteroids or concurrent immunosuppressive drugs based on prior history or received systemic steroids within the last 2 weeks prior to enrollment (inhaled or topical steroids at standard doses are allowed) * Human immunodeficiency virus (HIV) seropositivity or other congenital or acquired immune deficiency state; if there is a positive result in the infectious disease testing that was not previously known, the patient will be referred to their primary physician and/or infectious disease specialist * Hepatitis B or C seropositivity with evidence of ongoing liver damage; if there is a positive result in the infectious disease testing that was not previously known, the patient will be referred to their primary physician and/or infectious disease specialist * Dementia or significantly altered mental status that would prohibit the understanding or rendering of informed consent and compliance with the requirements of this protocol * Known clinically active brain metastases; prior evidence of brain metastasis successfully treated with surgery or radiation therapy will not be exclusion for participation as long as they are deemed under control at the time of study enrollment and there are no neurological signs of potential brain metastases * Pregnancy or breast-feeding; female patients must be surgically sterile or be postmenopausal for two years, or must agree to use effective contraception during the period of treatment and 6 months after; all female patients with reproductive potential must have a negative pregnancy test (serum/urine) within 24 hours from starting the conditioning chemotherapy; the definition of effective contraception will be based on the judgment of the study investigators; patients who are breastfeeding are not allowed on study * Since IL-2 is administered following cell infusion: * Patients will be excluded if they have a history of clinically significant electrocardiogram (ECG) abnormalities, symptoms of cardiac ischemia or arrhythmias and have a left ventricular ejection fraction (LVEF) \< 45% on a cardiac stress test (stress thallium, stress multi gated acquisition scan \[MUGA\], dobutamine echocardiogram, or other stress test) * Similarly, patients who are \>= 50 years old with a baseline LVEF \< 45% will be excluded * Patients with ECG results of any conduction delays (PR interval \> 200 ms, corrected QT interval \[QTC\] \> 480 ms), sinus bradycardia (resting heart rate \< 50 beats per minute), sinus tachycardia (heart rate \> 120 beats per minute) will be evaluated by a cardiologist prior to starting the trial; patients with any arrhythmias, including atrial fibrillation/atrial flutter, excessive ectopy (defined as \> 20 premature ventricular contractions \[PVCs\] per minute), ventricular tachycardia, third (3rd) degree heart block will be excluded from the study unless cleared by a cardiologist * Patients with pulmonary function test abnormalities as evidenced by a forced expiration volume in one second (FEV1)/forced vital capacity (FVC) \< 70% of predicted for normality will be excluded * Evidence of diverticulitis at baseline, including evidence limited to computed tomography (CT) scan only * Received 3 or more prior myelotoxic treatment regimens * Bone marrow involvement based on CT or PET scan at screening