Gene-Modified T Cells, Vaccine Therapy, and Ipilimumab in Treating Patients With Locally Advanced or Metastatic Malignancies

Inclusion Criteria: * Stage IV or locally advanced cancers for which no alternative therapies with proven survival advantage are available * At least 1 lesion amenable for outpatient biopsies; this should be a cutaneous or palpable metastatic site or a deeper site accessible by image-guided biopsy that is deemed safe to access by the treating physicians and interventional radiologists; patients without accessible lesions for biopsy but with prior tissue available from metastatic disease would be eligible at the investigator's discretion * NY-ESO-1 positive malignancy by immunohistochemistry (IHC) utilizing commonly available NY-ESO-1 antibodies * Human leukocyte antigen (HLA)-A\*0201 (HLA-A2.1) positivity by molecular subtyping * Life expectancy greater than 3 months assessed by a study physician * A minimum of one measurable lesion defined as: * Meeting the criteria for measurable disease according to Response Evaluation Criteria in Solid Tumors (RECIST) * Skin lesion(s) selected as non-completely biopsied target lesion(s) that can be accurately measured and recorded by color photography with a ruler to document the size of the target lesion(s) * No restriction based on prior treatments * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 * Absolute neutrophil count \>= 1.5 x 10\^9 cells/L * Platelets \>= 100 x 10\^9/L * Hemoglobin \>= 10 g/dL * Aspartate and alanine aminotransferases (AST, ALT) =\< 2.5 x upper limit of normal (ULN) (=\< 5 x ULN, if documented liver metastases are present) * Total bilirubin =\< 2 x ULN (except patients with documented Gilbert's syndrome) * Creatinine \< 2 mg/dl (or a glomerular filtration rate \> 60) * Must be willing and able to accept two leukapheresis procedures * Must be willing and able to provide written informed consent Exclusion Criteria: * Previously known hypersensitivity to any of the agents used in this study * Received systemic treatment for cancer, including immunotherapy, within one month prior to initiation of dosing within this protocol * History of, or significant evidence of risk for, chronic inflammatory or autoimmune disease (eg, Addison's disease, multiple sclerosis, Graves' disease, Hashimoto's thyroiditis, inflammatory bowel disease, psoriasis, rheumatoid arthritis, systemic lupus erythematosus, hypophysitis, pituitary disorders, etc.); patients will be eligible if prior autoimmune disease is not deemed to be active (e.x. fibrotic damage of the thyroid after thyroiditis or its treatment, with stable thyroid hormone replacement therapy); vitiligo will not be a basis for exclusion * History of inflammatory bowel disease, celiac disease, or other chronic gastrointestinal conditions associated with diarrhea or bleeding, or current acute colitis of any origin * Potential requirement for systemic corticosteroids or concurrent immunosuppressive drugs based on prior history or received systemic steroids within the last 4 weeks prior to enrollment (inhaled or topical steroids at standard doses are allowed) * Human immunodeficiency virus (HIV) seropositivity or other congenital or acquired immune deficiency state, which would increase the risk of opportunistic infections and other complications during chemotherapy-induced lymphodepletion; if there is a positive result in the infectious disease testing that was not previously known, the patient will be referred to their primary physician and/or infectious disease specialist * Hepatitis B or C seropositivity with evidence of ongoing liver damage, which would increase the likelihood of hepatic toxicities from the chemotherapy conditioning regimen and supportive treatments; if there is a positive result in the infectious disease testing that was not previously known, the patient will be referred to their primary physician and/or infectious disease specialist * Dementia or significantly altered mental status that would prohibit the understanding or rendering of informed consent and compliance with the requirements of this protocol * Clinically active brain metastases; radiological documentation of absence of active brain metastases at screening is required for all patients; prior evidence of brain metastasis successfully treated with surgery or radiation therapy will not be exclusion for participation as long as they are deemed under control at the time of study enrollment * Pregnancy or breast-feeding; female patients must be surgically sterile or be postmenopausal for two years, or must agree to use effective contraception during the period of treatment and 6 months after; all female patients with reproductive potential must have a negative pregnancy test (serum/urine) within 14 days from starting the conditioning chemotherapy; the definition of effective contraception will be based on the judgment of the study investigators * Since IL-2 is administered following cell infusion: * Patients will be excluded if they have a history of clinically significant electrocardiogram (ECG) abnormalities, symptoms of cardiac ischemia or arrhythmias and have a left ventricular ejection fraction (LVEF) \< 45% on a cardiac stress test (stress thallium, stress multi gated acquisition (MUGA) scan, dobutamine echocardiogram, or other stress test) * Similarly, patients who are 50 years old with a baseline LVEF \< 45% will be excluded * Patients with ECG results of any conduction delays (PR interval \> 200 ms, corrected QT interval \[QTC\] \> 480 ms), sinus bradycardia (resting heart rate \< 50 beats per minute), sinus tachycardia (heart rate \> 120 beats per minute) will be evaluated by a cardiologist prior to starting the trial; patients with any arrhythmias, including atrial fibrillation/atrial flutter, excessive ectopy (defined as \> 20 PVCs per minute), ventricular tachycardia, 3rd degree heart block will be excluded from the study unless cleared by a cardiologist * Patients with pulmonary function test abnormalities as evidenced by a forced expiratory volume of the lung in 1 second (FEV1)/forced vital capacity (FVC) \< 70% of predicted for normality will be excluded * Evidence of diverticulitis at baseline, including evidence limited to computed tomography (CT) scan only